Skip to content

SS-348: Avoid eager materialization of diff pairs in iter_diff_pairs#37817

Merged
martykulma merged 1 commit into
MaterializeInc:mainfrom
martykulma:maz-lazy-iter-diff-pairs
Jul 23, 2026
Merged

SS-348: Avoid eager materialization of diff pairs in iter_diff_pairs#37817
martykulma merged 1 commit into
MaterializeInc:mainfrom
martykulma:maz-lazy-iter-diff-pairs

Conversation

@martykulma

Copy link
Copy Markdown
Contributor

Skip materialization of diff pairs (i.e. collect()) as it causes memory bloat in some cases. This PR changes the return type to a lazy iterator and adds a unit test to validate the issue and confirm the fix.

ref: @def- found a pathological case in refactor of for_each_diff_pair that results in high memory usage shortly after it was merged.

- returns an Iterator instead of a Vec
- adds a unit test to validate a cap on clones
@martykulma martykulma changed the title SS-348: Avoid eager materialization of batch in iter_diff_pairs SS-348: Avoid eager materialization of diff pairs in iter_diff_pairs Jul 23, 2026
@martykulma
martykulma marked this pull request as ready for review July 23, 2026 11:47
@martykulma
martykulma requested a review from a team as a code owner July 23, 2026 11:47
@martykulma

Copy link
Copy Markdown
Contributor Author

The test failures don't appear to be related to the change (the SHOW CREATE CLUSTER is missing EXPERIMENTAL ARRANGEMENT COMPRESSION).

@martykulma
martykulma requested review from a team and def- July 23, 2026 11:49
@def-

def- commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Nightly failures are indeed unrelated, fixed in #37822

@patrickwwbutler patrickwwbutler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Agreed that test failures on nightly seem unrelated

@martykulma

Copy link
Copy Markdown
Contributor Author

TFTR!

@martykulma
martykulma merged commit 2d842fe into MaterializeInc:main Jul 23, 2026
210 of 213 checks passed
@martykulma
martykulma deleted the maz-lazy-iter-diff-pairs branch July 23, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants