[SPARK-57561][SQL] Add size validation to bitmap_or_agg and bitmap_and_agg - #57567
Open
ybapat wants to merge 1 commit into
Open
[SPARK-57561][SQL] Add size validation to bitmap_or_agg and bitmap_and_agg#57567ybapat wants to merge 1 commit into
ybapat wants to merge 1 commit into
Conversation
… oversized input validation The ExpressionDescription for bitmap_or_agg and bitmap_and_agg stated 'input should be bitmaps from bitmap_construct_agg()' while examples used 1-byte literals. Clarify that shorter inputs are supported and add a runtime error for inputs exceeding 4096 bytes (which would otherwise be silently truncated). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
Spark-57561 is a task for this PR #56999 Isn't it? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
bitmap_or_aggandbitmap_and_aggsilently accept input bitmaps of any size. When an input bitmap is larger thanBitmapExpressionUtils.NUM_BYTES(4096 bytes),bitmapMerge/bitmapAndMergesilently reads only the first 4096 bytes, discarding the rest. This can produce incorrect results without any indication that data was dropped.This PR adds an explicit runtime check in the
update()methods of bothBitmapOrAggandBitmapAndAgg. If the input exceeds 4096 bytes, aSparkRuntimeExceptionwith error classBITMAP_INPUT_TOO_LARGEis thrown.Changes:
BITMAP_INPUT_TOO_LARGEerror condition toerror-conditions.json(SQLSTATE22001— string/binary data right truncation).bitmapInputTooLargeErrorfactory method toQueryExecutionErrors.BitmapOrAgg.update()andBitmapAndAgg.update().@ExpressionDescriptionfor both aggregates to accurately document short-input behavior and the size limit.BitmapExpressionsQuerySuiteverifying the error is thrown.Why are the changes needed?
Without validation, oversized inputs are silently truncated, producing incorrect aggregate results with no warning. Explicit validation follows Spark's design principle of failing fast with a structured error rather than silently producing wrong results.
Does this PR introduce any user-facing change?
Yes. Queries that pass bitmaps larger than 4096 bytes to
bitmap_or_aggorbitmap_and_aggwill now fail withBITMAP_INPUT_TOO_LARGEinstead of silently truncating. Inputs created bybitmap_construct_agg()are always exactly 4096 bytes and are unaffected.How was this patch tested?
"bitmap_or_agg rejects input larger than 4096 bytes"and"bitmap_and_agg rejects input larger than 4096 bytes"inBitmapExpressionsQuerySuite.Closes #57561
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com