Skip to content

[core] Use full-range anchor as row representative in DE read-type pruning - #9899

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/de-scan-anchor-representative
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/de-scan-anchor-representative

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9898

When a data-evolution row-id-range group has no file that writes the query's projected columns, read-type pruning keeps one file as a row-count representative so the reader emits the right number of NULL-filled rows. The representative was group.get(0), which can be a blob or vector-store file. Those files cover only a sub-range of the group's row ids, so the split silently emitted fewer rows than the group actually contains. This uses retrieveAnchorFile instead, which returns a full-range normal data file and never a blob or vector-store file (the same primitive the deletion-vector anchor path already relies on).

The pruning core is extracted into a @VisibleForTesting static method so the representative choice can be pinned by a unit test with a blob file ordered first in the group. The filter-field logic added by #9872 is preserved: the instance method still folds in the fields referenced only by the filter and passes them to the static method, so the set of kept files is unchanged.

Tests

DataEvolutionFileStoreScanTest#testReadTypePruningKeepsAnchorAsRowRepresentative builds a group of a blob file covering rows 0-1 (ordered first) plus two full-range normal files covering rows 0-9, with a readType referencing a column no file writes. It asserts the representative is the full-range normal file. Against the old group.get(0) fallback the representative is the blob file, so the assertion fails.

…uning

When every file in a data-evolution row-id-range group lacks the columns
of the query projection (e.g. a freshly added column), pruneByReadType
keeps one file as a row-count representative so the reader can emit the
right number of NULL-filled rows. The representative was group.get(0),
which can be a blob or vector-store file: such files cover only a
sub-range of the group's row ids, so the split silently emitted fewer
rows than the group contains.

Use retrieveAnchorFile instead, which selects the oldest full-range
normal file and never a blob/vector-store file.

The pruning core is extracted into a static method (mirroring
evolutionStats) so the representative choice can be pinned by a unit
test with a blob file ordered first in the group.

Assisted-by: GLM-5.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Data-evolution read-type pruning collapses a split's row count when the group's first file is a blob/vector-store file

1 participant