Search before asking
Paimon version
master
Compute Engine
Any (data-evolution table read)
Minimal reproduce step
On a data-evolution table with deletion vectors disabled, read a row-id-range group where a blob or vector-store file is ordered first in the group, and the query projects a column that no file in the group wrote (for example a freshly added column over a pre-ALTER group). This drives DataEvolutionFileStoreScan read-type pruning down its empty-kept fallback.
What doesn't meet your expectations?
The split reports fewer rows than the group actually contains, so rows are silently dropped from the result.
Root cause: when every file in the group lacks the projected columns, pruning keeps one file as a row-count representative so the reader can emit the right number of NULL-filled rows. The representative was group.get(0), which can be a blob or vector-store file. Those files cover only a sub-range of the group's row ids, so the split's row count collapses to that sub-range and the rest of the group's rows disappear.
Anything else?
The fix is to pick the row-count representative with retrieveAnchorFile, which returns a full-range normal data file and never a blob or vector-store file.
Are you willing to submit a PR?
Search before asking
Paimon version
master
Compute Engine
Any (data-evolution table read)
Minimal reproduce step
On a data-evolution table with deletion vectors disabled, read a row-id-range group where a blob or vector-store file is ordered first in the group, and the query projects a column that no file in the group wrote (for example a freshly added column over a pre-
ALTERgroup). This drivesDataEvolutionFileStoreScanread-type pruning down its empty-keptfallback.What doesn't meet your expectations?
The split reports fewer rows than the group actually contains, so rows are silently dropped from the result.
Root cause: when every file in the group lacks the projected columns, pruning keeps one file as a row-count representative so the reader can emit the right number of NULL-filled rows. The representative was
group.get(0), which can be a blob or vector-store file. Those files cover only a sub-range of the group's row ids, so the split's row count collapses to that sub-range and the rest of the group's rows disappear.Anything else?
The fix is to pick the row-count representative with
retrieveAnchorFile, which returns a full-range normal data file and never a blob or vector-store file.Are you willing to submit a PR?