Search before asking
Paimon version
master
Compute Engine
Spark / core (data-evolution read)
Minimal reproduce step
On a data-evolution table where a column is written only by a "dedicated" file (a blob or vector-store file that covers just a sub-range of a row-id-range group), project only that column. For example, a group has a full-range normal file writing v over row ids [0, 9] and a blob file writing b over row ids [0, 2], then SELECT b.
What doesn't meet your expectations?
The read returns fewer rows than the group contains. DataEvolutionFileStoreScan.pruneByReadType keeps only the files that write a projected (or filter) field, so the kept list holds just the blob file. A dedicated file never spans the group's full row-id range, so the reader derives the group's logical range from the blob's sub-range [0, 2] and silently drops the rows outside it ([3, 9]). The deletion-vector path already keeps a full-range anchor normal file unconditionally; the plain read path does not.
Expected: the group always contributes a full-range normal file (the anchor) so the reader sees the correct row count and NULL-fills the projected column for the rows the dedicated file does not cover.
Anything else?
Fix direction: when read-type pruning would leave a group's kept list holding only dedicated (blob / vector-store) files, also keep the full-range anchor normal file, mirroring what the deletion-vector path already does.
Are you willing to submit a PR?
Search before asking
Paimon version
master
Compute Engine
Spark / core (data-evolution read)
Minimal reproduce step
On a data-evolution table where a column is written only by a "dedicated" file (a blob or vector-store file that covers just a sub-range of a row-id-range group), project only that column. For example, a group has a full-range normal file writing
vover row ids [0, 9] and a blob file writingbover row ids [0, 2], thenSELECT b.What doesn't meet your expectations?
The read returns fewer rows than the group contains.
DataEvolutionFileStoreScan.pruneByReadTypekeeps only the files that write a projected (or filter) field, so the kept list holds just the blob file. A dedicated file never spans the group's full row-id range, so the reader derives the group's logical range from the blob's sub-range [0, 2] and silently drops the rows outside it ([3, 9]). The deletion-vector path already keeps a full-range anchor normal file unconditionally; the plain read path does not.Expected: the group always contributes a full-range normal file (the anchor) so the reader sees the correct row count and NULL-fills the projected column for the rows the dedicated file does not cover.
Anything else?
Fix direction: when read-type pruning would leave a group's kept list holding only dedicated (blob / vector-store) files, also keep the full-range anchor normal file, mirroring what the deletion-vector path already does.
Are you willing to submit a PR?