Search before asking
Paimon version
master
Compute Engine
Any (primary-key table vector search)
Minimal reproduce step
On a first-row merge-engine table with deletion vectors disabled and a pk-vector index, write a key, let it compact, then write the same key again (a version first-row drops on read). Run a vector search with vector-index.search-mode set to full or detail.
What doesn't meet your expectations?
The vector search can return the superseded row version, which a normal merge-on-read query would never return.
Root cause: the exact scan in PrimaryKeyVectorBucketSearch returns physical row positions per file with no dedup by primary key, and the readback (PrimaryKeyIndexedSplitRead / RawFileSplitRead) reads those positions directly without a merge-on-read pass. first-row resolves duplicate keys only through the merge/compaction path, so when a superseded version has not yet been merged away it is physically present and the exact scan can surface it. This is specific to the first-row engine with deletion vectors disabled; deletion-vector-enabled tables mark superseded rows and exclude them from the exact scan.
This predates #9894. That change extends the exact scan to level-0 APPEND files, which makes the surfacing more frequent (a fresh superseded write is the common case), but the underlying gap already applied to superseded versions promoted into compacted files.
Anything else?
One direction would be to apply a primary-key dedup to the exact-scan candidates before ranking (or require deletion vectors for full/detail on primary-key vector search). Filed for tracking, not blocking #9894.
Are you willing to submit a PR?
Search before asking
Paimon version
master
Compute Engine
Any (primary-key table vector search)
Minimal reproduce step
On a
first-rowmerge-engine table with deletion vectors disabled and apk-vectorindex, write a key, let it compact, then write the same key again (a versionfirst-rowdrops on read). Run a vector search withvector-index.search-modeset tofullordetail.What doesn't meet your expectations?
The vector search can return the superseded row version, which a normal merge-on-read query would never return.
Root cause: the exact scan in
PrimaryKeyVectorBucketSearchreturns physical row positions per file with no dedup by primary key, and the readback (PrimaryKeyIndexedSplitRead/RawFileSplitRead) reads those positions directly without a merge-on-read pass.first-rowresolves duplicate keys only through the merge/compaction path, so when a superseded version has not yet been merged away it is physically present and the exact scan can surface it. This is specific to thefirst-rowengine with deletion vectors disabled; deletion-vector-enabled tables mark superseded rows and exclude them from the exact scan.This predates #9894. That change extends the exact scan to level-0 APPEND files, which makes the surfacing more frequent (a fresh superseded write is the common case), but the underlying gap already applied to superseded versions promoted into compacted files.
Anything else?
One direction would be to apply a primary-key dedup to the exact-scan candidates before ranking (or require deletion vectors for
full/detailon primary-key vector search). Filed for tracking, not blocking #9894.Are you willing to submit a PR?