[Bug] Vector search drops rows when the scalar index cannot fully evaluate the filter

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master

Compute Engine

Any (data-evolution vector search: local / Spark / Flink)

Minimal reproduce step

On a data-evolution table with a vector index, run a vector search whose WHERE filter the scalar global index cannot evaluate. Two shapes trigger it:

1. The filter uses a function the scalar index does not support, e.g. IS NOT NULL on a multivalue-indexed array column, or references a column that has no scalar index.
2. The filter is a partially-evaluable AND, e.g. a = 1 AND tags IS NOT NULL where tags IS NOT NULL is unsupported.

What doesn't meet your expectations?

Rows that match the filter go missing from the result (shape 1), or the top-K is polluted by rows that do not match the filter and matching rows are displaced out of it (shape 2).

Root cause: vector search delegates pre-filtering to the scalar global index. scalarMatchedRows returned an empty bitmap when the index could not evaluate the predicate, and that bitmap was AND-ed into every index split, so every index-covered row was dropped. For a partially-evaluable AND, the evaluator drops the unsupported conjunct and returns a superset that the index path trusts as exact. The index read path has no final-read filter, so an inexact bitmap can neither be corrected nor ignored.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!