[Bug] Java Parquet reads can drop matching rows when filter columns are not projected

[wangzhigang1999]

Paimon version

Reproduced on master at 43e9ad701ebee6d9af8316f35533f4d0836a1fc0 (2.2-SNAPSHOT), with Parquet 1.16.0.

Compute Engine

Java API on JDK 17, through FileStoreTable.newReadBuilder().

Minimal reproduce step

Create an append-only table with bucket = -1, file.format = parquet, and write-only = true. Use schema id INT, status STRING and commit these four rows through the Java batch write API:

id	status
0	A
1	B
2	NULL
3	A

Read with status = 'A' as the filter and only id in the read type:

ReadBuilder readBuilder =
        table.newReadBuilder()
                .withFilter(
                        new PredicateBuilder(rowType)
                                .equal(1, BinaryString.fromString("A")))
                .withReadType(rowType.project(new int[] {0}));
List<Integer> ids = new ArrayList<>();
try (RecordReader<InternalRow> reader =
        readBuilder.newRead().createReader(readBuilder.newScan().plan().splits())) {
    reader.forEachRemaining(row -> ids.add(row.getInt(0)));
}

Actual result: ids is empty. The same reproduction fails with file-index.bitmap.columns = status enabled.

What doesn't meet your expectations?

The result must contain IDs 0 and 3. The Java read API allows best-effort filtering and extra candidate rows, but matching rows must not be lost.

Anything else?

Related closed PR: #5385, which described the same empty-result behavior. In that PR, @JingsongLi suggested considering only projected fields when converting Parquet filters and discarding other filters. The author's implementation added filter fields to the read projection; both commits preceded the review comments, and the PR was closed without a subsequent implementation of that suggestion.

I reproduced the problem through the public Java table-read API on the commit above. In that version, ParquetReaderFactory converts predicates against the physical file schema without first removing conditions on unprojected fields. During page pruning, Parquet treats a field absent from the requested columns as a missing column with null values, even when that field exists in the file.

The candidate fix follows the suggested direction: retain AND conjuncts covered by the read projection before Parquet conversion, and discard an entire OR conjunct if it references an unprojected field. It keeps the requested columns and the Parquet page-index implementation unchanged.

Spark/Flink SQL reproduction has not been established. PyPaimon's PyArrow reader does not call this Java reader.