[Feature] Integrate Paimon with Sirius's table-aware GPU scan framework

[SteNicholas]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Sirius is a GPU-native analytics engine built on NVIDIA cuDF that plugs into existing databases through Substrait; today it accelerates DuckDB. Its GPU scan path covers a fixed set of sources: native DuckDB tables, read_parquet and iceberg_scan (sirius-db/sirius#1425). Paimon tables cannot be read on the GPU.

Sirius is replacing per-format scan code with one table-aware scan framework that Iceberg, DuckLake and Paimon plug into (tracking issue sirius-db/sirius#1760, framework contracts in sirius-db/sirius#1796). A format joins by registering an adapter that states what it supports, and the framework applies the same rules to every format:

• The snapshot, schema and transaction are recorded when the query is bound, and every later step (GPU plan, per-batch checks, CPU fallback) reads that same snapshot. Nothing re-resolves "latest" mid-query.
• Before any GPU reading, Sirius decides whether the GPU can run the query: supported, unsupported, or not enough information yet.
• Inputs are prepared on background workers; each file or batch goes to the existing GPU scheduler as soon as it is ready and checked.
• If something fails after execution starts, Sirius stops and either replays DuckDB's saved CPU plan (read-only query, every source allows it, no rows returned yet) or returns an explicit error.

Paimon does not fit the approach Sirius used for Iceberg, which subclasses the Parquet reader: merge-on-read primary-key tables are not a set of independently readable Parquet files. Sirius therefore plans to read Paimon on its native-reader path: paimon-cpp, the library duckdb-paimon is built on, produces Arrow record batches, which Sirius checks against the bound schema and imports into cuDF.

Solution

Add a Paimon adapter on Sirius's native-reader path, and fill the gaps on the Paimon side that the adapter needs:

1. Snapshot-pinned split planning. Plan splits for the snapshot recorded at bind time, so the GPU plan, the per-batch checks and any CPU replay all read the same data.
2. Capability report before reading. For the bound table and each split, report what reading it requires (LSM merge, deletion vectors, schema evolution, file format, external paths). The adapter maps this to supported, unsupported or not enough information, and refuses what it cannot handle instead of returning wrong results.
3. Arrow batches Sirius can import safely. Batches carry field IDs and match the bound schema, and stay valid until the GPU has consumed them. Reads run on Sirius's background workers; GPU tasks never perform blocking native reads.
4. Filter pushdown contract. Report which predicates Paimon fully applied (partition, file and row-group pruning), so Sirius evaluates only the remaining filters on the GPU and never drops one that was not applied.
5. Explicit read failures. Read errors surface as explicit statuses, so Sirius can stop the query and apply its replay-or-error rule.

Acceptance: for the same snapshot, GPU queries over append-only and primary-key tables return the same results as DuckDB with duckdb-paimon on the CPU, and table features the adapter does not support are refused with an explicit reason.

Follow-up (not required to close this issue): for splits that need no merge (append-only tables, or primary-key tables with deletion vectors), expose the data files together with their deletion vectors, the equivalent of Java Split#convertToRawFiles() and Split#deletionFiles(). Sirius could then decode those Parquet files on the GPU with cuDF instead of decoding on the CPU and importing Arrow. The public DataSplit::GetFileList() in paimon-cpp covers append-only tables only.

Anything else?

Out of scope (same as sirius-db/sirius#1760): writes, compaction, streaming reads, LSM merge on the GPU, and routing native readers through Sirius's I/O layer.

Dependencies: the Sirius adapter depends on the shared scan framework (sirius-db/sirius#1796); the Paimon child issue in Sirius has not been opened yet. Changes needed in paimon-cpp may need companion issues in apache/paimon-cpp.

Are you willing to submit a PR?

• I'm willing to submit a PR!