[EPIC] Faster manifest reads through pyiceberg-core

[mbutrovich]

Feature Request / Improvement

PyIceberg reads manifests and manifest lists with its Cython Avro decoder. pyiceberg-core (the iceberg-rust Python binding) already exposes read_manifest_entries and read_manifest_list, but using them today makes manifest reads 4x to 5x slower than Cython. This issue tracks the work in both repos to make the pyiceberg-core path faster than Cython, and then to switch PyIceberg to it. The iceberg-rust side is tracked in apache/iceberg-rust#3262, which repeats those tasks in more detail.

The earlier attempt (apache/iceberg-rust#1280, #2493) was closed. It converted every Rust manifest entry into a PyIceberg DataFile in Python, which is the rust_convert path measured below. The full benchmark writeup from @kevinjqliu, including an in-crate profile of the Rust parser and the script, is in kevinjqliu#45.

Where the time goes

These numbers come from manifests written by PyIceberg (V2, deflate, 12 columns with full column stats, identity partition). pyiceberg is at main (0d58407) with the Cython decoder. pyiceberg-core is built locally from the v0.10.1 tag, once as released (opt-level = "z") and once with opt-level = 3. Each number is the best of 7 runs in ms, single threaded, on Apple Silicon with Python 3.12. v0.10.1 predates apache/iceberg-rust#3028, which its PR measured as making the Rust parse about 5% faster.

entries	Cython	pyiceberg-core as released	pyiceberg-core with opt-level = 3
1,000	8.7	43.9	30.8
10,000	94.1	452.3	320.4
50,000	565.9	2308.7	1628.0

The pyiceberg-core columns include building PyIceberg ManifestEntry and DataFile objects, so all three columns produce the same result. At 10,000 entries with opt-level = 3, the Rust parse costs 21.7 us per entry and converting the result into PyIceberg objects costs another 10.3 us per entry. The whole Cython path costs 10.3 us per entry in the same run. The conversion alone costs as much as Cython, so a faster Rust parser can't reach parity unless PyIceberg also stops building one Python object per entry from the binding's output.

Manifest lists are closer. With opt-level = 3 the Rust parse of a 10,000-entry manifest list takes 8.0 ms against 23.9 ms for Cython, and the conversion into ManifestFile objects is what makes the end-to-end path slower (39.1 ms).

Task list

These tasks are in PyIceberg.

• Write equality_ids as list<int> per the spec. PR: Manifest: Use int for equality_ids in manifest schema per Iceberg spec (#3840) #3842.
• Flush manifest entries in size-bounded Avro blocks. ManifestWriter.add_entry calls write_block once per entry (manifest.py#L1213-L1238), so every entry gets its own deflate stream and sync marker. Java writes manifests through Avro's DataFileWriter and keeps its default sync interval, which the linked issue reports as 64 KB. In the linked benchmark, a 10,000-entry manifest shrank from 3.6 MB to 0.7 MB and read about 1.5x faster with both decoders after re-encoding with 64 KB blocks. The manifest-list writer already writes one block (manifest.py#L1389). This task doesn't depend on anything else. PR: perf(avro): buffer records into size-bounded blocks #4008
• Build inspect.entries() and inspect.files() from pyiceberg-core Arrow output. Both methods build a DataFile per entry and then flatten them back into a pa.Table (inspect.py#L152, inspect.py#L860). These are the first consumer of the Arrow output from the iceberg-rust tasks, because they want Arrow anyway. This task depends on that output being in a pyiceberg-core release.
• Plan scans through pyiceberg-core. _open_manifest (table/__init__.py#L2321-L2334) filters each entry with Python partition and metrics evaluators while decoding (feat: Fuse partition and metrics filtering into manifest deserialization #3658), but every entry still becomes a DataFile before it is filtered. To beat Cython, filtering has to happen on Arrow columns or inside Rust, so that only the matching entries become Python objects. Which of the two to use is open and is settled together with the iceberg-rust task for filtering. pyiceberg-core is an optional extra, so the Cython reader stays as the fallback.
• Read manifest lists through pyiceberg-core. This uses the same approach as scan planning, applied to read_manifest_list.

These tasks are in iceberg-rust and tracked in apache/iceberg-rust#3262.

• Faster manifest parsing. Upgrade to apache-avro 0.22 and decode entries directly from the writer schema instead of through apache_avro::Value and schema resolution (Upgrade to using apache-avro 0.22 iceberg-rust#3063).
• Build the pyiceberg-core wheel with opt-level = 3. In the numbers above, this makes the parse about 1.6x faster and the end-to-end path about 1.4x faster. It also nearly doubles the wheel size.
• Return manifest entries and manifest lists as Arrow from pyiceberg-core.
• Release pyiceberg-core with the above. PyIceberg pins pyiceberg-core>=0.10.1,<0.11.0.

[mbutrovich]

I am working on this, but can't assign it to myself like the iceberg-rust counterpart.