Skip to content

[python] Support reading shared-shredding maps - #9763

Merged
JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/pypaimon-shared-shredding-map-read
Sep 13, 2026
Merged

[python] Support reading shared-shredding maps#9763
JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/pypaimon-shared-shredding-map-read

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon currently exposes the physical struct when reading a MAP<STRING, T> stored with the shared-shredding layout. Restore the complete logical map instead.

Changes

  • parse shared-shredding field metadata with none, lz4, and zstd dictionary compression
  • assemble fixed physical columns and overflow entries into the logical Arrow MAP
  • preserve null and empty maps and arbitrary supported value types
  • recover Paimon field metadata from the ORC ARROW:schema entry across supported PyArrow versions
  • restore ORC physical representations of TIME, TIMESTAMP, and TIMESTAMP_LTZ, including values nested in rows, arrays, and maps
  • resolve shared-shredding metadata through source paths so nested-projection output aliases remain transparent

Selected-key projection and shared-shredding writes are intentionally out of scope.

Tests

53 related reader tests passed locally on Python 3.11 / PyArrow 19. The seven shared-shredding tests also passed on Python 3.6 / PyArrow 6; the existing nested VARIANT reader suite passed there as well. Java-written ORC files using all three metadata compressions and TIME values were verified with both PyArrow versions. PyArrow-written ORC reader regressions cover TIME, TIMESTAMP precision, and nested TIMESTAMP_LTZ values. The projection regression covers a nested a.b output colliding with a complete shared-shredding a_b column. A mixed-language E2E now has Java write Parquet and ORC shared-shredding MAPs across two files, and Python verify fixed/overflow keys, null values, empty maps, null maps, and per-file dictionaries. Flake8, Spotless, shell syntax, and git diff --check passed.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 13, 2026 04:04

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some E2E test cases.

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor Author

Please add some E2E test cases.

Added

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit bfd6abc into apache:master Sep 13, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants