Avro: Cache Avro schema conversion in AvroFileHeader.get_schema (#3662) - #3844
hedger9487 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR optimizes Avro manifest/schema handling by caching the Avro-schema-string → Iceberg Schema conversion, avoiding repeated json.loads() + recursive conversion work across many manifest reads during scan planning.
Changes:
- Added an
@lru_cache(maxsize=128)-backed_parse_avro_schema()helper to cache schema parsing/conversion by schema string. - Updated
AvroFileHeader.get_schema()to use the cached helper. - Added a unit test asserting schema conversion results are reused (object identity) across headers with the same embedded schema string.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
pyiceberg/avro/file.py |
Introduces an LRU-cached schema parsing/conversion helper and routes get_schema() through it. |
tests/avro/test_file.py |
Adds a regression test to ensure schema conversion is cached and reused across calls. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Description
Fixes #3662.
Every manifest file under a partition spec embeds an identical Avro schema string (
avro.schema). Previously,AvroFileHeader.get_schema()inpyiceberg/avro/file.pyperformed a fulljson.loads()and recursiveAvroSchemaConversion().avro_to_iceberg()conversion on every manifest read during scan planning.This PR adds an LRU cache for Avro schema string parsing via
_parse_avro_schema, avoiding redundant conversions across repeated manifest reads during scan planning (e.g. acceleratingscan().plan_files()by ~1.8x on tables with many manifests).Testing
test_get_schema_is_cachedintests/avro/test_file.py.