refactor(spec): make UnboundPartitionField fields private and carry source-ids - #3267
Open
moomindani wants to merge 1 commit into
Open
moomindani wants to merge 1 commit into
moomindani wants to merge 1 commit into
Conversation
The four public fields are replaced by accessors, and the derived builder's visibility is limited to the crate, so an instance can only be built inside the crate or read out of a partition spec JSON. That is what lets the struct guarantee its own invariant: source_ids always holds at least one id. source_ids: Vec<i32> replaces source_id: i32. source_id() returns the single id, and an error for a multi-argument field rather than quietly handing back the first one. The serde module reads either spelling -- source-id, or source-ids for a v3 multi-argument transform -- and writes back the one that matches the field. Binding a multi-argument field now fails with a clear error. A bound PartitionField still carries a single source_id, so until that struct is converted there is nowhere to put the extra ids, and failing loudly beats dropping them on the way into TableCreation or TableUpdate::AddSpec.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #3172 and #2801. First of the three per-struct PRs @blackmwk asked for in #2802, starting with
UnboundPartitionFieldas he suggested. #2802 stays open and will be narrowed toPartitionFieldonce this lands.What changes are included in this PR?
UnboundPartitionFieldgets the #3172 treatment — private fields with accessors — and with it the v3source-idsspelling.source_id() -> Result<i32>,source_ids() -> &[i32],field_id(),name()andtransform()accessors. The derivedTypedBuilderis kept but its visibility is limited, so an instance can only be built inside the crate or read out of a spec JSON.source_ids: Vec<i32>replacessource_id: i32and always holds at least one id.source_id()returns the single id, and an error for a multi-argument field rather than quietly handing back the first one._serdemodule reads either spelling —source-id, orsource-idsfor a v3 multi-argument transform — and writes back the one that matches the field. An emptysource-ids, a missing id, and asource-idthat disagrees with the firstsource-idsentry are all rejected.PartitionFieldstill carries a singlesource_id, so until that struct is converted there is nowhere to put the extra ids, and failing loudly beats dropping them on the way intoTableCreationorTableUpdate::AddSpec.check_for_redundant_partitionscompares the whole id list instead of one id.Three derived decisions worth your call:
pub(crate)rather than module-private, becausetable_metadata_builder,partitioningand the expression visitors all construct these. Say the word if you want it tighter and I will route those throughUnboundPartitionSpecBuilder.add_unbound_fieldandadd_unbound_fieldsstay public, but outside the crate they can now only be fed a field cloned out of an existing spec. Happy to make thempub(crate)if that is the intent.table_metadata_buildermutatedfield_idin place, which private fields no longer allow, so there is a crate-internalwith_field_id.Not in this PR, and not lost:
PartitionField(#2802, to be narrowed once this lands) andSortFieldget the same treatment, and with them the multi-argument read support and theTransform::Unknownmapping that #2801 is about.Are these changes tested?
Yes — four new unit tests in
spec/partition.rs: reading asource-ids-only field and writing it back, the single-id round trip, the three malformed-id rejections, and binding a multi-argument field failing loudly.cargo test -p iceberg --libpasses (1768 tests), withcargo fmt,cargo clippy --workspace --all-targets --all-features -- -D warningsand a full workspace build clean.crates/iceberg/public-api.txtis regenerated: four public fields and the publicbuilder()out, five accessors in.AI Disclosure
This pull request and its description were written by Isaac.