[SPARK-58389][SQL] Pass all options while loading tables - #57582
Open
yyanyy wants to merge 1 commit into
Open
Conversation
### What changes were proposed in this pull request? This is a follow-up to apache#56044 (which passed all options while loading changelogs). It does the same for table reads by adding `TableCatalog.loadTable(Identifier, TableContext, CaseInsensitiveStringMap)`, where `TableContext` carries the parsed, Spark-recognized parameters (time travel, write privileges) and the `CaseInsensitiveStringMap` carries all raw user options. - New public connector types `TableContext` and `TimeTravel` (a clean sealed interface with `Version`/`Timestamp` records), mirroring how apache#56044 introduced `ChangelogContext`/`ChangelogRange` rather than leaking the catalyst-internal `TimeTravelSpec`. - The new `loadTable` overload has a default implementation that delegates to the existing `loadTable` overloads based on `TableContext`, so existing connectors keep working unchanged. - `CatalogV2Util.getTable`/`loadTable` now build a `TableContext` from the catalyst `TimeTravelSpec` + write-privileges string and forward the user options, making the Java default the single dispatch site. - Options are threaded through the read paths in `RelationResolution` and `DataSourceV2Utils`. This PR does not touch the `RelationCatalog` single-RPC `loadRelation(Identifier)` read path, which for a table-and-view catalog is the primary path for a plain read (no time travel / write privileges) and so does not forward options. That is an independent improvement -- it needs nothing from this change (a new `loadRelation(Identifier, CaseInsensitiveStringMap)` overload plus wiring) -- and will be a separate PR. ### Why are the changes needed? To make the API usable in connectors like Iceberg and Delta, which need the user options while reading a table. The functionality hasn't been released yet. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests in `CatalogV2UtilSuite` (the default-dispatch mapping for each `TableContext` shape, and the time-travel/write-privileges mutual-exclusion invariant) and `DataSourceV2OptionSuite` (end-to-end option forwarding via the DataFrame API, SQL, streaming, time travel, and the write path). Existing `SupportsCatalogOptionsSuite`, `ChangelogResolutionSuite`, `ChangelogEndToEndSuite`, and the DataSourceV2 SQL/DataFrame suites pass.
yyanyy
force-pushed
the
spark-dsv2-loadtable-options-20260727
branch
from
July 27, 2026 21:21
b86022f to
e1e6c66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a follow-up to #56044 (which passed all options while loading changelogs). It does the same for table reads by adding
TableCatalog.loadTable(Identifier, TableContext, CaseInsensitiveStringMap), whereTableContextcarries the parsed, Spark-recognized parameters (time travel, write privileges) and theCaseInsensitiveStringMapcarries all raw user options.TableContextandTimeTravel(a clean sealed interface withVersion/Timestamprecords), mirroring how [SPARK-56961][SQL] Pass all options while loading changelog #56044 introducedChangelogContext/ChangelogRangerather than leaking the catalyst-internalTimeTravelSpec.loadTableoverload has a default implementation that delegates to the existingloadTableoverloads based onTableContext, so existing connectors keep working unchanged.CatalogV2Util.getTable/loadTablenow build aTableContextfrom the catalystTimeTravelSpec+ write-privileges string and forward the user options, making the Java default the single dispatch site.RelationResolutionandDataSourceV2Utils.Potential follow-up (intentionally out of scope here): the
RelationCatalogsingle-RPCloadRelation(Identifier)path is not updated to forward options. It applies only to catalogs that expose both tables and views, and fires only when there is no time travel or write privileges; closing that gap cleanly would add aloadRelation(Identifier, CaseInsensitiveStringMap)overload and is better done as a separate change.Why are the changes needed?
To make the API usable in connectors like Iceberg and Delta, which need the user options while reading a table. The functionality hasn't been released yet.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
New tests in
CatalogV2UtilSuite(the default-dispatch mapping for eachTableContextshape, and the time-travel/write-privileges mutual-exclusion invariant) andDataSourceV2OptionSuite(end-to-end option forwarding via the DataFrame API, SQL, streaming, time travel, and the write path). ExistingSupportsCatalogOptionsSuite,ChangelogResolutionSuite,ChangelogEndToEndSuite, and the DataSourceV2 SQL/DataFrame suites pass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude code Opus 4.8