Conversation
difin
changed the title
7e54757 to
769a954
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
|
There were already discussions happening under HIVE-29599 for server side planning? |
Co-authored-by: Cursor <cursoragent@cursor.com>
769a954 to
3ef3d65
Compare
This is different - this PR is about adding client-side support. |
difin
changed the title
tanishq-chugh
left a comment
tanishq-chugh
left a comment
There was a problem hiding this comment.
Hi @difin
Checked out the PR, great improvement 🚀
| */ | ||
| public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) { | ||
| if (shouldReloadForServerSideScanPlanning(conf)) { | ||
| Table table = Catalogs.loadTable(conf); |
There was a problem hiding this comment.
I believe this will create a new RestCatalog Object/HttpClient on every call, curious if this can cause any connection leak?
There was a problem hiding this comment.
Yes - Catalogs.loadTable creates a new REST catalog (and HTTP client) on each call, and we don’t close it on this path today. That’s the same pattern as other Catalogs.loadTable usages in the handler. Here it runs once per getSplits during split planning, not per input split. A reload only happens when shouldReloadForServerSideScanPlanning is true.
|
This branch has not been deployed
4 participants
What changes were proposed in this pull request?
Hive wires up Iceberg REST server-side scan planning: when
scan-planning-mode=server, split planning can go through the REST catalog (POST /plan) instead of only the serialized table snapshot on Tez/MR workers. Adds a smallRestCatalogScanPlanninghelper for config and copying catalog settings into job conf, and updates the Iceberg storage handler / input path to reload a live REST table on executors where appropriate. Tests cover configuration, job propagation, and embedded REST planning behavior.Why are the changes needed?
Iceberg REST catalogs can plan scans on the catalog service instead of on every client. That keeps planning close to authoritative metadata and catalog credentials, can reduce work on HiveServer2 and task containers, and lets the REST service apply catalog-side optimizations as Iceberg evolves.
Hive today mostly plans reads from a serialized table snapshot shipped in the job configuration, which fits local/Hadoop catalogs but does not use the REST server’s planning API. For deployments that already run a REST catalog with scan planning, operators expect
scan-planning-mode=serverto work for Hive queries on Tez/MR - not only for single-node Iceberg clients. This change closes that gap so Hive participates in the same server-side planning model as the rest of the Iceberg REST ecosystem.Does this PR introduce any user-facing change?
Yes. For REST Iceberg catalogs, operators can set
iceberg.catalog.<catalog>.scan-planning-mode=server(viahive-site.xmlor session SET) alongside existing REST catalog properties. When enabled, read jobs that use the Iceberg input format can use server-side scan planning instead of local planning from the serialized snapshot. Default behavior is unchanged when the property is unset or not server. Hive and non-REST catalogs are unaffected.How was this patch tested?
TestRestCatalogScanPlanning— Tests config keys, server-mode detection, job propagation gating and copying.TestRestCatalogScanPlanningServerIT— Uses embedded REST catalog; Verifies thatRESTTable/RESTTableScanandPlanTableScanRequest on planTasks()are used when file planning is on.TestHiveIcebergServerSideScanPlanning— Checks that Hive only reloads a table from the REST catalog for split planning when server mode is on and it is safe to do so - not when planning stays local or when the job must use uncommitted metadata from the same transaction.TestHiveIcebergServerSideScanPlanningServerIT— Tests that executor job conf with/without propagated catalog props and table reload over REST.