[MINOR] Clean up residual Spark 3.3 references - #13089
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
No unresolved issues were identified.
Review effort: Lite
Findings: None
What changed in this PR
Cleans up remaining Spark 3.3 references by updating benchmark provisioning to Spark 3.5.5 and clarifying Spark 3.4 compatibility documentation.
Changes:
- Updates Spark, Py4J, and YARN shuffle artifact versions.
- Removes the obsolete Spark 3.3 reference.
| File | Description |
|---|---|
tools/workload/benchmark_velox/initialize.ipynb |
Updates Spark benchmark cluster provisioning versions. |
gluten-substrait/src/main/scala/org/apache/gluten/execution/WriteFilesExecTransformer.scala |
Corrects the Spark compatibility comment. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Run Gluten Clickhouse CI on x86 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Two Spark 3.3 references that the earlier cleanup (#13003) missed, both a straightforward version/text update.
tools/workload/benchmark_velox/initialize.ipynbprovisioned its benchmark cluster with Spark 3.3.1 (wgetofspark-3.3.1-bin-hadoop3.tgz, the unpack/symlink, and the yarn-shuffle jar). Following the notebook end to end produced a Spark that Gluten no longer runs on. Bumped to 3.5.5, the well-tested supported version and the repo's defaultsparkbundle.version, and bumped the py4j onPYTHONPATHfrom0.10.9.5to0.10.9.7to match (Spark 3.4 upgraded py4j; 3.5.x ships0.10.9.7).WriteFilesExecTransformercarried a// To be compatible with Spark 3.3/3.4comment; dropped the3.3, since 3.4 is now the minimum. The line right below it already notes Spark 3.5+ strips these fields via SPARK-43123, so the comment now reads 3.4-only, which is exactly the range that still needs the manual cleanup.The other remaining Spark 3.2/3.3 references (the bolt
Makefiledead targets and an apparently-unused ClickHouse test fixture) need an owner decision and are filed as #13088 instead.How was this patch tested?
The comment change is comment-only. The notebook change is a provisioning version-string bump that I could not execute (it drives a multi-node cluster over ssh);
spark-3.5.5-bin-hadoop3.tgzandspark-3.5.5-yarn-shuffle.jarare published on the Apache archive, and the notebook JSON still parses.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5
Related issue: #13003