[WIP][SPARK-59685][SQL] Keep clause-free SQL/JSON constructor canonical SQL bound to the built-in - #58938
Open
ganeshashree wants to merge 1 commit into
Open
[WIP][SPARK-59685][SQL] Keep clause-free SQL/JSON constructor canonical SQL bound to the built-in#58938ganeshashree wants to merge 1 commit into
ganeshashree wants to merge 1 commit into
Conversation
…L bound to the built-in A clause-free JSON_VALUE / JSON_QUERY / JSON_EXISTS / JSON_ARRAY call routes through function resolution (SPARK-59144), but these built-ins' canonical `sql` drops default clauses, so a call that is the built-in only because of a default clause renders clause-free SQL that, reparsed under a shadowing PATH, binds a same-named routine instead of the built-in. Render the default clause (RETURNING STRING, or FALSE ON ERROR for JSON_EXISTS) so canonical SQL reparses back to the built-in; `usePrettyExpression` renders the clean form so auto-generated column names are unaffected. Generated-by: Claude Code (Opus 4.8) Co-authored-by: Isaac <no-reply@databricks.com>
ganeshashree
force-pushed
the
SPARK-59685
branch
from
September 21, 2026 08:59
3de9c17 to
689711f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Since SPARK-59144, a clause-free SQL/JSON function call routes through function resolution, so a same-named routine on the SQL PATH can shadow the built-in. However these built-ins' canonical
sqlomits default clauses, so a call that is a built-in only because of a default clause renders as clause-free SQL; reparsing that under a shadowing PATH binds the routine instead of the built-in.This PR renders the default clause (
RETURNING STRING, orFALSE ON ERRORforJSON_EXISTS) in canonicalsqlso the rendering reparses back to the built-in. A sharedRoutedSqlJsonExpressiontrait letsusePrettyExpressionkeep rendering the clean clause-free form for auto-generated column names (which are never reparsed), so display names stay unaffected.Why are the changes needed?
Without this, a canonical-SQL round-trip (e.g. persisting and reloading a view) can silently rebind a built-in SQL/JSON function to a user routine that shadows it on the SQL PATH, changing results.
Does this PR introduce any user-facing change?
Yes, within unreleased master only. Canonical SQL of a clause-free SQL/JSON constructor now includes its default clause. Auto-generated top-level column names are unchanged; a nested constructor's generated name now shows
FORMAT JSON, e.g.JSON_ARRAY(JSON_ARRAY(1, 2) FORMAT JSON, 3).How was this patch tested?
New unit tests in
JsonValueSuite/JsonQuerySuite/JsonExistsSuite/JsonArraySuite: each asserts that a default-clause constructor's canonical SQL reparses to the built-in under a shadowing PATH, and that its auto-generated column name stays clean. Golden files were regenerated.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)