[SPARK-57534][PYSPARK] Expose multi-argument Oracle-style decode() in PySpark - #57568
Open
ybapat wants to merge 1 commit into
Open
[SPARK-57534][PYSPARK] Expose multi-argument Oracle-style decode() in PySpark#57568ybapat wants to merge 1 commit into
ybapat wants to merge 1 commit into
Conversation
Contributor
|
Spark-57534 is a task for this PR Isn't it? |
…EN form The PySpark decode() function previously only exposed the 2-argument binary decode form. Spark SQL has supported both the 2-arg binary form and the Oracle-style CASE WHEN multi-argument form (decode(expr, search1, result1, ..., default)) since 3.2.0. This change updates the Python API to pass through variable arguments so both forms work from PySpark. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ybapat
force-pushed
the
issue-57534-pyspark-decode
branch
from
July 27, 2026 20:27
3a4567e to
299e8ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The SQL
decodefunction in Spark supports two forms:decode(binary_col, charset)— decodes binary data to a string.decode(expr, search1, result1, ..., [default])— returns the result for the first matching search value.The CASE WHEN form is implemented in Scala (
object Decode { def createExpr(...) }) and available viaspark.sql("SELECT decode(...)"), but PySpark'sdecode()function only exposed the 2-argument binary form. Issue #57534 tracks this gap.This PR updates
pyspark.sql.functions.decode(both classic and Connect paths) to accept variadic*argsand dispatch to the correct form:PySparkTypeErrorwithNOT_ENOUGH_ARGSChanges:
python/pyspark/sql/functions/builtin.py: Change signature todecode(col, *args), add 3-branch dispatch, update docstring with examples.python/pyspark/sql/connect/functions/builtin.py: Mirror changes for Spark Connect path.python/pyspark/errors/error-conditions.json: AddNOT_ENOUGH_ARGSerror class.Why are the changes needed?
PySpark users cannot use the Oracle-style
decodevia the Python API without resorting tospark.sql(). This change closes the gap between the SQL and Python APIs.Does this PR introduce any user-facing change?
Yes — new functionality. The existing 2-argument binary decode is fully backward compatible. Users can now call:
How was this patch tested?
Manual verification of both dispatch paths. The existing doctest for binary decode continues to pass. The CASE WHEN examples are marked
# doctest: +SKIPas they require a live Spark session.Closes #57534
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com