[SPARK-58372][SQL] Truncate fractional to integral casts pushed down to JDBC sources by marko-sisovic-db · Pull Request #57564 · apache/spark

marko-sisovic-db · 2026-07-27T10:27:43Z

What changes were proposed in this pull request?

Spark truncates toward zero when casting a fractional value to an integral type, but several databases round half away from zero. When such a cast is pushed down, the query returns a different result than Spark would produce locally.

This PR adds a JdbcDialect.truncateFractionalValue hook that is applied once in JDBCSQLBuilder.visitCast, so the pushed down cast truncates the value first:

// JdbcDialect
def truncateFractionalValue(expr: String): String = expr

Dialects for databases that round override it with the database's truncating function:

Dialect	Pushed down SQL for `CAST(c AS INT)`
MySQL	`CAST(TRUNCATE(c, 0) AS ...)` (MySQL has no single argument `TRUNCATE`)
Oracle	`CAST(TRUNC(c) AS ...)`
Postgres	`CAST(TRUNC(c) AS ...)`
Snowflake	`CAST(TRUNC(c) AS ...)`

The default returns the expression unchanged, so dialects whose cast already truncates like Spark (MS SQL Server, DB2, Derby, H2, Teradata) are unaffected, as are dialects for databases that have no single argument truncating function.

spark.sql.legacy.jdbc.roundIntegralCastPushdown.enabled restores the previous behavior.

Why are the changes needed?

The pushed down cast silently returns wrong results - there is no error, just different values than Spark computes. For a table with the values 1.5, 2.5 and -1.5:

SELECT CAST(c AS INT) FROM t

Spark returns 1, 2, -1 (truncating toward zero), while MySQL, Oracle, Postgres and Snowflake return 2, 3, -2 (rounding half away from zero). The same divergence affects filters, where rows are then filtered out on the database side, e.g. WHERE CAST(c AS INT) = 2 matches a different row.

Does this PR introduce any user-facing change?

Yes. A fractional to integral cast that is pushed down to MySQL, Oracle, Postgres or Snowflake now returns the same result as Spark evaluating it locally (truncated toward zero) instead of the database's rounded result. Set spark.sql.legacy.jdbc.roundIntegralCastPushdown.enabled to true to restore the previous behavior.

How was this patch tested?

New test in V2JDBCTest, so it runs for every JDBC docker integration suite that mixes in the trait (MySQL, Postgres, Oracle, DB2, MS SQL Server). It covers a DOUBLE and a DECIMAL source, a projected cast and a pushed down filter, and includes a negative value, where truncating and rounding also differ.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

…to JDBC sources Spark truncates toward zero when casting a fractional value to an integral type, but MySQL, Oracle, Postgres and Snowflake round half away from zero. A pushed down cast therefore returns different results than Spark evaluating it locally, silently, with no error. Wrap such casts in the dialect's truncating function via a new JdbcDialect.truncateFractionalValue hook, applied once in JDBCSQLBuilder.visitCast. The default returns the expression unchanged, so dialects whose cast already truncates (or that have no truncating function) are unaffected. spark.sql.legacy.jdbc.roundIntegralCastPushdown.enabled restores the previous behavior.

urosstan-db · 2026-07-27T13:22:37Z

+      val castedExpr = if (exprDataType.isInstanceOf[FractionalType] &&
+        dataType.isInstanceOf[IntegralType] &&
+        !SQLConf.get.getConf(SQLConf.LEGACY_JDBC_ROUND_INTEGRAL_CAST_PUSHDOWN)) {
+        truncateFractionalValue(expr)


nit: Do we need to fix this on v2 expression builder level? Other data sources may benefit of stricter API for this truncation fix.

urosstan-db · 2026-07-27T13:27:42Z

    }
  }

+  test("fractional to integral cast pushed down truncates toward zero like Spark") {


Can we add test to verify legacy behaviour, conf offf

urosstan-db · 2026-07-27T13:29:21Z

+   * @param expr The SQL string of the value being cast.
+   * @return The SQL string to cast instead of `expr`.
+   */
+  def truncateFractionalValue(expr: String): String = expr


Nice job on having default value, it would be better to be strict here for third-party dialects, but that would break compilation on spark upgrade, so I think it is better to preserve this default implementation. What is the policy, is robustness more important than avoiding of breaking changes or vice versa @cloud-fan @uros-b ?

I would advise to avoid breaking change. In any case, please add the @Since annotation here (like every sibling public method has one).

JdbcDialect is @DeveloperApi, so a breaking change is technically permitted, but the established practice for this trait is to add new hooks as concrete methods with a safe default (e.g. see getFetchSize, isSyntaxErrorBestEffort, isObjectNotFoundException, isNotSelectableObjectException). So yeah, keep the deafult and just add @Since("4.3.0") if you're targeting the last 4.x version (or 5.0.0 if you're targeting master only).

uros-b · 2026-07-27T13:49:24Z

+      }
+    }
+  }
+


Also, IIUC - no test runs in the default PR CI. The only test lives in V2JDBCTest (docker integration suites), which is not in the default check set, and there is no unit test asserting the generated SQL string per dialect (e.g. a compileExpression assertion in JDBCSuite, as done for other pushdowns). So, Snowflake's TRUNC override is unexercised. If this is the case, please add a cheap per-dialect unit test to guard against silent regression and properly cover Snowflake.

uros-b · 2026-07-27T13:51:08Z

+      // rounds half away from zero would return 2, 3 and -2 instead.
+      Seq("double_col", "decimal_col").foreach { col =>
+        val projected = sql(s"SELECT CAST($col AS INT) FROM $tbl ORDER BY $col")
+        assert(projected.collect().map(_.getInt(0)) === Array(-1, 1, 2))


This assertion does not seem to exercise the change. JDBCScanBuilder pushes down only column pruning, filters, aggregates, limit/topN and joins, but never a projection expression. So, this test case in a SELECT list is evaluated locally by Spark and passes regardless of the fix.

Excellent comment,
Since we may support projection pushdown in the future (but I doubt that would happen TBH), maybe it's good to preserve this check and also to add aggregation query, since it is pushed, so we will cover filter, aggregates and be future-proof for projections as well?
wdyt @marko-sisovic-db @uros-b

uros-b

The fix is generally correct and well scoped, but please see the outstanding comments and make sure to add a proper [SPARK] Jira ticket. Also cc @cloud-fan who has more context in V2 expression pushdown & JDBC dialect API.

MySQL has no INTEGER cast target, so the base CAST(... AS INTEGER) is a syntax error there. It was never hit before because no test pushed an integral cast to MySQL. Override visitCast in MySQLSQLBuilder to render SIGNED, composing with truncateFractionalValue. Also read the new conf through a SQLConf accessor, like the other legacy JDBC confs.

marko-sisovic-db changed the title ~~[SPARK-XXXXX][SQL] Truncate fractional to integral casts pushed down to JDBC sources~~ [SQL] Truncate fractional to integral casts pushed down to JDBC sources Jul 27, 2026

marko-sisovic-db force-pushed the msisovic/jdbc-truncate-integral-cast-pushdown branch from 15d5bd5 to 9637585 Compare July 27, 2026 10:56

urosstan-db approved these changes Jul 27, 2026

View reviewed changes

urosstan-db reviewed Jul 27, 2026

View reviewed changes

uros-b reviewed Jul 27, 2026

View reviewed changes

marko-sisovic-db changed the title ~~[SQL] Truncate fractional to integral casts pushed down to JDBC sources~~ [SPARK-58372][SQL] Truncate fractional to integral casts pushed down to JDBC sources Jul 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-58372][SQL] Truncate fractional to integral casts pushed down to JDBC sources - #57564

[SPARK-58372][SQL] Truncate fractional to integral casts pushed down to JDBC sources#57564
marko-sisovic-db wants to merge 2 commits into
apache:masterfrom
marko-sisovic-db:msisovic/jdbc-truncate-integral-cast-pushdown

marko-sisovic-db commented Jul 27, 2026 •

edited

Loading

Uh oh!

urosstan-db Jul 27, 2026

Uh oh!

urosstan-db Jul 27, 2026

Uh oh!

uros-b Jul 27, 2026

Uh oh!

urosstan-db Jul 27, 2026

Uh oh!

uros-b Jul 27, 2026 •

edited

Loading

Uh oh!

uros-b Jul 27, 2026 •

edited

Loading

Uh oh!

uros-b Jul 27, 2026

Uh oh!

uros-b Jul 27, 2026

Uh oh!

urosstan-db Jul 27, 2026

Uh oh!

uros-b left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

marko-sisovic-db commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

urosstan-db Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

urosstan-db Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

urosstan-db Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uros-b Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uros-b Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

urosstan-db Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marko-sisovic-db commented Jul 27, 2026 •

edited

Loading

uros-b Jul 27, 2026 •

edited

Loading

uros-b Jul 27, 2026 •

edited

Loading