Skip to content

fix: count the days partition transform in UTC - #6348

Queued
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix-days-transform-utc
Queued

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix-days-transform-utc

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #6333.

Rationale for this change

Spark's Days and Hours partition transforms are Unevaluable: evaluating either one in a projection throws PARTITION_TRANSFORM_EXPRESSION_NOT_IN_PARTITIONED_BY. Comet evaluates both natively, but it counted them differently:

  • CometDays cast the timestamp to a date in the session timezone.
  • CometHours divides the raw UTC microseconds.

Iceberg's own days and hours transforms are UTC-based, and so are Comet's native versions of them. So in a non-UTC session, Comet's days gave a different day than hours and Iceberg imply for the same timestamp. For example, in America/Los_Angeles, days(TIMESTAMP'2024-07-01T03:00:00Z') returned 19904 (2024-06-30) instead of 19905.

What changes are included in this PR?

  • CometDays casts a timestamp to a date in UTC instead of in the session timezone. Date inputs are unchanged.
  • The days - timestamp input and days - literal edge cases tests in CometTemporalExpressionSuite now compare against a UTC day count, floor(unix_micros(ts) / 86400000000), like the hours test does. Before, they compared against unix_date(CAST(ts AS DATE)), which depends on the session timezone.

How are these changes tested?

  • Old serde, updated tests. With the old serde, days - timestamp input fails against the new baseline in the America/Los_Angeles and Asia/Tokyo sessions it runs in.
  • With this change. All 35 tests in CometTemporalExpressionSuite pass on Spark 4.1, including the spotless and scalastyle checks.

CometDays cast timestamps to dates in the session timezone, while CometHours and Iceberg's days transform count in UTC. Spark cannot evaluate the transform, so there is no Spark result to match. Use UTC so the transforms agree with each other and with Iceberg.

Closes apache#6333.
@andygrove andygrove added backport-1.0 Candidate for backporting to 1.0 release branch backport-1.1 Candidate for backporting to 1.1 release branch labels Sep 28, 2026
@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Sep 28, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

  • Prior state and problem: CometDays used the session timezone for timestamp inputs, producing day counts inconsistent with CometHours and Iceberg near UTC day boundaries.
  • Design approach: Set the existing timestamp-to-date cast’s timezone to UTC.
  • Correctness / compatibility analysis: No introduced P1/P2 issues found within this review. Spark sources for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 confirm that Days is unevaluable. The new behavior agrees with Iceberg’s UTC day-count semantics. Date inputs and supported types remain unchanged.
  • Key design decisions: Reusing the existing cast chain keeps the change simple. It introduces no additional native passes or abstractions. No performance benchmark was run.
  • Implementation sketch: Replace the session timezone with Some("UTC"), remove the unused import, and compare timestamp tests against floor(unix_micros(ts) / 86400000000D).
  • Behavioral changes worth calling out: In Los Angeles, 2024-07-01T03:00:00Z now produces day 19905 instead of 19904. A bounded probe reproduced this correction and passed 60,108 cast results covering nulls, negative timestamps and midnight boundaries.
  • Suggested improvements: None at the P1/P2 threshold.

Reviewed the entire two-file diff from e1d2c11729c2fc60a5def4e87bb17e5b28df2a29 to de46799d3dd2ab229f2cd7756b42b81803ac47ed. The PR remained open and non-draft. Existing reviews, issue comments, inline comments and threads were empty. The linked issue describes the behavior corrected here.

Routed skills: review-comet-pr, review-comet-expression-pr, and review-comet-iceberg-write-pr for the transform-parity cross-check.

Exact-head CI at 2026-09-28 19:10 UTC: 21 successful checks, 43 skipped, and 2 running, with no reported failures. The Spark 4.1/JDK 17 build passed. Native build and Rust tests remained running. Spark SQL, Iceberg and macOS jobs were skipped.

Validation limits: The local probe used timezone helpers extracted from the reviewed source with the locked Arrow 59.3.0 cast implementation. It was not an end-to-end Comet execution. No full local Comet build, Scala suite, Spark SQL suite or Iceberg suite was run, and CI had not finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation backport-1.0 Candidate for backporting to 1.0 release branch backport-1.1 Candidate for backporting to 1.1 release branch bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

days transform is evaluated in the session timezone, while hours and Iceberg use UTC

2 participants