Conversation
CometDays cast timestamps to dates in the session timezone, while CometHours and Iceberg's days transform count in UTC. Spark cannot evaluate the transform, so there is no Spark result to match. Use UTC so the transforms agree with each other and with Iceberg. Closes apache#6333.
sunchao
left a comment
There was a problem hiding this comment.
Summary
- Prior state and problem:
CometDaysused the session timezone for timestamp inputs, producing day counts inconsistent withCometHoursand Iceberg near UTC day boundaries. - Design approach: Set the existing timestamp-to-date cast’s timezone to
UTC. - Correctness / compatibility analysis: No introduced P1/P2 issues found within this review. Spark sources for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 confirm that
Daysis unevaluable. The new behavior agrees with Iceberg’s UTC day-count semantics. Date inputs and supported types remain unchanged. - Key design decisions: Reusing the existing cast chain keeps the change simple. It introduces no additional native passes or abstractions. No performance benchmark was run.
- Implementation sketch: Replace the session timezone with
Some("UTC"), remove the unused import, and compare timestamp tests againstfloor(unix_micros(ts) / 86400000000D). - Behavioral changes worth calling out: In Los Angeles,
2024-07-01T03:00:00Znow produces day19905instead of19904. A bounded probe reproduced this correction and passed 60,108 cast results covering nulls, negative timestamps and midnight boundaries. - Suggested improvements: None at the P1/P2 threshold.
Reviewed the entire two-file diff from e1d2c11729c2fc60a5def4e87bb17e5b28df2a29 to de46799d3dd2ab229f2cd7756b42b81803ac47ed. The PR remained open and non-draft. Existing reviews, issue comments, inline comments and threads were empty. The linked issue describes the behavior corrected here.
Routed skills: review-comet-pr, review-comet-expression-pr, and review-comet-iceberg-write-pr for the transform-parity cross-check.
Exact-head CI at 2026-09-28 19:10 UTC: 21 successful checks, 43 skipped, and 2 running, with no reported failures. The Spark 4.1/JDK 17 build passed. Native build and Rust tests remained running. Spark SQL, Iceberg and macOS jobs were skipped.
Validation limits: The local probe used timezone helpers extracted from the reviewed source with the locked Arrow 59.3.0 cast implementation. It was not an end-to-end Comet execution. No full local Comet build, Scala suite, Spark SQL suite or Iceberg suite was run, and CI had not finished.
Which issue does this PR close?
Closes #6333.
Rationale for this change
Spark's
DaysandHourspartition transforms areUnevaluable: evaluating either one in a projection throwsPARTITION_TRANSFORM_EXPRESSION_NOT_IN_PARTITIONED_BY. Comet evaluates both natively, but it counted them differently:CometDayscast the timestamp to a date in the session timezone.CometHoursdivides the raw UTC microseconds.Iceberg's own
daysandhourstransforms are UTC-based, and so are Comet's native versions of them. So in a non-UTC session, Comet'sdaysgave a different day thanhoursand Iceberg imply for the same timestamp. For example, inAmerica/Los_Angeles,days(TIMESTAMP'2024-07-01T03:00:00Z')returned 19904 (2024-06-30) instead of 19905.What changes are included in this PR?
CometDayscasts a timestamp to a date in UTC instead of in the session timezone. Date inputs are unchanged.days - timestamp inputanddays - literal edge casestests inCometTemporalExpressionSuitenow compare against a UTC day count,floor(unix_micros(ts) / 86400000000), like thehourstest does. Before, they compared againstunix_date(CAST(ts AS DATE)), which depends on the session timezone.How are these changes tested?
days - timestamp inputfails against the new baseline in theAmerica/Los_AngelesandAsia/Tokyosessions it runs in.CometTemporalExpressionSuitepass on Spark 4.1, including thespotlessandscalastylechecks.