fix: normalize session timezone IDs before passing them to native code

[andygrove]

Which issue does this PR close?

Closes #6329.

Rationale for this change

Comet passed the session timezone to native code as the raw string stamped on each expression. Native code parses that string with arrow's Tz::from_str, which only accepts:

• IANA zone names
• offsets written as +HH, +HHMM or +HH:MM

Spark resolves the ID with ZoneId.of(id, ZoneId.SHORT_IDS), which also accepts:

• Z
• offsets such as +8 and +08:00:00
• prefixed offsets such as GMT+8 and UTC+08:00
• short IDs such as PST and IST

The spark.sql.session.timeZone documentation lists Z and (+|-)HH:mm:ss explicitly. With any of these IDs, most timestamp expressions failed at execution time with Parser error: Invalid timezone "GMT+8". That covered CAST(ts AS STRING/DATE), hour, year, string- and date-to-timestamp casts, unix_timestamp of a date, casts between TIMESTAMP and TIMESTAMP_NTZ, and df.show(). GMT+8 in particular is a common production setting.

What changes are included in this PR?

• A new CometTimeZone.nativeId normalizes the stamped timezone into an ID native code parses. It uses Spark's own DateTimeUtils.getZoneId, then ZoneId.normalized():
  • fixed offsets become +HH:MM
  • zero offsets become UTC
  • short IDs become their region
  • an expression with no timezone gets UTC, since Spark only leaves it unset on casts that don't use one (the measurement is in [EPIC] Timezone handling bugs #6335)
• An offset with seconds, such as +05:45:30, can't be written for native code. nativeId returns None for it, and the serde reports the expression as unsupported. The cast then goes through the codegen dispatcher, and expressions without a dispatcher path fall back.
• Every serde that sends a timezone to native code now goes through the helper. That covers Cast, hour, minute, second, unix_timestamp, date_trunc, from_unixtime, to_json, from_json, to_csv, ToPrettyString (both shims) and the native Parquet scan's session timezone. These were the getOrElse("UTC") sites that Review use of hard-coded UTC assumptions #2730 asked about.
• date_trunc and date_format decide "is this session UTC" through the same helper. GMT, Z and +00:00 sessions now take their native UTC paths as well.
• CometDays is left alone, because fix: count the days partition transform in UTC #6348 moves it to UTC.

How are these changes tested?

• New tests:
  • A unit test in CometTemporalExpressionSuite for the normalization.
  • session_timezone_ids.sql, which runs casts, field extraction, unix_timestamp, TIMESTAMP_NTZ casts, date_trunc and date_format under GMT+8, UTC+08:00, +8, -08, +08:00:00, Z, PST and IST.
  • session_timezone_ids_unsupported.sql, which covers +05:45:30.
• Before the change: with main's serdes, session_timezone_ids.sql fails for every timezone except -08, which arrow already parses. The unsupported file fails too.
• With the change on Spark 4.1: all 200 tests in these suites pass, with the spotless and scalastyle checks included:
  • CometTemporalExpressionSuite
  • every expressions/datetime/ SQL file test
  • CometJsonExpressionSuite
  • CometCsvExpressionSuite
• Casts: all 185 CometNativeCastSuite tests pass on Spark 4.1.
• Spark 3.5: the new tests pass on Spark 3.5 as well, which also compiles the 3.5 ToPrettyString shim.

[sunchao]

Summary

• Prior state and problem: Raw Spark session timezone IDs such as GMT+8, Z, and PST could fail native parsing.
• Design approach: Centralize normalization in CometTimeZone, using Spark’s getZoneId and Java’s normalized().
• Correctness / compatibility analysis: Checked Spark sources across 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0. Normalization preserves timezone rules, including DST. Reviewed unsupported-offset guards and existing fallback paths.
• Key design decisions: Reuse Spark’s resolver, preserve region rules, and reject fixed offsets containing nonzero seconds. The helper adds planning-time work without adding per-row normalization or a new dependency.
• Implementation sketch: Update cast, temporal, JSON/CSV, pretty-string, and native-scan serialization. Add normalization and SQL regression tests.
• Behavioral changes worth calling out: Accepted aliases reach native code in parseable form. Fixed-zero aliases also qualify for existing UTC execution paths.
• Suggested improvements: None at P1/P2. No introduced P1/P2 issues found within this review.

Reviewed the full 11-file diff from e1d2c11729c2fc60a5def4e87bb17e5b28df2a29 to c2aa0b427c20bb6396da025e3396051c52fce050. The PR is not a draft. Routed skills: review-comet-pr and review-comet-expression-pr. Existing discussion collections were empty.

Exact-head CI: Linux builds, lint, Rust tests, scans, execution, shuffle, and TPC-H/TPC-DS passed. The expression job passed 1,554 tests, including both new SQL fixtures and the normalization test, with one canceled and 12 ignored tests.

Local validation: Compiled the production helper in an isolated Spark 4.1.3 probe with minimal support-level test types. Checked 788 timezone IDs and verified all 88 rewritten IDs against Arrow 59.3.0 at eight historical, DST-boundary, and future instants. Unsupported-offset checks passed.

Validation limits: No full local Comet build or end-to-end suite was run. Spark’s own SQL suites, additional runtime profiles, macOS, and Iceberg suites were skipped in exact-head CI. Project files remain unchanged.