Skip to content

fix: warn when native and JVM timezone databases differ - #6353

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix-tzdata-version-warning
Open

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix-tzdata-version-warning

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #6331.

Rationale for this change

Spark converts between instants and local time using the JVM's timezone rules (tzdb.dat). The version of those rules depends on the JDK build and on whether TZUpdater has been run. Comet's native kernels use chrono-tz instead, which compiles its own copy of the IANA database into libcomet: chrono-tz 0.10.4 carries 2025b.

When the two versions disagree about a zone, every local-time operation on the affected timestamps returns a different answer, and nothing reports it. For example, with AppleJDK 17.0.10, which ships tzdata 2023c, hour(ts) for 2025-06-15T12:00:00Z in America/Asuncion is 8 in Spark and 9 in Comet.

Matching Spark in every deployment would mean making native code use the JVM's rules, which is a much larger change. This PR makes the mismatch visible instead.

What changes are included in this PR?

  • A JNI function, NativeBase.getTzdataVersion, returns the database version compiled into libcomet (chrono_tz::IANA_TZDB_VERSION). chrono-tz becomes a direct dependency of datafusion-comet. It was already in the dependency tree through arrow's chrono-tz feature, so the only Cargo.lock change is that one dependency line.
  • When the native library loads, NativeBase compares that version with the JVM's (ZoneRulesProvider.getVersions("UTC").lastKey()) and logs a warning if they differ. The warning names both versions.
  • A new "Timezone Database Versions" section in the datetime compatibility guide explains the difference and which operations it affects.
  • A test in CometTemporalExpressionSuite checks that the version is reported and that the warning is built.

How are these changes tested?

  • The new test and all 36 tests in CometTemporalExpressionSuite pass on Spark 4.1, including the spotless and scalastyle checks.
  • On this machine the suite's log shows the warning with the real versions: Comet's native library uses timezone database 2025b, but the JVM uses 2023c.
  • cargo clippy --all-targets -- -D warnings is clean for datafusion-comet.
  • prettier --check passes on the edited docs page.

Native code converts between instants and local time with the IANA database that chrono-tz compiles into libcomet, while Spark uses the JVM's tzdb.dat. When the versions differ, local times in timezones whose rules changed between them silently differ from Spark's. Expose the bundled version over JNI, log a warning when the library loads and the versions differ, and document the limitation.

Closes apache#6331.
@andygrove andygrove added backport-1.0 Candidate for backporting to 1.0 release branch backport-1.1 Candidate for backporting to 1.1 release branch labels Sep 28, 2026
@github-actions github-actions Bot added the bug Something isn't working label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-1.0 Candidate for backporting to 1.0 release branch backport-1.1 Candidate for backporting to 1.1 release branch bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native timezone rules come from chrono-tz's bundled tzdata, so results differ from Spark when the JVM's tzdata differs

1 participant