Conversation
Native code converts between instants and local time with the IANA database that chrono-tz compiles into libcomet, while Spark uses the JVM's tzdb.dat. When the versions differ, local times in timezones whose rules changed between them silently differ from Spark's. Expose the bundled version over JNI, log a warning when the library loads and the versions differ, and document the limitation. Closes apache#6331.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6331.
Rationale for this change
Spark converts between instants and local time using the JVM's timezone rules (
tzdb.dat). The version of those rules depends on the JDK build and on whether TZUpdater has been run. Comet's native kernels use chrono-tz instead, which compiles its own copy of the IANA database into libcomet: chrono-tz 0.10.4 carries 2025b.When the two versions disagree about a zone, every local-time operation on the affected timestamps returns a different answer, and nothing reports it. For example, with AppleJDK 17.0.10, which ships tzdata 2023c,
hour(ts)for2025-06-15T12:00:00ZinAmerica/Asuncionis 8 in Spark and 9 in Comet.Matching Spark in every deployment would mean making native code use the JVM's rules, which is a much larger change. This PR makes the mismatch visible instead.
What changes are included in this PR?
NativeBase.getTzdataVersion, returns the database version compiled into libcomet (chrono_tz::IANA_TZDB_VERSION).chrono-tzbecomes a direct dependency ofdatafusion-comet. It was already in the dependency tree through arrow'schrono-tzfeature, so the onlyCargo.lockchange is that one dependency line.NativeBasecompares that version with the JVM's (ZoneRulesProvider.getVersions("UTC").lastKey()) and logs a warning if they differ. The warning names both versions.CometTemporalExpressionSuitechecks that the version is reported and that the warning is built.How are these changes tested?
CometTemporalExpressionSuitepass on Spark 4.1, including thespotlessandscalastylechecks.Comet's native library uses timezone database 2025b, but the JVM uses 2023c.cargo clippy --all-targets -- -D warningsis clean fordatafusion-comet.prettier --checkpasses on the edited docs page.