Repository navigation
fix: accept up to 9 fractional digits for micros timestamps - #4142
Open
Shahzeb-DataEngineer wants to merge 1 commit into
Open
Shahzeb-DataEngineer wants to merge 1 commit into
Shahzeb-DataEngineer wants to merge 1 commit into
Conversation
Widen ISO_TIMESTAMP and ISO_TIMESTAMPTZ to accept 1-9 fractional digits and zone offsets with optional seconds, and parse micros timestamps through the nanosecond path, truncating to microseconds, mirroring the Java implementation. Previously, valid ISO-8601 strings with nanosecond precision were rejected with a ValueError when binding string literals to timestamp/timestamptz columns or parsing JSON single values. Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4141
Rationale for this change
timestamp_to_microsandtimestamptz_to_microsrejected valid ISO-8601 strings with 7-9 fractional second digits becauseISO_TIMESTAMP/ISO_TIMESTAMPTZcapped the fraction at 6 digits, andISO_TIMESTAMPTZdid not accept zone offsets with seconds. These converters are used when binding string literals totimestamp/timestamptzcolumns and when parsing JSON single values (partition values and initial defaults), so timestamps written by other engines with nanosecond precision failed to load or filter.The Java implementation accepts 0-9 fractional digits (
appendFraction(NANO_OF_SECOND, 0, 9, true)) and truncates to the target precision. This change mirrors that behavior:ISO_TIMESTAMPandISO_TIMESTAMPTZto accept 1-9 fractional digits, and allow optional seconds in the zone offset (also inISO_TIMESTAMPTZ_NANO).nanos_to_micros, so the sub-microsecond handling stays in one place. Nanosecond types keep full precision.Are these changes tested?
Yes. Added parametrized cases for 7, 8, and 9 fractional digits for both
timestamp_to_microsandtimestamptz_to_microsintests/utils/test_datetime.py, including the offset-with-seconds form and the existing specific error messages, plus a JSON single-value deserialization test with 9-digit fractions intests/test_conversions.py. Rantests/utils/test_datetime.py,tests/test_conversions.py, andtests/expressions/test_literals.pylocally; all pass (283 + 172 tests).Are there any user-facing changes?
Yes: strings that were previously rejected with
ValueErrorare now accepted and truncated to microsecond precision, matching the Java implementation. No previously valid input changes meaning.