Describe the bug
Partition values for dates and timestamps outside the range chrono and arrow can represent (beyond roughly year 262143) are handled inconsistently by the native Iceberg writer:
day and hour transforms, and identity on a date column, fall through iceberg_partition_path.rs:199 into iceberg-rust's human-string rendering, which panics on the overflow. The panic is caught at the JNI boundary, so the task fails and the abort guard cleans up, but the write fails where iceberg-java succeeds.
year and month transforms go through arrow's date_part, which returns NULL for values it cannot represent. Those rows get a NULL partition value, where iceberg-java computes the real one. That is silent wrong metadata.
Only reachable with extreme values such as date_from_unix_date(100000000) or timestamp_micros(9000000000000000000), so low priority. Found by reading the code; not reproduced.
Steps to reproduce
Write rows whose date or timestamp partition source value lies beyond year 262143 into tables partitioned by days(ts), hours(ts), years(ts) and months(ts) with spark.comet.iceberg.write.enabled=true.
Expected behavior
Either the same partition values as iceberg-java, or a clean fallback. At minimum the year/month case should not write a NULL partition value.
Additional context
Found in an audit of the native Iceberg write path. Part of #5649.
Describe the bug
Partition values for dates and timestamps outside the range chrono and arrow can represent (beyond roughly year 262143) are handled inconsistently by the native Iceberg writer:
dayandhourtransforms, andidentityon adatecolumn, fall throughiceberg_partition_path.rs:199into iceberg-rust's human-string rendering, which panics on the overflow. The panic is caught at the JNI boundary, so the task fails and the abort guard cleans up, but the write fails where iceberg-java succeeds.yearandmonthtransforms go through arrow'sdate_part, which returns NULL for values it cannot represent. Those rows get a NULL partition value, where iceberg-java computes the real one. That is silent wrong metadata.Only reachable with extreme values such as
date_from_unix_date(100000000)ortimestamp_micros(9000000000000000000), so low priority. Found by reading the code; not reproduced.Steps to reproduce
Write rows whose date or timestamp partition source value lies beyond year 262143 into tables partitioned by
days(ts),hours(ts),years(ts)andmonths(ts)withspark.comet.iceberg.write.enabled=true.Expected behavior
Either the same partition values as iceberg-java, or a clean fallback. At minimum the
year/monthcase should not write a NULL partition value.Additional context
Found in an audit of the native Iceberg write path. Part of #5649.