Skip to content

Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145

Description

@andygrove

Describe the bug

Partition values for dates and timestamps outside the range chrono and arrow can represent (beyond roughly year 262143) are handled inconsistently by the native Iceberg writer:

  • day and hour transforms, and identity on a date column, fall through iceberg_partition_path.rs:199 into iceberg-rust's human-string rendering, which panics on the overflow. The panic is caught at the JNI boundary, so the task fails and the abort guard cleans up, but the write fails where iceberg-java succeeds.
  • year and month transforms go through arrow's date_part, which returns NULL for values it cannot represent. Those rows get a NULL partition value, where iceberg-java computes the real one. That is silent wrong metadata.

Only reachable with extreme values such as date_from_unix_date(100000000) or timestamp_micros(9000000000000000000), so low priority. Found by reading the code; not reproduced.

Steps to reproduce

Write rows whose date or timestamp partition source value lies beyond year 262143 into tables partitioned by days(ts), hours(ts), years(ts) and months(ts) with spark.comet.iceberg.write.enabled=true.

Expected behavior

Either the same partition values as iceberg-java, or a clean fallback. At minimum the year/month case should not write a NULL partition value.

Additional context

Found in an audit of the native Iceberg write path. Part of #5649.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions