Skip to content

docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md #6147

Description

@andygrove

What is the problem the feature request solves?

The Iceberg user guide has errors and gaps about native writes:

  • docs/source/user-guide/latest/iceberg-writes.md:273-276 says float/double partition directories use Rust's shortest representation (f=1 where iceberg-java writes f=1.0). Since fix: render float and double Iceberg partition values like iceberg-java #5840 they use Java's Double.toString rules and match iceberg-java.
  • iceberg-writes.md:316-319 says the parity suite compares committed manifests byte-for-byte. CometIcebergWriteActionSuite compares aggregated readable_metrics (min/max, counts), not manifest bytes.
  • iceberg-writes.md does not say that writes whose input is a local relation, such as INSERT ... VALUES and df.writeTo(...).append() on a local DataFrame, need spark.comet.exec.localTableScan.enabled (default false). Without it they silently use iceberg-java even with both write flags on.
  • iceberg-writes.md does not list Native Iceberg writer keeps a dictionary page for high-cardinality columns where iceberg-java writes none #6114 (dictionary page kept for high-cardinality columns) among the accepted divergences.
  • docs/source/user-guide/latest/iceberg.md:184 lists "Iceberg writes (reads are accelerated, writes use Spark)" as unsupported. Native writes exist behind spark.comet.iceberg.write.enabled; that page should point to iceberg-writes.md.

Describe the potential solution

Fix the four points above. The "experimental, disabled by default" wording will need another pass when the defaults change (#5644).

Additional context

Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions