You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Iceberg user guide has errors and gaps about native writes:
docs/source/user-guide/latest/iceberg-writes.md:273-276 says float/double partition directories use Rust's shortest representation (f=1 where iceberg-java writes f=1.0). Since fix: render float and double Iceberg partition values like iceberg-java #5840 they use Java's Double.toString rules and match iceberg-java.
iceberg-writes.md:316-319 says the parity suite compares committed manifests byte-for-byte. CometIcebergWriteActionSuite compares aggregated readable_metrics (min/max, counts), not manifest bytes.
iceberg-writes.md does not say that writes whose input is a local relation, such as INSERT ... VALUES and df.writeTo(...).append() on a local DataFrame, need spark.comet.exec.localTableScan.enabled (default false). Without it they silently use iceberg-java even with both write flags on.
docs/source/user-guide/latest/iceberg.md:184 lists "Iceberg writes (reads are accelerated, writes use Spark)" as unsupported. Native writes exist behind spark.comet.iceberg.write.enabled; that page should point to iceberg-writes.md.
Describe the potential solution
Fix the four points above. The "experimental, disabled by default" wording will need another pass when the defaults change (#5644).
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.
What is the problem the feature request solves?
The Iceberg user guide has errors and gaps about native writes:
docs/source/user-guide/latest/iceberg-writes.md:273-276says float/double partition directories use Rust's shortest representation (f=1where iceberg-java writesf=1.0). Since fix: render float and double Iceberg partition values like iceberg-java #5840 they use Java'sDouble.toStringrules and match iceberg-java.iceberg-writes.md:316-319says the parity suite compares committed manifests byte-for-byte.CometIcebergWriteActionSuitecompares aggregatedreadable_metrics(min/max, counts), not manifest bytes.iceberg-writes.mddoes not say that writes whose input is a local relation, such asINSERT ... VALUESanddf.writeTo(...).append()on a local DataFrame, needspark.comet.exec.localTableScan.enabled(defaultfalse). Without it they silently use iceberg-java even with both write flags on.iceberg-writes.mddoes not list Native Iceberg writer keeps a dictionary page for high-cardinality columns where iceberg-java writes none #6114 (dictionary page kept for high-cardinality columns) among the accepted divergences.docs/source/user-guide/latest/iceberg.md:184lists "Iceberg writes (reads are accelerated, writes use Spark)" as unsupported. Native writes exist behindspark.comet.iceberg.write.enabled; that page should point toiceberg-writes.md.Describe the potential solution
Fix the four points above. The "experimental, disabled by default" wording will need another pass when the defaults change (#5644).
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.