[Bug] Fix Iceberg metadata unreadable by Snowflake - add Avro schema/partition-spec metadata to manifest files

[zhang-arvin]

What changes were proposed in this pull request?

Fix #9012: Paimon's Iceberg manifest files (Avro format) were missing the required Avro file-level metadata (schema, partition-spec, partition-spec-id, format-version) that Snowflake and other Iceberg readers require.

Changes

1. AvroFileFormat: Added AVRO_METADATA config option and setAvroMetadata() static method to allow setting Avro file-level metadata key-value pairs in the container file header.

2. IcebergManifestFile: Added create(FileStoreTable, IcebergPathFactory, Map<String, String>) overload that passes Avro metadata through to the Avro format writer.

3. IcebergCommitCallback: Computes the Iceberg schema and partition spec from the table schema at construction time and passes them as Avro metadata when creating the manifest file.

How was this patch tested?

• Compiled successfully with mvn -pl paimon-format,paimon-core -am -Pfast-build compile
• The metadata now conforms to the Iceberg spec requirements for manifest files

[JingsongLi]

I found several interoperability blockers in the manifest metadata change. Details are attached inline.

[JingsongLi]

[P1] This still writes field ID 0 into the Iceberg schema. Schema.Builder assigns the first Paimon column ID 0, and IcebergDataField(DataField) preserves it. I verified that a manifest produced by this PR has "id" : 0 in its schema header, which is the incompatibility reported in #9012. Adding the header therefore does not demonstrate that Snowflake can read the table. Please introduce a consistent positive-ID mapping everywhere Iceberg IDs are emitted (schema, partition source IDs, metrics maps, and any physical schema IDs), and cover it with a compatibility regression test.

[JingsongLi]

[P1] Iceberg's partition-spec manifest metadata is a JSON array of partition fields, not the complete partition-spec object. This serializes an unpartitioned spec as {"spec-id":0,"fields":[]}. I reproduced the resulting failure with Iceberg 1.6.1 ManifestFiles.read: Cannot parse partition spec fields, not an array. Please use the equivalent of PartitionSpecParser.toJsonFields(spec) here, keep partition-spec-id separate, and add a test that opens the generated manifest through Iceberg without supplying an external spec map.

[JingsongLi]

[P1] Iceberg v2/v3 manifests require a content header whose value is data or deletes, but this map omits it; the generated manifest has content = null. A single constructor-level value would also be insufficient because this IcebergManifestFile writes both Content.DATA and Content.DELETES, selected only by rollingWrite. Please build the metadata per writer from its Content (or use separate writer factories), and test both data and delete manifests.

[JingsongLi]

[P2] This only adds metadata to manifests created after the upgrade. createMetadataWithBase retains baseDataManifestFileMetas for add-only commits and retains existing DV manifests when there is no new index, so an already affected table remains a mixture of new and legacy headerless manifests and Snowflake still has to traverse the legacy files. Please provide a one-time manifest rewrite/migration path (or an explicit operational migration) and add an upgrade test starting from existing manifests.

[zhang-arvin]

Thanks for the review @JingsongLi! I've addressed all three P1 comments:

1. content header (C3): IcebergManifestFile now builds per-writer metadata from the Content being written — create() iterates Content.values(), puts content = data/deletes into each writer's options, and createWriter() picks the factory by Content. Both Content.DATA and Content.DELETES manifests get the header.

2. partition-spec (C2): the manifest partition-spec metadata is now the JSON array of partition fields via JsonSerdeUtil.toJson(partitionFields) (equivalent to Iceberg's PartitionSpecParser.toJsonFields). partition-spec-id remains a separate key. No more {"spec-id":0,"fields":[]} object.

3. schema field IDs (C1): added withPositiveFieldIds which rebuilds the schema with top-level field IDs assigned from 1 (preserving name/required/type/dataType/doc), so manifests no longer carry "id": 0.

Being transparent on the remaining follow-ups, which I'll continue working on in this PR:

• Nested field IDs inside struct/map/list types are not yet remapped to positive, monotonically-increasing IDs.
• Metrics maps (lower/upper-bound keyed by field ID) still reference the original Paimon column IDs.
• The P2 one-time rewrite/migration of already-committed legacy headerless manifests (C4) plus the corresponding upgrade tests.

I'll post updates here as these land. Feedback welcome in the meantime.