[core] Reject merge-schema type widening for primary-key and partition columns

[zhang-arvin]

What is the purpose of the change

Fixes #9993. The automatic schema merge path bypasses the invariant that the explicit schema change path enforces: key columns must not change type.

Brief change log

• SchemaMergingUtils#mergeSchemas(TableSchema, RowType, ...) merges every incoming field type and then re-attaches the existing primaryKeys() / partitionKeys() name lists, without ever validating that those columns kept their type. SchemaManagerUtils already rejects this for explicit changes via assertNotUpdatingPartitionKeys / assertNotUpdatingPrimaryKeys, but FileSystemSchemaManager#mergeSchema (i.e. every write-time automatic schema merge) called mergeSchemas directly and skipped those guards.
• Add assertNotUpdatingKeyColumn, invoked right before the merged TableSchema is constructed, which rejects the merge with an UnsupportedOperationException when a primary-key or partition column's type changed.
• Non-key column widening behaviour is unchanged.

Why it matters

Widening a key column changes its value encoding, so the same logical key hashes into a different bucket. Concretely, for a bucketed primary-key table, INT(-100000) hashes to bucket 3 while BIGINT(-100000) hashes to bucket 13 — the same logical key ends up in two different buckets, so intra-bucket deduplication no longer applies and duplicate rows accumulate.

Tests

• testRejectTypeWideningOnPrimaryKeyColumn — covers the reported INT -> BIGINT primary-key scenario, asserting the merge is rejected.
• testRejectTypeWideningOnPartitionColumn — same for a partition column.
• testAllowTypeWideningOnNonKeyColumn — guards against over-rejection: non-key widening still merges.

All 17 tests in SchemaMergingUtilsTest pass.

[zhang-arvin]

The paimon-core failures in CI (MockRESTCatalogTest.testDataTokenExpired, FormatTableCommitTest.testAbortFailureDoesNotReplaceInterruptedCleanupFailure) are pre-existing on upstream master, not caused by this change.

Evidence — the same jobs fail on a clean master run:

• run https://github.com/apache/paimon/actions/runs/35684053322 (sha 0afddbd37, branch master, no PR changes)
• its Java / Core and integrations / JDK 8 job reports:
  MockRESTCatalogTest>RESTCatalogTest.testDataTokenExpired:2544 expected: <token is expired> but was: <Requested presigned URL validity exceeds the remaining REST credential lifetime after refresh.>

This PR only touches SchemaMergingUtils (automatic merge-schema validation) and its test; neither failing class is on that path. SchemaMergingUtilsTest passes 17/17 locally, including the INT→BIGINT primary-key case from the report.

This comment was generated by an AI agent (Hermes Agent).

[JingsongLi]

Reviewed 17406ca9e0ff1d4d857670a5c74887557a9aec9e. Requirement fit: SUPPORTED; this closes a real correctness gap in path-based schema merging, and the guard belongs before schema persistence. Keep this PR open.

The core checks pass locally on JDK 8: all 17 SchemaMergingUtilsTest cases plus three independent reviewer tests covering rejected key/partition widening without schema persistence, case-insensitive matching, column addition with widening disabled, and the reported INT/BIGINT bucket difference. The two rejection tests fail against the pre-fix helper and pass with this change.

However, the current Spark3 CI has a directly related failure in Paimon Sink: enable schema evolution, so the PR needs the integration-test update described inline. This failure is separate from the reported core-suite failures. I inspected the current CI trace; I did not rerun the Spark suite locally.

[JingsongLi]

[P2] Update the Spark evolution tests for the new key-type contract

This correctly rejects a scenario that the existing Spark success tests still require: paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala:242 creates primary key a INT, feeds MemoryStream[(Long, Date, Int)] with type widening enabled, and expects the key to become BIGINT. The Spark 3.4 copy at paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala:239 does the same. Current Spark3 CI fails that test with Cannot update primary key type from INT NOT NULL to BIGINT NOT NULL: [a], with the stack pointing to this new guard. Please update both successful evolution tests to retain the key type while evolving non-key columns, and add a separate streaming rejection case checking that schema metadata and existing rows remain unchanged. The guard should remain; the integration tests need to reflect the intended invariant in this PR.

[Akash3121]

This test is named  testRejectTypeWideningOnPartitionColumn , but it changes  VARCHAR(MAX)  to  VARCHAR(100) , which is a narrowing conversion, and it only succeeds as a merge candidate because  allowExplicitCast  is enabled below. Could we instead use  INT -> BIGINT  for the partition column and call  mergeSchemas  with  allowExplicitCast=false ? That would directly verify that the normal type-widening path rejects partition-column type changes, independently of explicit-cast behavior.

[Akash3121]

The corresponding corrected test setup would look like:

 DataField b = new DataField(1, "b", new IntType());
 // ...
 DataField bWidened = new DataField(-1, "b", new BigIntType());
 RowType t = new RowType(Lists.newArrayList(a, bWidened));
 
 assertThatThrownBy(() -> SchemaMergingUtils.mergeSchemas(current, t, true, false, true))
         .isInstanceOf(UnsupportedOperationException.class)
         .hasMessageContaining("Cannot update partition column type");