[core] Make compact_manifest perform a full manifest sort

[jianguotian]

Purpose

This is the follow-up requested in #9784 and builds on #9791 and #9792. Existing tables can retain an older manifest layout, while regular manifest compaction may skip stable runs. compact_manifest should explicitly rebuild those manifests using the table's current options.

Changes

• When manifest sorting is enabled, make an explicit compact_manifest enter the full-sort path by default.
• Bypass the routine full-compaction threshold and LSM run picker, include all existing sorted runs in one global section, and allow rewriting a single already-compacted manifest.
• Do not add a force flag or sort-order argument; the layout is selected from the current table options.
• Continue honoring manifest-sort.max-rewrite-size, and leave regular commit compaction unchanged.
• Preserve the existing full-compaction behavior when manifest sort is unavailable.

This rewrites manifest files only, not data files.

Verification

• Core manifest/commit tests: 173 passed, 1 skipped.
• Flink CompactManifestProcedureITCase: 7 passed.
• Spark CompactManifestProcedureTest: 2 passed.
• Core, Flink common, Spark common, and Spark UT compile with normal checks enabled.

[JingsongLi]

Why we need to add this? We should just respect options in the table.

[jianguotian]

Why we need to add this? We should just respect options in the table.

Thanks for review. I have remove it and follow table option.

[JingsongLi]

Do not introduce this too. By default, do full sort.

[jianguotian]

Thanks. I removed the force option; compact_manifest now performs a full sort by default using the table options.

[JingsongLi]

[P2] Do not charge unchanged singleton sections against the rewrite budget

Removing this shortcut also changes ordinary commits (fullSort=false). rewriteSection still returns a singleton unchanged when it does not require delete cleanup, but rewriteSections now charges its size and can set budgetExhausted=true. Subsequent overlapping sections without default-compaction files are then skipped, even though no manifest has actually been rewritten.

I reproduced this with six ADD-only manifests, each with fileSize = 8 MiB: the first contains partition 0, and the other five have overlapping partition ranges [1, 2]. With bucket=-1, manifest sorting enabled, and a 4 MiB rewrite budget, the picker selects all five runs, but the unchanged first singleton exhausts the budget and all six manifests are returned unchanged. Restoring the original singleton shortcut for !ctx.fullSort makes the regression test pass and allows the overlapping manifests to be merged.

Please preserve the existing singleton handling for ordinary compaction, or only deduct budget for files that are actually rewritten, while retaining forced singleton rewriting for explicit full sort.

[jianguotian]

Thanks. I restored the singleton shortcut for ordinary compaction while keeping singleton rewrites for explicit full sort, and added a regression test.

[JingsongLi]

+1