Skip to content

[core] Generate sidecars for newly written manifests - #9889

Merged
JingsongLi merged 2 commits into
apache:masterfrom
leaves12138:codex/manifest-sidecar-write
Sep 16, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
leaves12138:codex/manifest-sidecar-write

Conversation

@leaves12138

@leaves12138 leaves12138 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow up on #9845 and #9886 by generating sidecars for newly created Java manifests. Extracted from #9743 after rebasing that tracking PR onto current master.

  • Add manifest.sidecar.enabled, independently overridable and inheriting manifest-sort.enabled when unset.
  • Pass table options into the manifest factory and writer. Generate partition coverage for every table, row-ID coverage for data evolution, and bucket coverage when bucket != -1.
  • Build from the completed physical manifest after closing its output stream, including rolling writes and raw manifest rewrites.
  • Publish _EXTRA_FILES only after the sidecar closes successfully. Writer aborts and sidecar open/write/close failures clean up their own manifest/sidecar pairs, including earlier rolled outputs.
  • Keep row-ID statistics unavailable for invalid input ranges.
  • Apply the requested blank line after the ManifestIndexTestUtils class declaration.

This PR does not introduce sidecar scan pruning, a read option, Python changes, or a format change. Those remain separate follow-ups.

Validation

Java 8: 309 tests passed; one pre-existing disabled test was skipped. Checkstyle, Spotless and micro-benchmark test compilation passed.

New tests cover unset/default/explicit write switches, no sidecar input I/O when disabled, rolling outputs, raw-copy rewrites, publication after close, abort cleanup, partitioned/unpartitioned tables, independent coverage dimensions, invalid row IDs, and failures during sidecar creation, writing and closing.

mvn -B -ntp -nsu -pl paimon-benchmark/paimon-micro-benchmarks -am \
  -DwildcardSuites=none -DfailIfNoTests=false -Dflink.forkCount=2 \
  '-Dtest=CoreOptionsTest,ManifestSidecarWriteTest,ManifestSidecarTest,ManifestBlockIndexTest,ManifestFileTest,ManifestFileMetaTest,NoPartitionManifestFileMetaTest,ManifestRewriteCleanupTest,ManifestSidecarLifecycleTest,FileStoreCommitTest#testCommitManifestMerge+testManifestSortCompactManifestRespectsCompactionThresholds,ExpireSnapshotsTest#testSidecarsFollowSnapshotAndTagRetention+testCleanUnusedManifestExtraFiles+testCleanManifestExtraFilesFollowsManifestRetention,LocalOrphanFilesCleanTest#testOrphanCleanupProtectsReferencedSidecars+testPreservesManifestExtraFiles' test

.withDescription("Suggested file size of a manifest file.");

public static final ConfigOption<Boolean> MANIFEST_SIDECAR_WRITE =
key("manifest.sidecar.write")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

manifest.sidecar.enabled?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit c8c1402 into apache:master Sep 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants