[eno][tombstone-recovery 5/7] Record Ready-synthesis inventories as Secrets

[Ruinan Liu (ruinan-liu)]

Summary

Add inventory recording to the tombstone recovery controller. Once an eligible Composition's current synthesis is Ready, save its resource inventory as Secrets in the downstream cluster's kube-system namespace.

Switch recovery from ConfigMap inventories to Secret snapshots.

Behavior

• Record only when the operator is enabled, the Composition is in scope and opted in, and its current synthesis is Ready with a finished recovery decision.
• Read all current ResourceSlices before preparing inventory; stop on missing, terminating, unreadable, or malformed slices.
• Include resource identities, labels, and readiness/deletion-group annotations. Exclude tombstones and Patch resources.
• Split inventory deterministically into Secret chunks within the 1 MiB data limit.
• Use stable names: <comp-name>-<inventory-lineage>-<chunk-index>.
• Record empty inventory as one Secret containing [].
• Reuse existing Secrets on retries and avoid rewriting unchanged snapshots.

Recovery and Safety

• Find snapshots by lineage and select the newest source synthesis timestamp.
• Require a complete set of chunks with consistent snapshot metadata.
• Skip missing, mixed, malformed, or timestamp-ambiguous snapshots with InventoryInvalid; do not generate tombstones from ambiguous history.
• Recheck Composition identity, synthesis UUID, and recording eligibility immediately before writes.
• Use resourceVersion-protected updates to prevent overwriting concurrent changes.
• Preserve inventory lookup across Composition recreation without requiring the previous Composition UID.

Limitations and Scope

• Multi-Secret replacement is not atomic. A partial replacement may cause recovery to skip historical cleanup; this is an intentional best-effort tradeoff.
• Ignore leftover chunks after a smaller replacement; general inventory garbage collection and Composition-deletion handling remain separate work.
• No ConfigMap compatibility is retained because that inventory format has not been released.
• Inventory contains resource identities and deletion metadata, not full resource manifests.

[xiazhan]

selectNewestCompleteInventory only replaces the selected inventory when inventory.synthesized.After(selected.synthesized). If two complete snapshots have the same inventorySynthesizedAnnotation, the selected inventory depends on map/slice iteration order. This is plausible because timestamps are stored with time.RFC3339, which drops sub-second precision, so two syntheses in the same second become indistinguishable.

[Ruinan Liu (ruinan-liu)]

Thanks for flagging this. A tie should be extremely unlikely: both syntheses must become Ready and have their inventories recorded within the same Synthesized second. A crash before cleanup can extend how long both snapshots coexist, but does not make a timestamp tie more likely.

Even if a tie occurs, either complete snapshot is safe. Recovery only creates tombstones for identities absent from the current synthesis, so it cannot remove currently wanted resources; selecting the older snapshot may only miss some cleanup. Treating the tie as InventoryInvalid would skip recovery entirely, which is less useful than either snapshot.

[xiazhan]

The secret or configmap has 1mb size limit. Does control-plane-core beyond this limit? I think it doesn't because we only store meta data.

[xiazhan]

groupInventorySecretsBySynthesisUUID reads inventorySynthesizedAnnotation from inventorySecrets[0] and assumes all chunks for that synthesis share it. orderInventoryChunks validates index/count completeness but not that source metadata is consistent across chunks. A mixed/corrupted chunk set with the same synthesis UUID and consistent count can be treated as one complete snapshot even if chunks disagree on synthesized time.

Suggested fix: while ordering chunks, verify all chunks in the synthesis group have the same synthesized timestamp and expected labels/annotations, not just matching chunk count.

[Ruinan Liu (ruinan-liu)]

Hey Allan, all chunks for a synthesis UUID are written fro mthe same Synthesized value under UUID-specific names, and writeInventoryChunck errors if an existing Secret's content don't match. So mismatched chunks can only come from someone manually editing the inventory Secrets in Kube-system so it should be covered once we merge the planned admission-policy followup

[Ruinan Liu (ruinan-liu)]

The secret or configmap has 1mb size limit. Does control-plane-core beyond this limit? I think it doesn't because we only store meta data.

Yes, you are right. It won't hit the limit. Inventory only stores resource identity (group/version/kind/namespace/name), labels and the eno.azure.io/readiness-group and eno.azure.io/deletion-group annotations, not specs or data. If it ever exceed 1mb, recording splits it across multiple secrets.

[Ruinan Liu (ruinan-liu)]

reply to comment