[eno][tombstone-recovery 4/7] Recover missing tombstones

[Ruinan Liu (ruinan-liu)]

Summary

Add an opt-in tombstone recovery controller that restores missing deletion instructions from saved inventory. Gate reconstitution and readiness until the current synthesis has a terminal recovery decision.

This PR reads existing inventory snapshots; recording and rotation are in a follow-up PR.

Behavior

• Enable the operator with --enable-tombstone-recovery. Recovery requires POD_NAMESPACE and watches Compositions within that namespace, respecting --composition-label-selector.
• Only Compositions annotated with eno.azure.io/recovery-enabled: "true" opt into recovery. Recovery eligibility does not evaluate --resource-filter; normal resource filtering remains unchanged.
• Read the latest inventory only when TombstoneRecoveryRequired is true; otherwise record NotNeeded.
• Compare complete current ResourceSlices against inventory and generate tombstones only for missing historical identities.
• Use a pure packing planner to fill existing slices first, then batch overflow into deterministic new slices. The writer handles persistence, freshness checks, and idempotent retries.
• Publish overflow references and the terminal recovery decision atomically before allowing reconstitution.
• Withhold Reconciled and Ready for opted-in, non-deleting Compositions until recovery reaches a terminal decision. Normal resource-status checks still apply afterward, including status entries for newly appended tombstones.
• When recovery is disabled or outside its namespace, reconstitution records TombstoneRecoveryOperatorNotEnabled or OutsideRecoveryNamespace. These decisions release the gates but preserve any outstanding recovery obligation for successor syntheses.
• Unannotated and deleting Compositions bypass recovery gating.

Race Conditions and Deletion Safety

• Superseded recovery: Re-read the Composition immediately before each ResourceSlice write and abandon obsolete work if deletion starts, the recovery annotation is removed, or the synthesis UUID changes. Existing-slice updates use optimistic locking.
• Atomic publication: Publish recovery completion and overflow references together using Composition UID, resourceVersion, and current-synthesis UUID preconditions. Skipped decisions written by reconstitution also use optimistic locking.
• Complete input before diffing: Missing, terminating, unreadable, or malformed current slices stop recovery rather than implying resource absence. Identity matching uses normal synthesis semantics, including Patch targets and existing tombstones. Recovery never changes an existing desired manifest to Deleted: true.
• Terminating current slices: If an opted-in, non-deleting Composition references a terminating slice while recovery is unfinished, the ResourceSlice controller requests resynthesis using the existing guards. This breaks the recovery/reconciliation/finalizer wait cycle while preserving the recovery obligation.
• Overflow cleanup: Marked recovery overflow slices are temporarily protected while their current synthesis requires unfinished recovery and the Composition remains opted in. Published references then provide normal cleanup protection.
• Successor synthesis: Outstanding recovery requirements survive unfinished recovery and disabled/out-of-scope decisions. Normal slicing carries forward unreconciled tombstones.

These safeguards are not a cross-object or cross-cluster transaction. This PR does not fix the pre-existing executor slice-UUID assignment issue or eliminate the downstream race where an already-running deletion overlaps a newer synthesis that wants the resource.

AKS Deployment Assumptions

• Each Composition has a one-to-one relationship with its eno-reconciler.
• Overlaymgr triggers a new synthesis after adding the recovery annotation. Adding the annotation to an already-cached synthesis without resynthesis remains a known limitation.
• ResourceSlices are Eno-owned; external deletion is outside the expected ownership contract.

Stack and Follow-ups

1. Recovery API and helper foundation.
2. This PR: tombstone recovery controller, reconstitution/readiness gates, and lifecycle safeguards.
3. Follow-up stacked PR: ready-synthesis inventory recording and rotation.

End-to-end coverage is deferred to a later effort. Secret-based inventory storage and admission-policy protection are separate follow-up items.

[Ruinan Liu (ruinan-liu)]

reply to comments

[xiazhan]

A recreated Composition can produce zero resources while downstream inventory still contains historical resources requiring deletion.

If reading that inventory fails temporarily, recovery remains unfinished. The new gate blocks reconstitution, but the separate ResourceSlice controller can still mark the empty synthesis Ready. Dependent Compositions then proceed before the missing tombstones have been recovered and processed.

Fix: also gate readiness publication on recovery completion when recovery is enabled.

eno/internal/controllers/resourceslice/slice.go

Lines 178 to 192 in a609882

	if comp.Status.CurrentSynthesis == nil {
	logger.Info("no current synthesis, returning early")
	return true // nothing else to do
	}
	// Empty compositions should logically become ready immediately after reconciliation
	if len(comp.Status.CurrentSynthesis.ResourceSlices) == 0 {
	logger.Info("empty composition, setting ready time to reconciled time")
	snapshot.ReadyTime = comp.Status.CurrentSynthesis.Reconciled
	}
	now := metav1.Now()
	comp.Status.CurrentSynthesis.Reconciled = snapshot.GetReconciled(comp, &now, logger)
	comp.Status.CurrentSynthesis.Ready = snapshot.GetReady(comp, logger)
	return true

eno/internal/controllers/scheduling/dependency.go

Lines 10 to 19 in a609882

	// buildReadySet creates a set of namespace/name keys for compositions that
	// have CurrentSynthesis.Ready != nil
	func buildReadySet(comps *apiv1.CompositionList) map[string]bool {
	readyMap := make(map[string]bool, len(comps.Items))
	for _, comp := range comps.Items {
	if comp.Status.CurrentSynthesis != nil && comp.Status.CurrentSynthesis.Ready != nil {
	readyMap[path.Join(comp.GetNamespace(), comp.GetName())] = true
	}
	}
	return readyMap

[xiazhan]

rename internal/controllers/backup to internal/controllers/tombstonerecovery ?

[Ruinan Liu (ruinan-liu)]

A recreated Composition can produce zero resources while downstream inventory still contains historical resources requiring deletion.

If reading that inventory fails temporarily, recovery remains unfinished. The new gate blocks reconstitution, but the separate ResourceSlice controller can still mark the empty synthesis Ready. Dependent Compositions then proceed before the missing tombstones have been recovered and processed.

Fix: also gate readiness publication on recovery completion when recovery is enabled.

eno/internal/controllers/resourceslice/slice.go

Lines 178 to 192 in a609882

	if comp.Status.CurrentSynthesis == nil {
	logger.Info("no current synthesis, returning early")
	return true // nothing else to do
	}
	// Empty compositions should logically become ready immediately after reconciliation
	if len(comp.Status.CurrentSynthesis.ResourceSlices) == 0 {
	logger.Info("empty composition, setting ready time to reconciled time")
	snapshot.ReadyTime = comp.Status.CurrentSynthesis.Reconciled
	}
	now := metav1.Now()
	comp.Status.CurrentSynthesis.Reconciled = snapshot.GetReconciled(comp, &now, logger)
	comp.Status.CurrentSynthesis.Ready = snapshot.GetReady(comp, logger)
	return true

eno/internal/controllers/scheduling/dependency.go

Lines 10 to 19 in a609882

	// buildReadySet creates a set of namespace/name keys for compositions that
	// have CurrentSynthesis.Ready != nil
	func buildReadySet(comps *apiv1.CompositionList) map[string]bool {
	readyMap := make(map[string]bool, len(comps.Items))
	for _, comp := range comps.Items {
	if comp.Status.CurrentSynthesis != nil && comp.Status.CurrentSynthesis.Ready != nil {
	readyMap[path.Join(comp.GetNamespace(), comp.GetName())] = true
	}
	}
	return readyMap

ResourceSlice status aggregation now withholds both  Ready  and  Reconciled  for recovery-opted-in, non-deleting Compositions until  TombstoneRecoveryComplete()  confirms a terminal decision for the current synthesis UUID, including empty syntheses.

Reconstitution records an explicit terminal decision when recovery is disabled or the Composition is outside the recovery namespace, avoiding an indefinite readiness wait. After recovery completes, normal resource readiness checks still apply. Aggregation also treats a status array shorter than  spec.resources  as unfinished, so newly appended tombstones cannot be overlooked.

[Ruinan Liu (ruinan-liu)]

Reply to comment

[Ruinan Liu (ruinan-liu)]

reply comment

[Ruinan Liu (ruinan-liu)]

Respond to comments