Catalog maintenance deletes completed Backup objects after a short barman-cloud-backup-list result

[ebertlion]

Versions: plugin-barman-cloud v0.13.0, CloudNativePG 1.30.0, Kubernetes 1.35, S3-compatible object storage (STACKIT).

What happened

On 2026-09-18 the plugin sidecar on the primary of two unrelated clusters logged Deleting backup not in the catalog for the two newest completed Backup objects (the base backups of the 17th and the 18th) and removed them:

{"level":"info","ts":"2026-09-18T09:50:05Z","msg":"Applying backup retention policy","logging_pod":"portal-postgres-2","retentionPolicy":"30d"}
{"level":"info","ts":"2026-09-18T09:50:07Z","msg":"Deleting backup not in the catalog","logging_pod":"portal-postgres-2","backup":"portal-postgres-daily-backup-20260917011500"}
{"level":"info","ts":"2026-09-18T09:50:07Z","msg":"Deleting backup not in the catalog","logging_pod":"portal-postgres-2","backup":"portal-postgres-daily-backup-20260918011500"}

The operator then logged terminal error: Backup.postgresql.cnpg.io "portal-postgres-daily-backup-20260917011500" not found for both objects. The same happened on a second cluster in another namespace at 13:05 UTC.

The bucket itself was intact: ObjectStore.status.serverRecoveryWindow kept its firstRecoverabilityPoint (2026-08-25) and lastSuccessfulBackupTime, WAL archiving never failed, and the next night's scheduled backup completed normally. The kube-apiserver showed a short 5xx spike in the same minute (about 16 errors/s for two five-minute windows, otherwise ~0.02/s), so the listing most likely came back short or empty.

Cause, as far as I can see

internal/cnpgi/instance/retention.go deletes every completed Backup object of the cluster whose status.backupID is not in the current GetBackupList result. There is no guard for the listing itself: an empty or partial catalog (transient object-store or API error, eventual consistency) deletes valid objects. On this side the effect was a false no backup in 26h alert; on a cluster that relies on Backup objects for restore selection it would hide two valid base backups.

Suggestions

• Skip the deletion pass when the listing is empty, or when it lists fewer backups than the Backup objects that are older than the newest catalog entry.
• Only delete objects whose backup is older than the retention window, since that is the only case where the catalog is expected to have dropped them.
• Or require an ID to be missing in two consecutive listings before the object is removed.

Happy to provide more logs or test a fix.