Extend PRA coverage to deletions, overwrites and non-versioned buckets - #2532
Open
delthas wants to merge 3 commits into
Conversation
Contributor
Author
|
@DarkIsDude Moved #2529 here (to get the right ticket ref in the branch) |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # yq eval 'sortKeys(.)' -i deps.yaml | ||
| backbeat: | ||
| sourceRegistry: ghcr.io/scality | ||
| sourceRegistry: ghcr.io/scality/playground/delthas |
There was a problem hiding this comment.
backbeat pulls from ghcr.io/scality/playground/delthas — a personal playground registry. The review criteria require Scality-internal images to resolve from ghcr.io/scality with a concrete released tag. The PR description says this will be replaced before merge; flagging so it isn't overlooked.
| dashboard: zenko-operator/zenko-operator-dashboards | ||
| image: zenko-operator | ||
| tag: v1.8.16 | ||
| tag: cf7e60c45ce51a598ef5c39ea3a58057c86627ac |
There was a problem hiding this comment.
zenko-operator is pinned to a bare commit SHA (cf7e60c...), not a released tag. The PR description notes that zenko-operator#631 is still open. This needs a released tag before merge.
| image: vault2 | ||
| tag: 8.11.9 | ||
| envsubst: VAULT_TAG | ||
| zenko-operator: |
Contributor
There was a problem hiding this comment.
we can't bump ZKOP to 1.9 at this point, analytics is still in flux...
Contributor
Author
There was a problem hiding this comment.
Keeping this thread open until we decide what to do
DarkIsDude
removed their request for review
September 15, 2026 08:51
delthas
force-pushed
the
wip/ZENKO-5355/pra-delete-and-overwrite-tests
branch
from
September 16, 2026 13:55
6e73248 to
1ce9109
Compare
The examples table carried `Non versioned` alone until 137e69d swapped it for `Versioned`, in the same commit that added the cross-site restore steps. Nothing records why: the commit message says only "Complete PRA logic", the pull request body names the tests it was missing, its review discussed the zenkoversion, the locations and the scripts, and the ticket has no description at all. Nor does the code suggest a reason -- the restore helper that commit extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state, which is where a restore writes `x-amz-restore` onto a master with nothing beneath it. Both rows rather than the swap back: the version delete this scenario covers needs a bucket with versions to delete. Issue: ZENKO-5355
delthas
force-pushed
the
wip/ZENKO-5355/pra-delete-and-overwrite-tests
branch
from
September 16, 2026 16:44
1ce9109 to
a87e734
Compare
delthas
changed the base branch from
development/2.16
to
improvement/ZENKO-5359/backbeat-configuration-overrides
September 16, 2026 16:44
Three paths the PRA scenario never touched: it deletes nothing, and its only bucket is versioned, so an object is never overwritten in place and never deleted without a version to remove. None of the deletions D/R has never replicated, nor the metadata a non-versioned overwrite has to carry, is visible to CI today. Removing a version and overwriting in place are steps of the nominal scenario, which already holds the state they need. Removing an unversioned object gets a scenario of its own: a failing step abandons the rest of its scenario, and this is the one whose outcome on a released deployment is not yet known. @pra is assigned at most one pickle at a time, so scenarios cannot fight over the two clusters. Delete markers stay uncovered, as ZENKO-4886 records: a marker carries no placement, so the location the source selects on cannot match it. A suspended bucket gets a scenario of its own too. A put there writes a null version, which the delete addresses as a version even though the master may be the only document holding it, so what removes the object is the teardown of the placeholder left on that master -- a write the source makes in no other case. Issue: ZENKO-5355
Not for merge. Exercises ZKOP-562, BB-811 and DRCTL-83 end to end: the @pra CTST run replicates cold objects through the mongo-processor rather than the Kafka Connect mongo sink. - backbeat 5e8a9c0b (BB-811, the D/R processor mode, with an object rewritten in place taken whole, and BB-853 on top: the sink omits the metrics block, and the producer is a hard startup dependency without it) - zenko-operator cf7e60c4 (ZKOP-562, the sink deployment and the source pipeline that shapes the entries it reads, rebased on the 1.9 that carries ZKOP-588 -- so drctl no longer needs DRCTL-81's merge patch to get past the analytics section marshalling as a null) - drctl v1.1.0-preview.1 (carries DRCTL-83, the objects topic on both D/R CRs, and the merged DRCTL-79 that sets the backbeat image on the sink; its vendor tree carries an operator new enough that the analytics section marshals as absent rather than as a null the apiserver rejects) The retag prefix is 9.5.3 because that branch really is 9.5.2, and every operator gate on the backbeat tag sits either below 9.5.1 or at 9.6.0-preview.1, so it reads exactly as 9.5.1 did. drctl is pinned to a released tag rather than to the per-commit image its own CI pushes.
delthas
force-pushed
the
wip/ZENKO-5355/pra-delete-and-overwrite-tests
branch
from
September 16, 2026 16:53
a87e734 to
e535ca5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PRA scenario deleted nothing, and its only bucket was versioned — so an object was never overwritten in place, and never deleted without a version to remove. This adds those paths and points the solution at the builds that make them pass.
Scenarios
PRA (nominal case)—VersionedPRA (nominal case)—Non versionedPRA (nominal case)—SuspendedPRA (null version in a versioned bucket)PRA (in-place overwrite)The
Non versionedrow was in the table until137e69dd91swapped it forVersioned, in the same commit that added the cross-site restore steps. Nothing records why — not the commit message, not the pull request, not the ticket — and the restore helper it extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state. Both rows run again.Deletion rides the Examples matrix rather than getting scenarios of its own: the steps are identical, and what differs is the bucket they run against. A versioned bucket removes a version; an unversioned one has no version to remove, so the delete takes away the object's only document; a suspended one creates a null delete marker, and the marker itself is not replicated — it carries the bucket's location constraint, which can never be cold. What removes the object there is the other write the delete makes: cloudserver removes the null version the marker replaces, and that removal carries the cold object's own metadata. Three different metadata events, one set of steps.
Two cases cannot be parameterised, so they run as their own scenarios rather than being repeated identically in every row. The in-place overwrite needs a non-versioned bucket, which no other row provides. The null version in a versioned bucket is about ordering: an object written before versioning is enabled stays a bare master with no version id, and stays one after the bucket becomes
Enabled— the state cloudserver describes as "a single null version [that] looks like a master key". Both scenarios install D/R first and write afterwards, so the write travels the change stream rather than the bootstrap dump.Deleting an unversioned object already passes on
development/2.16, but for the wrong reason — the delete flag lands on the master, the document a version-lessHeadObjectreads, so the object reads as absent while its document stays behind forever. That row is a regression guard, not a demonstration.What the bumps point at
v1.1.0-preview.1— carries DRCTL-83, the objects topic on both D/R CRs, and the merged DRCTL-79 that sets the backbeat image on the sinkThe two backbeat PRs stay pinned as one build, since the sink needs both and #2833 has not merged. The bump commit is replaced by released versions before this merges, so expect it to be amended until then.
A known race, not fixed here
Bucket definitions reach the sink through the Kafka Connect entity sync; object entries reach it through the mongo-processor. Nothing orders the two. An object entry that arrives before its bucket gets
NoSuchBucket, and the offset is committed regardless — so that object's metadata is dropped for good, and the scenario fails on an assertion no amount of waiting can satisfy.We have not tried to fix it here. The entity path this races is expected to be retired.
Issue: ZENKO-5355