From 5873bb3035de4fb4e37bb6afaa28b393bc37ad54 Mon Sep 17 00:00:00 2001 From: Yetkin Timocin Date: Tue, 22 Sep 2026 14:47:13 -0700 Subject: [PATCH 01/20] feat: [FEP-0001] support annotation-based placement (#829) * feat: [FEP-0001] parse the place-to annotation into cluster selectors Adds the parser behind annotation-based placement: it turns the value of the kubefleet.dev/place-to annotation into the cluster selectors of a placement policy. The controller that keeps a policy object in sync with the annotation follows separately. The grammar is the one FEP-0001 specifies: a semicolon-separated list of selectors, each a comma-separated list of LABEL_KEY=LABEL_VALUE matchers with an optional count=N|All directive that defaults to 1, and with `region` and `alias` accepted as shorthands for the well-known region label and the reserved cluster alias label. Every key and value is validated against the rules Kubernetes itself enforces, and the parser's own limits are held to the ones the placement policy CRD declares, so a mistaken annotation is reported against the annotation rather than surfacing later as a policy object the API server rejects and a reconciler that retries it forever. Where the FEP is silent the parser errs toward rejecting the input: duplicate label keys (including a shorthand colliding with the key it expands to), a repeated count directive, and an empty label value are all errors rather than silently resolved. A selector carrying only a count directive is allowed, so `count=All` on its own selects the whole fleet. The count ceiling is enforced client-side only. The API's pattern marker constrains the string form of the field alone, so the API server today accepts an integer count both above the ceiling and below one; this is noted on the constant and belongs with the other CEL gaps in the placement policy API. Signed-off-by: Yetkin Timocin * feat: [FEP-0001] derive the placement policy generated from an annotation Computes what the placement policy for an annotated resource should look like: its name, the labels recording where it came from, its owner reference, and the resource selector that places the annotated resource itself. The reconciler that creates, updates and deletes it follows separately. The policy's scope follows the annotated resource's own, so the owner reference is always same-scope: a cluster-scoped object owned by a namespaced one is accepted on creation and then never collected at all. The reference claims neither controller nor blocking ownership, since setting either requires permission to update the finalizers subresource of the owner, and the owner here can be a resource of any kind. Names and label values are derived through a new pkg/utils/naming, added rather than written twice: the placement policy controller of #786 has its own private copies of the same helpers, with the same semantics, and can move onto this package when it merges. Two bugs have already been fixed once in those copies, and a second copy is how they come back. Every part of an identity that can outgrow where it is put is shortened through that package, including the API group, which is validated as a DNS-1123 subdomain and so runs to four times what a label value holds. The generated name is shortened the same way and finished with a hash of the whole, untruncated identity, so that resources which differ only past the truncation point, or only in the case of their kind, cannot converge on one policy. Fields the API server defaults are set explicitly. Left unset they are stored with the default and then read as a difference on the next pass, which would have the reconciler issue an update that changes nothing for as long as the policy exists. Signed-off-by: Yetkin Timocin * refactor: [FEP-0001] name the placement annotation after what it holds The annotation key was introduced as a placeholder. Rename it to kubefleet.dev/cluster-selectors, which names the value rather than the action taken on it and matches the clusterSelectors field of the policy generated from it, so that the annotation and the field it stands in for do not have to be translated between when read together. Renames the constant to ClusterSelectorsAnnotation and the parser entry point to parseClusterSelectors to follow. No behavior changes. Signed-off-by: Yetkin Timocin * feat: [FEP-0001] reconcile the placement policy generated from an annotation Adds the reconciler that keeps a generated placement policy in step with the cluster-selectors annotation on a resource: it creates the policy when the annotation appears, updates it when the annotation changes, and deletes it when the annotation is removed. A resource that is itself deleted is deliberately left alone. The generated policy carries an owner reference back to it, so garbage collection removes the policy without this controller racing the collector to delete an object that is already doomed. An annotation the parser rejects is reported to the user as an event on the annotated resource and the key is dropped rather than retried: no number of retries makes a malformed annotation parse. A policy generated from an earlier, valid annotation is left standing, since tearing down a running placement is a worse answer to a typo than keeping the last placement the user did express. Only what the controller generates is overwritten on an existing policy -- the spec, the provenance labels, and its own owner reference. Labels and owner references from elsewhere are preserved so that a generated policy can be labelled by an operator or a GitOps tool without the two taking turns undoing each other. Signed-off-by: Yetkin Timocin * feat: [FEP-0001] run annotation-based placement in the hub agent Wires the annotation placement controller into the resource watcher, behind --enable-annotation-based-placement. The flag defaults to off: the feature reads and writes the placement.kubefleet.dev/v1alpha1 policies, whose definitions this chart does not install yet, so the hub agent checks they exist and refuses to start without them rather than retrying against a kind the cluster does not have. The controller shares the resource watcher's informers but is fed only the resources that carry the annotation, which keeps every unrelated object in the cluster out of its queue. An update is enqueued when either the old or the new object carries the annotation: looking at the new one alone would filter out the removal of an annotation, which is precisely the event that has to delete the generated policy. A deleted resource is not enqueued at all, since the policy is garbage collected through its owner reference. The controller gets a rate limiter of its own rather than the one the placement controllers share. An exponential failure limiter keys its backoff on the queued item alone, and this controller queues the same cluster wide keys as the resource change controller, so one limiter between them would let a success here clear the backoff that repeated failures there had earned. Grants the hub agent write access to the generated policies; the broad rule it otherwise relies on is read-only. No status subresource rule: the controller never writes status. Adds an envtest suite covering the round trip against a real API server -- annotation set, changed, made invalid, and removed -- for both a namespaced and a cluster-scoped resource. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] repair generated policy drift and close the cleanup gaps The controller synchronized in one direction only: it acted on events about the annotated resources, so anything that happened to a generated policy directly went unseen. Deleting a policy, or editing its spec, produced no event on the resource it came from, and for a resource that never changes again the policy stayed missing or wrong forever -- a resync does not step in, since updates whose resource version did not move are dropped. The resource watcher now watches the generated policies themselves and, for every policy event, enqueues the resources the policy names as its owners, so the next reconciliation restores whatever the event changed. The owner references are used rather than the parent labels, which are lossy and, being labels, can be stripped -- itself drift this path exists to repair. The placement.kubefleet.dev group joins the resource config's default skip list along the way: everything in it is KubeFleet's own bookkeeping, and treating the generated policies as placeable resources would let a placement that selects a namespace propagate them. A deleted resource now has its generated policy deleted explicitly rather than left to garbage collection. The collector removes a dependent only once every owner reference on it is gone, and the merge deliberately preserves owner references that other parties add -- any live one of which would have kept the policy standing indefinitely. A resource that stops passing the resource watcher's filter without being deleted -- a ReplicaSet adopted by a Deployment, for instance -- is reported by the watcher as a deletion and then never seen again. The reconciler therefore applies the same eligibility test itself and deletes the generated policy of a live but ineligible resource, that being its last chance to clean up. A key whose kind the API server does not know, which the policy watch can produce by enqueuing whatever a policy names as its owner, is dropped rather than retried: no retry makes the kind exist, and the key would otherwise back off on the queue forever. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] do not report a deletion another pass already performed Deleting a generated policy re-enters the reconciler through the generated policy watch, and the second pass's cached read can still see the object the first pass just deleted. The delete it then issues comes back not found, which was being swallowed into a success: one real deletion produced two "deleted" log lines, and on the annotation-removal path would have produced two events. Surfaced by exercising the controller on a kind cluster rather than by any test. A delete that finds nothing now reports that this pass deleted nothing, matching what the read-side already did when the policy was gone before the pass began. Also settles the whenUnfulfilled question the policy derivation had flagged as open: generated selectors keep the API's RequestCluster default deliberately, since asking the platform for a cluster that does not exist yet is the point of these APIs and the annotation is meant to be the lowest-friction way to do so. Guarding against a selector that nothing can ever satisfy belongs to the claim fulfillment layer. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] bound the integer form of a selector count The count of a cluster selector is an int-or-string whose only bound was a pattern, and a pattern constrains the string form alone: the API server rejected count: "1000" while accepting count: 1000, along with zero and negative integers the field's own documentation rules out. A CEL rule now bounds the integer form to the same 1-999 range the pattern holds the string form to. The parser had been holding this line by itself for annotation-derived counts; hand-authored policies had no line at all. The bounds are pinned twice: envtest submits both forms of both verdicts to a real API server, and the parser's limits test asserts the CRD's rule and message name the same ceiling the parser enforces, without binding either to an exact spelling. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] keep the minCount rule out of problems that are not its own The minCount<=count rule guarded its string comparison with an unbounded digit match, so a count too long for an int64 reached the int() conversion and turned a plain validation failure into an opaque evaluation error beside it. The guard now recognizes only the numeric strings of at most the three digits the count field itself permits; anything else -- "All", or junk the field's own validation rejects -- passes the rule vacuously, so the real problem is reported alone, and the message drops a qualifier about a case the rule no longer treats specially. Count additionally gains a MaxLength matching its longest legal values, which rejects an over-long string outright. The digit guard is spelled [0-9]{1,3} rather than mirroring the count pattern's own numeric branch verbatim: the CEL cost estimator prices a regex by its length against an unbounded string, an int-or-string being opaque to the sibling MaxLength, and the longer spelling does not fit the budget. Signed-off-by: Yetkin Timocin * fix: bound the integer form of the rolling update settings The maxUnavailable and maxSurge fields of a rolling update strategy are int-or-strings whose only bound was a pattern, and a pattern constrains the string form alone: the API server rejected maxUnavailable: "-1" while accepting maxUnavailable: -1, a value the field's documentation rules out and the rollout controller cannot act on. A CEL rule now keeps the integer form non-negative in both the v1 and v1beta1 APIs. For ClusterResourcePlacement the validating webhook already rejected negative values, so this closes a gap only where the webhook is not running. ResourcePlacement has no working webhook behind it at all, so there the schema is the only line of defense; the new tests cover both kinds for that reason. Note that these are served API versions: a manifest carrying a negative integer bound, previously accepted and stored, is rejected on its next update that changes the value. Unchanged stored values are ratcheted through, since the field sits on a plain struct path that validation correlates across updates. Signed-off-by: Yetkin Timocin * fix: judge the object a tombstone wraps rather than the tombstone A deletion the watch missed arrives as a tombstone wrapping the object's final state, and the resource watcher's filter derived its key from the tombstone itself, which is not a runtime object. Every relist-detected deletion was therefore dropped before any handler saw it, even though the delete handler has always known how to unwrap one. The filter now unwraps the tombstone before judging the object, and the test that pinned the dropped-deletion behavior as intended now pins the opposite. Signed-off-by: Yetkin Timocin * feat: [FEP-0001] seed the cluster alias label on member cluster join FEP-0001 reserves the kubefleet.dev/cluster-alias label for selecting member clusters by name, and its own examples select by alias -- yet nothing has ever set the label, so every alias selector matched no clusters at all. With annotation-based placement enabled, the member cluster controller now seeds the alias from the cluster name on join, and only when the label is absent entirely. Unlike the member name label beside it, which states a fact the controller owns and reasserts, the alias exists to be renamed: it is the indirection that lets a selector follow the cluster playing a role rather than a fixed name, so a value an admin changed is never reverted. Fleets that do not run the feature are left untouched. The guard rail that optionally denies member cluster label changes exempted only the kubernetes-fleet.io prefix; it now exempts the kubefleet.dev prefix as well. The hub agent is not in system:masters, and denying it this label would have failed the same update that maintains the member name label, wedging the cluster's reconciliation. Signed-off-by: Yetkin Timocin * fix: bound the digit form of the rolling update settings The digit branch of the rolling update pattern accepted a string of any length, which passed validation only to fail in whatever consumed it later. It is now bounded to nine digits, all of which fit an int32 comfortably, in both the v1 and v1beta1 APIs; the same ratcheting considerations apply as for the sign bound before it. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] name the generated kind in the controller's events An event about a ClusterPlacementPolicy called it a placement policy, so pasting the name from the event into kubectl against that kind found nothing. The created, updated, and deleted events now name the kind verbatim. The kind is asked of the same function that decides the scope, keeping that decision in the one place its contract promises, and the deleted event's doc comment now covers both of its causes. Signed-off-by: Yetkin Timocin * feat: [FEP-0001] warn on a duplicate cluster alias The cluster alias label is seeded on member cluster join and selects a cluster by a name of the admin's choosing, so it is meant to identify one cluster; two clusters sharing an alias makes an alias-based selector match both. The member cluster validating webhook now warns when the alias being set is already held by another cluster. It is a warning, never a denial: labelling a replacement cluster with the outgoing one's alias before removing it from the outgoing one is exactly the handoff the alias exists to allow, and that state must stay legal while transient. For the same reason a failure to list the member clusters admits the request without a warning rather than blocking it -- the webhook fails closed, so an advisory check that errored on a listing hiccup would wedge every member cluster write. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] follow only the generating owner and skip excluded namespaces Two correctness gaps found in review of the generated-policy watch and the placement eligibility check. The watch re-enqueued every owner reference a policy carried, so a foreign owner -- one the merge preserves, or any owner on a policy that merely shares these informers -- was treated as a generating source. If its kind was one the resource watcher does not track, reading it lazily created an informer outside the resource configuration. Only the owner whose identity reproduces the policy's own generated name is followed now; matching a 64-bit identity hash admits nothing else. The eligibility check read metadata.namespace to decide whether a resource's namespace is skipped, but a Namespace object carries none and is itself the namespace, so an annotated "default" or "kube-system" generated a policy for a namespace KubeFleet excludes. The namespace is now taken from the object's own name for a Namespace, matching how the resource change controller already special-cases the kind. Also clarifies, per review, that the event reasons are the same for both generated kinds, that eligibility can change over a resource's life and is re-checked on the events that change it, and where the placement CRDs come from until a chart installs them. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] restrict the kubefleet.dev/ label exemption to fleet controllers The guard rail that optionally denies member cluster label changes exempted the whole kubefleet.dev/ prefix for every user. Unlike the member-name label the controller reasserts each reconcile, the cluster alias is left as it is set, so a non-admin's edit to it would persist and redirect alias-based placements to another cluster. The prefix is now exempt only for a whitelisted identity (the hub agent, which seeds the alias) or a cluster admin; ordinary users are denied. The kubernetes-fleet.io/ prefix stays exempt for everyone as before, and admins can still rename an alias for a cluster handoff, since they are exempted before this check runs. Signed-off-by: Yetkin Timocin * refactor: [FEP-0001] compute the generated policy name once on the delete path deletePolicy derived the generated policy name and deleteGeneratedPolicy derived it again for the same policy -- two SHA-256 hashes of the same identity per deletion. deleteGeneratedPolicy now returns the name it computed, and deletePolicy uses it. No behavior change. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] harden the generated placement policy against review feedback Address review findings on the annotation-based placement controller: - Sanitize characters legal in a resource's own name but not in a Kubernetes object name (an RBAC name like system:aggregate-to-admin, or a dot beside another separator) before they reach a generated policy's name, which the API server would otherwise reject and hot-loop the reconciler. The uniqueness hash is still taken over the full, unsanitized identity. - Leave a policy this controller did not generate untouched: before updating or deleting the policy at a resource's deterministic name, verify it carries this controller's provenance labels, so a hand-authored policy colliding with that name is neither overwritten nor deleted. The conflict is surfaced to the user through an event. - Clean up a stale generated policy when its source's API is gone: a source of a config-excluded API stops being placed, and a key whose kind the API server no longer knows has its generated policy deleted rather than left behind. A key naming a version retired while the kind lives on falls back to the served version instead of being treated as a gone kind. - Converge the owner reference across a source deleted and recreated under the same name: match the existing reference on the source's group, kind, and name rather than its UID, so the one reference is updated in place instead of a stale one accumulating every cycle. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] keep a generated policy repairable and correctly scoped Two further review findings on the annotation-based placement controller: - A generated policy was recognized as this controller's own only when all of its provenance labels matched, and that check gates both the repair path, which restores drifted labels, and the delete path. Editing one label on a generated policy therefore made it neither repairable nor removable, so its placement could keep running after the annotation was removed. Recognize ownership from either marker this controller stamps -- the owner reference to the source or the provenance labels -- so a single drifted marker no longer strands the policy, while a foreign policy that merely collides with the name still carries neither and is left untouched. - The source's scope was read from the queued key's group-version-kind, which the informer manager keys on the exact version. When a served version is retired, or the version-agnostic mapping fallback resolves a different one than the key named, a cluster-scoped source was read through a namespaced lookup, missed, and taken for deleted, deleting and recreating its policy. Read the scope from the resolved REST mapping instead. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] read generated policies uncached and requeue on name conflict Two further review findings on the annotation-based placement controller: - Generated policies are watched through a different informer than the controller's cache read from, so an edit or deletion the watch delivered first could be met with a stale object: the pass saw no change and returned, leaving drift until the next resync and, for a deletion -- which a later event or resync never re-delivers, the object being gone from the watch's own cache -- leaving the placement removed indefinitely. Read the policy through an uncached reader, current as of the moment the watch fired; writes still go through the cached client. The queue holds only annotated resources and generated-policy owners, so the read is not on a hot path. - When a policy occupies a resource's generated name but was authored by someone else, the reconciler left it untouched and returned without asking to be run again. Removing that policy fires no event that reaches the source, since it carries no owner reference to it, and the annotation does not change, so the requested policy would never be created once the conflict was cleared. Requeue the source while the conflict stands so the placement is created as soon as the name is free. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] guard the generated policy delete against a replacement race deleteGeneratedPolicy read the policy through the uncached reader and confirmed it was one this controller generated, then deleted it by name with no precondition. A delete targeting the name alone removes whatever occupies it, so a policy replaced between the read and the delete -- deleted and a hand-authored one created at the same name, or overwritten in place with its provenance stripped -- would be deleted even though it was no longer this controller's. Carry the resource version the read returned as a delete precondition, so the delete removes only the exact object this pass verified. A replacement moves the resource version, turning the delete into a conflict that requeues; the next pass reads the current object and, finding it foreign, declines it. The resource version guards both a delete-and-recreate and an in-place overwrite, where a UID precondition would miss the latter. Signed-off-by: Yetkin Timocin * chore: [FEP-0001] adopt the ClusterClaim rename from #803 The placement policy API renamed cluster requests to cluster claims; the generated policy's default WhenUnfulfilled option follows suit (RequestCluster -> AddClusterClaim). Signed-off-by: Yetkin Timocin * chore: [FEP-0001] leave annotation-based placement unwired in the hub agent Land the annotation-based placement controller as dormant code, per the deferred-integration ask on the review: the experience is not usable until the rest of FEP-0001 is in place, and the flag, chart values, and hub agent setup that connected the controller would reach every existing installation through backports and release cuts for no benefit. Removed, to return with the integration change once the chain works end to end: the --enable-annotation-based-placement flag and its chart value, RBAC and README entries; the controller construction and CRD check in the hub agent's workload setup; the scheme registration and alias-seeding option in main; and the resource watcher hooks that fed annotated resources and generated-policy events into the controller's queue, with their test. Kept: the controller package with its unit and envtest coverage, the API constants and validation, the naming helpers, and the tombstone unwrap fix in the resource watcher's filter, which stands on its own. Doc comments that described the removed wiring as present now describe the contract the integration change has to meet instead. The kind e2e suite that drove the feature through a live hub agent needs the flag to run and is parked on parked/fep0001-annotation-placement-e2e. Signed-off-by: Yetkin Timocin * refactor: [FEP-0001] port annotation-based placement to controller-runtime Move the controller to pkg/v1/controllers, where the rest of the new placement experience lives, and rebuild it on controller-runtime rather than the in-house queue framework, per review. The controller reconciles resources of any kind the hub agent watches, so it runs as a generic controller over a request that carries the kind alongside the namespaced name. The annotated resources reach it through a channel source that the resource watcher will feed once the feature is wired in; the generated policies are watched through the manager's cache, with an event on one mapped back to the resource it came from, so that an edit or a deletion of a generated policy is repaired even when the resource never changes again. Annotated resources are read straight from the API server: they can be of a kind no cache has seen, and a cached read would start an informer for it -- for a kind that cannot be listed or watched, one that never syncs. The generated policies are read through the manager's cache instead, which the policy watch is served from as well. The create-or-update path, the owner reference, and the ownership check use controllerutil, and errors are categorised with the errors package. A policy that occupies a resource's generated name but was authored elsewhere is now reported as an error so that the queue backs off exponentially, instead of being re-examined at a fixed interval forever. The explicit, resource-version-guarded delete stays: garbage collection cannot cover a policy whose annotation was removed or whose resource stopped being eligible, and cannot override an owner reference another party added that the merge preserves. The envtest suite now also runs the controller under a manager, fed through the channel, to prove the watch on the generated policies. Signed-off-by: Yetkin Timocin * refactor: [FEP-0001] tidy the annotation placement reconciler Merge the not-found and no-match branches of Reconcile: both delete the generated policy and neither requeues, so one branch does. Replace the hand-rolled digit check plus ParseInt in parseCount with ParseUint, which already rejects signs, whitespace, and exponents; the out-of-range and not-an-integer error families are preserved through strconv.ErrRange. Signed-off-by: Yetkin Timocin * feat: seed the cluster alias label on member clusters Add the kubefleet.dev/cluster-alias label, seeded from the cluster name by the member cluster controller when it is absent and never reasserted, so an admin can move an alias between clusters and a selector can follow a role rather than a fixed name. The member cluster webhook warns, but does not deny, when another cluster already holds the alias: two holders is the normal middle state of a handoff. The member cluster label guard treats the kubefleet.dev/ prefix like kubernetes-fleet.io/: the hub agent writes both and is not in system:masters, so without this its own seeding update would be denied wherever denyModifyMemberClusterLabels is on. Placement policies and future APIs select on this label, so it lands on its own rather than behind the annotation-based placement work. Signed-off-by: Yetkin Timocin * chore: [FEP-0001] move the rolling update bounds and tombstone fixes to their own PRs Both are v1beta1 fixes for existing users and do not depend on this PR: the rolling update CEL and pattern bounds are #898, the tombstone unwrap in the resource watcher filter is #899. Signed-off-by: Yetkin Timocin * refactor: [FEP-0001] read generated policy specs through the API accessor #827 gives both policy kinds a GetSpec accessor, so the package's private policySpec switch, its nil branch, and the test that exercised the unreachable non-policy case go away. A generatedPolicy interface names what emptyPolicyForScope and desiredPolicy return: a client.Object with the accessor, so callers reach the spec without a type switch. Signed-off-by: Yetkin Timocin * feat: seed the cluster alias label on member clusters Add the kubefleet.dev/cluster-alias label, seeded from the cluster name by the member cluster controller when it is absent and never reasserted, so an admin can move an alias between clusters and a selector can follow a role rather than a fixed name. The member cluster webhook warns, but does not deny, when another cluster already holds the alias: two holders is the normal middle state of a handoff. The member cluster label guard treats the kubefleet.dev/ prefix like kubernetes-fleet.io/: the hub agent writes both and is not in system:masters, so without this its own seeding update would be denied wherever denyModifyMemberClusterLabels is on. Placement policies and future APIs select on this label, so it lands on its own rather than behind the annotation-based placement work. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] let only repairable drift through the generated policy watch The policy watches admitted every resourceVersion change, and a status write bumps it, so once the scheduling controllers write policy status each of their updates would have cost an uncached read of the annotated resource to learn that nothing drifted. The watch now admits creations, deletions, and updates to the generation, labels, or owner references, which is exactly what applyDesiredPolicy repairs. Also from review: the generatedPolicy interface duplicated #827's accessor, which already embeds client.Object, so the accessor is used directly; the defaults test reads the spec through it instead of a type assertion; the manager suite waits for Start to return so its assertion lands inside the spec tree; the unparsable owner apiVersion branch gains a case; and the unused ClusterScoped field on the shared fake lister, left over from the split-out watcher change, is restored to main. Signed-off-by: Yetkin Timocin * fix: [FEP-0001] feed annotated resources through a queue, not a channel Addresses review on the annotation placement controller. - Replace the channel source with a SourceQueue whose Add writes straight into the controller's own workqueue. The resource watcher has a fleet's events to deliver and no business blocking on this controller, which a full channel would have made it do; the queue also collapses repeated reports of one resource into a single reconcile. Resources reported before the manager starts are held, deduplicated, and flushed on Start. - Stop reading a resource under a version it was not reported under. A no-match now distinguishes a retired version from a gone kind: the kind still places, so its generated policy is kept and the request dropped, since API discovery re-reports the resource under the served version as a fresh request. Only a gone kind takes its policy with it. Falling back would have retargeted placement at a version the member clusters may not serve. - Categorize the parser's errors as user errors, keeping the message text intact because it is what the event on the annotated resource shows. - Record events through k8s.io/client-go/tools/events, which every other controller in the repo now uses; this was the last tools/record holdout, and its events would have landed in a different API group from the rest of the hub agent's. - Note at the resource selector that dependent resources are not selected, tracked in #927. Signed-off-by: Yetkin Timocin --------- Signed-off-by: Yetkin Timocin --- .../placement/v1alpha1/common.go | 55 + .../v1alpha1/placementpolicy_types.go | 20 +- .../v1alpha1/zz_generated.deepcopy.go | 2 +- ...ubefleet.dev_clusterplacementpolicies.yaml | 14 +- ...ement.kubefleet.dev_placementpolicies.yaml | 14 +- pkg/utils/apiresources.go | 7 + pkg/utils/naming/naming.go | 149 +++ pkg/utils/naming/naming_test.go | 249 ++++ .../annotationplacement/controller.go | 423 +++++++ .../controller_integration_test.go | 358 ++++++ .../annotationplacement/controller_test.go | 1055 +++++++++++++++++ .../manager_integration_test.go | 163 +++ .../controllers/annotationplacement/parser.go | 230 ++++ .../annotationplacement/parser_test.go | 485 ++++++++ .../controllers/annotationplacement/policy.go | 258 ++++ .../annotationplacement/policy_test.go | 463 ++++++++ .../annotationplacement/suite_test.go | 96 ++ .../controllers/annotationplacement/watch.go | 192 +++ .../annotationplacement/watch_test.go | 287 +++++ 19 files changed, 4506 insertions(+), 14 deletions(-) create mode 100644 pkg/utils/naming/naming.go create mode 100644 pkg/utils/naming/naming_test.go create mode 100644 pkg/v1/controllers/annotationplacement/controller.go create mode 100644 pkg/v1/controllers/annotationplacement/controller_integration_test.go create mode 100644 pkg/v1/controllers/annotationplacement/controller_test.go create mode 100644 pkg/v1/controllers/annotationplacement/manager_integration_test.go create mode 100644 pkg/v1/controllers/annotationplacement/parser.go create mode 100644 pkg/v1/controllers/annotationplacement/parser_test.go create mode 100644 pkg/v1/controllers/annotationplacement/policy.go create mode 100644 pkg/v1/controllers/annotationplacement/policy_test.go create mode 100644 pkg/v1/controllers/annotationplacement/suite_test.go create mode 100644 pkg/v1/controllers/annotationplacement/watch.go create mode 100644 pkg/v1/controllers/annotationplacement/watch_test.go diff --git a/apis/kubefleet.dev/placement/v1alpha1/common.go b/apis/kubefleet.dev/placement/v1alpha1/common.go index 9ccad8432..97ee0c0b5 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/common.go +++ b/apis/kubefleet.dev/placement/v1alpha1/common.go @@ -28,6 +28,61 @@ const ( WorkKind = "Work" ) +// The annotation and label keys that KubeFleet reserves for the placement APIs. +// +// Note that these keys carry the KubeFleet domain itself rather than the API group; they are +// set on arbitrary Kubernetes objects, not only on the objects of this group. +const ( + // KubeFleetPrefix is the domain that prefixes every reserved annotation and label key below, + // per the Kubernetes convention that reserves unprefixed keys for end users. + KubeFleetPrefix = "kubefleet.dev/" + + // ClusterSelectorsAnnotation is the annotation that requests annotation-based placement for the + // resource it is set on. Its value is a semicolon-separated list of cluster selectors, each of + // which is a comma-separated list of LABEL_KEY=LABEL_VALUE label matchers with an optional + // count=N|All directive, e.g.: + // + // kubefleet.dev/cluster-selectors: "env=staging,count=All;env=canary,region=eastus,count=1" + // + // KubeFleet keeps a PlacementPolicy (or ClusterPlacementPolicy) object in sync with the + // annotation for as long as it is present. + // + // The key names what the value holds rather than what KubeFleet does with it, matching the + // clusterSelectors field of the generated policy: the annotation is one way to write that field, + // and the two are read together often enough that they should not have to be translated. + // + // Within the annotation, `region` may be used in place of the well-known + // topology.kubernetes.io/region label key, and `alias` in place of the cluster alias label + // (placement/v1beta1's ClusterAliasLabel). + ClusterSelectorsAnnotation = KubeFleetPrefix + "cluster-selectors" +) + +// The labels that record, on a placement policy KubeFleet generated from an annotation, the +// resource whose annotation caused it to exist. +// +// These keys carry the API group rather than the bare KubeFleet domain, since unlike the keys +// above they are set on objects of this group. +// +// The labels exist so that a policy can be found with a List when its generated name is not known; +// the owner reference on the policy remains the authoritative record of where it came from. Note +// that ParentNameLabel is lossy: the name of a resource can run to 253 bytes while a label value +// stops at 63, so for a longer name the label holds a prefix and a hash instead, and selecting on +// it with the resource's own name matches nothing. +const ( + // ParentAPIGroupLabel holds the API group of the resource a policy was generated from. It is + // the empty string for resources in the core API group, and is always present, so that + // core-group resources can be selected as readily as any other. + ParentAPIGroupLabel = "placement.kubefleet.dev/parent-api-group" + + // ParentKindLabel holds the kind of the resource a policy was generated from, spelled as the + // kind itself is (Deployment, not deployment). + ParentKindLabel = "placement.kubefleet.dev/parent-kind" + + // ParentNameLabel holds the name of the resource a policy was generated from, shortened if it + // does not fit in a label value. + ParentNameLabel = "placement.kubefleet.dev/parent-name" +) + type ObjectReference struct { // The namespace of the referenced object. // diff --git a/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go b/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go index 2d54fc82f..4010f6293 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go @@ -149,7 +149,17 @@ type PlacementPolicySpec struct { Tolerations []Toleration `json:"tolerations,omitempty"` } -// +kubebuilder:validation:XValidation:rule="!has(self.minCount) || !has(self.count) || (type(self.count) == string && self.count == 'All') || (type(self.count) == int && self.minCount <= self.count) || (type(self.count) == string && self.count.matches('^[0-9]+$') && self.minCount <= int(self.count))",message="minCount must be less than or equal to count when count is not All" +// A string form of count is compared against minCount only when it is a numeric one of at most the +// three digits that the count field's own pattern permits -- the {1,3} bound below must move with +// that pattern's ceiling. Any other string ("All", or junk the pattern rejects) passes this rule +// vacuously, so that the field's own validation reports the real problem alone. The previous, +// unbounded digit guard let a string longer than an int64 reach int(), turning a plain pattern +// violation into an opaque evaluation error beside it. The digit class is spelled [0-9]{1,3} +// rather than mirroring the pattern's [1-9][0-9]{0,2} verbatim because the CEL cost estimator +// prices a regex by its length against an unbounded string -- an int-or-string is opaque to the +// sibling MaxLength -- and the longer spelling does not fit the budget. + +// +kubebuilder:validation:XValidation:rule="!has(self.minCount) || !has(self.count) || (type(self.count) == int && self.minCount <= self.count) || (type(self.count) == string && (!self.count.matches('^[0-9]{1,3}$') || self.minCount <= int(self.count)))",message="minCount must be less than or equal to count" type ClusterSelector struct { // A list of terms that form the selector. The terms are ORed, i.e., a cluster would match the selector // if it matches any of the terms. @@ -160,6 +170,12 @@ type ClusterSelector struct { // +kubebuilder:validation:MaxItems=5 Terms []ClusterLabelAndPropertySelectorTerm `json:"terms,omitempty"` + // The two validation markers below split the work by form: a pattern constrains only the string + // form of an int-or-string ("All", and digits arriving as a quoted string), so the CEL rule is + // what bounds the integer form. Without it, an unquoted count outside 1-999 is accepted while + // the same number in quotes is rejected. This block is deliberately detached from the field's + // doc comment: it is rationale for maintainers, not schema documentation for users. + // The desired number of clusters that KubeFleet should select based on the given terms. // // The default value is 1. To select all clusters that match the given terms, use the value "All". @@ -167,7 +183,9 @@ type ClusterSelector struct { // +kubebuilder:validation:Optional // +kubebuilder:default=1 // +kubebuilder:validation:XIntOrString + // +kubebuilder:validation:MaxLength=3 // +kubebuilder:validation:Pattern="^([1-9][0-9]{0,2}|All)$" + // +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 1 && self <= 999 : true",message="count must be between 1 and 999, or \"All\"" Count *intstr.IntOrString `json:"count,omitempty"` // The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able diff --git a/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go b/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go index 8f5a15a5d..d7f6bfc40 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go +++ b/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go @@ -21,7 +21,7 @@ limitations under the License. package v1alpha1 import ( - "k8s.io/apimachinery/pkg/apis/meta/v1" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" ) diff --git a/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml b/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml index 6e18f5a7e..782b672f7 100644 --- a/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml +++ b/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml @@ -63,8 +63,12 @@ spec: The desired number of clusters that KubeFleet should select based on the given terms. The default value is 1. To select all clusters that match the given terms, use the value "All". + maxLength: 3 pattern: ^([1-9][0-9]{0,2}|All)$ x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: count must be between 1 and 999, or "All" + rule: 'type(self) == int ? self >= 1 && self <= 999 : true' minCount: description: |- The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able @@ -238,13 +242,11 @@ spec: type: string type: object x-kubernetes-validations: - - message: minCount must be less than or equal to count when count - is not All + - message: minCount must be less than or equal to count rule: '!has(self.minCount) || !has(self.count) || (type(self.count) - == string && self.count == ''All'') || (type(self.count) == - int && self.minCount <= self.count) || (type(self.count) == - string && self.count.matches(''^[0-9]+$'') && self.minCount - <= int(self.count))' + == int && self.minCount <= self.count) || (type(self.count) + == string && (!self.count.matches(''^[0-9]{1,3}$'') || self.minCount + <= int(self.count)))' maxItems: 10 minItems: 1 type: array diff --git a/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml b/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml index 36a9c326e..303d06017 100644 --- a/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml +++ b/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml @@ -63,8 +63,12 @@ spec: The desired number of clusters that KubeFleet should select based on the given terms. The default value is 1. To select all clusters that match the given terms, use the value "All". + maxLength: 3 pattern: ^([1-9][0-9]{0,2}|All)$ x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: count must be between 1 and 999, or "All" + rule: 'type(self) == int ? self >= 1 && self <= 999 : true' minCount: description: |- The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able @@ -238,13 +242,11 @@ spec: type: string type: object x-kubernetes-validations: - - message: minCount must be less than or equal to count when count - is not All + - message: minCount must be less than or equal to count rule: '!has(self.minCount) || !has(self.count) || (type(self.count) - == string && self.count == ''All'') || (type(self.count) == - int && self.minCount <= self.count) || (type(self.count) == - string && self.count.matches(''^[0-9]+$'') && self.minCount - <= int(self.count))' + == int && self.minCount <= self.count) || (type(self.count) + == string && (!self.count.matches(''^[0-9]{1,3}$'') || self.minCount + <= int(self.count)))' maxItems: 10 minItems: 1 type: array diff --git a/pkg/utils/apiresources.go b/pkg/utils/apiresources.go index d81141dd6..f8fd1ad32 100644 --- a/pkg/utils/apiresources.go +++ b/pkg/utils/apiresources.go @@ -29,6 +29,7 @@ import ( metricsV1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1" clusterv1beta1 "github.com/kubefleet-dev/kubefleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" placementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" ) @@ -207,6 +208,12 @@ func NewResourceConfig(isAllowList bool) *ResourceConfig { // disable cluster group by default r.AddGroup(clusterv1beta1.GroupVersion.Group) + // Disable the placement.kubefleet.dev group wholesale: everything in it is KubeFleet's own + // bookkeeping (placement policies, bindings, snapshots, works, claims), including the policies + // that annotation-based placement generates. Treating those as placeable resources would let a + // placement that selects a whole namespace propagate KubeFleet's own generated objects. + r.AddGroup(kfplacementv1alpha1.GroupVersion.Group) + // disable some fleet networking resources r.AddGroupKind(serviceImportGK) r.AddGroupKind(trafficManagerProfileGK) diff --git a/pkg/utils/naming/naming.go b/pkg/utils/naming/naming.go new file mode 100644 index 000000000..5d0eea89c --- /dev/null +++ b/pkg/utils/naming/naming.go @@ -0,0 +1,149 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package naming derives the names and label values of objects that KubeFleet generates on a +// user's behalf. +// +// A controller that names an object after another one runs into the same two problems every time: +// a name legal for a Kubernetes object is not always legal where the controller puts it (object +// names run to 253 bytes, label values stop at 63), and shortening a name to fit can both produce +// a syntactically invalid result and collapse two distinct identities onto one string. Getting +// either wrong fails at the API server rather than at the call site, which surfaces as a +// reconciler that retries forever while the user sees nothing happen at all. +// +// The helpers here exist so that the answer is written once. They are deliberately primitive: +// callers compose the identity that is hashed, since only the caller knows what makes one of its +// objects distinct from another. +package naming + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "strings" + + "k8s.io/apimachinery/pkg/util/validation" +) + +const ( + // HashLength is the number of hexadecimal characters of Hash's output, i.e. 64 bits of a + // SHA-256 sum. A narrower hash is tempting and wrong: these hashes are what keep two distinct + // objects from generating one name, and whichever of them loses the race is not merely + // misnamed, it silently never gets its object at all. + HashLength = 16 + + // labelValuePrefixMaxLength is how much of an over-long value survives in front of the hash + // that LabelValue appends, allowing for the separator between the two. + labelValuePrefixMaxLength = validation.LabelValueMaxLength - HashLength - 1 + + // separators are the characters a Kubernetes name may contain but may not begin or end with. + separators = "-_." +) + +// Hash returns a short, stable, collision-resistant digest of s. +// +// Callers should hash the whole identity of an object, never the shortened form of it that appears +// in a generated name: shortening is precisely what collapses two identities into one, so a hash +// taken afterwards would agree where the identities differ. +func Hash(s string) string { + sum := sha256.Sum256([]byte(s)) + return hex.EncodeToString(sum[:])[:HashLength] +} + +// TrimSeparators removes the separators that a name may not end with, which truncating an +// otherwise valid name can leave exposed. +// +// The result can be empty only if the fragment consisted entirely of separators; since Kubernetes +// object names must begin with an alphanumeric character, a fragment taken from the front of one +// never can. +func TrimSeparators(fragment string) string { + return strings.TrimRight(fragment, separators) +} + +// Truncate shortens s to at most maxLength bytes, leaving a result that is still legal at the end +// of a Kubernetes name. +// +// It does not make the result unique; a caller that truncates must append Hash of the full, +// untruncated identity for that. +// +// A budget of zero or less leaves nothing, which is returned as the empty string rather than as a +// panic: this package exists so that a caller's arithmetic mistake surfaces as a name it can +// inspect, not as a crash in a reconcile loop. +func Truncate(s string, maxLength int) string { + if maxLength <= 0 { + return "" + } + if len(s) <= maxLength { + return s + } + return TrimSeparators(s[:maxLength]) +} + +// Sanitize renders s legal as a single label of a generated DNS-1123 name: it lowercases the value +// and replaces every character outside [a-z0-9-] with a dash. A name legal for its own API but not +// as a Kubernetes object name -- an RBAC name like system:aggregate-to-admin, whose colon a DNS-1123 +// name forbids -- becomes usable this way. +// +// The dot is mapped to a dash along with everything else, rather than kept as a label separator: a +// dot is only legal between two alphanumerics, so preserving it would let an invalid character next +// to it (or another dot) produce an empty or dash-bounded label that the API server still rejects -- +// the very failure Sanitize exists to prevent. Its one caller joins the sanitized parts with dashes +// anyway, so no dot is needed to keep them apart. +// +// It is lossy: distinct inputs can sanitize to the same string, so a caller must still append Hash +// of the full, unsanitized identity for uniqueness. +func Sanitize(s string) string { + return TrimSeparators(strings.TrimLeft(mapInvalid(strings.ToLower(s), isDNS1123Char), separators)) +} + +// sanitizeLabelValue is Sanitize's counterpart for a label value, whose legal character set is +// wider (it keeps case and underscores). It exists so LabelValue never returns a value the API +// server would reject; like Sanitize it is lossy and relies on a hash for identity. +func sanitizeLabelValue(value string) string { + return TrimSeparators(strings.TrimLeft(mapInvalid(value, isLabelValueChar), separators)) +} + +func mapInvalid(s string, valid func(rune) bool) string { + return strings.Map(func(r rune) rune { + if valid(r) { + return r + } + return '-' + }, s) +} + +func isDNS1123Char(r rune) bool { + return (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') || r == '-' +} + +func isLabelValueChar(r rune) bool { + return (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '-' || r == '_' || r == '.' +} + +// LabelValue renders value so that it fits in a label value, sanitizing characters the API server +// would reject and shortening it to a prefix and a hash of the whole when it is too long. +// +// The result is lossy by design -- both sanitization and shortening can collapse distinct values -- +// and callers must treat it that way: selecting on the label with the original value matches nothing +// once it has been rewritten, so anything that needs to resolve an exact identity has to read it +// from a field that has neither a character nor a length limit. +func LabelValue(value string) string { + sanitized := sanitizeLabelValue(value) + if len(sanitized) <= validation.LabelValueMaxLength { + return sanitized + } + return fmt.Sprintf("%s-%s", TrimSeparators(sanitized[:labelValuePrefixMaxLength]), Hash(value)) +} diff --git a/pkg/utils/naming/naming_test.go b/pkg/utils/naming/naming_test.go new file mode 100644 index 000000000..5c0ffae66 --- /dev/null +++ b/pkg/utils/naming/naming_test.go @@ -0,0 +1,249 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package naming + +import ( + "strings" + "testing" + + "k8s.io/apimachinery/pkg/util/validation" +) + +func TestHash(t *testing.T) { + testCases := []struct { + name string + a, b string + want bool + }{ + {name: "same input hashes the same", a: "apps/Deployment/prod/app", b: "apps/Deployment/prod/app", want: true}, + {name: "namespace is part of the identity", a: "apps/Deployment/prod/app", b: "apps/Deployment/staging/app", want: false}, + {name: "api group is part of the identity", a: "apps/Widget/prod/app", b: "example.com/Widget/prod/app", want: false}, + {name: "separator placement matters", a: "apps/Deployment/prod/app", b: "apps/Deployment/prod-app/", want: false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotA, gotB := Hash(tc.a), Hash(tc.b) + if (gotA == gotB) != tc.want { + t.Errorf("Hash(%q) = %v, Hash(%q) = %v, want equal to be %v", tc.a, gotA, tc.b, gotB, tc.want) + } + }) + } +} + +// TestHashShape pins the properties every generated name depends on: a fixed width, and a value +// made only of characters legal anywhere in a Kubernetes name. +func TestHashShape(t *testing.T) { + for _, input := range []string{"", "app", strings.Repeat("x", 4096), "ünïcödé/名前"} { + got := Hash(input) + if len(got) != HashLength { + t.Errorf("len(Hash(%q)) = %d, want %d", input, len(got), HashLength) + } + if strings.Trim(got, "0123456789abcdef") != "" { + t.Errorf("Hash(%q) = %v, want only lowercase hexadecimal characters", input, got) + } + } +} + +func TestTrimSeparators(t *testing.T) { + testCases := []struct { + name string + fragment string + want string + }{ + {name: "nothing to trim", fragment: "app", want: "app"}, + {name: "trailing dash", fragment: "app-", want: "app"}, + {name: "trailing dot", fragment: "app.", want: "app"}, + {name: "trailing underscore", fragment: "app_", want: "app"}, + {name: "run of separators", fragment: "app-._-", want: "app"}, + {name: "leading separators are left alone", fragment: "-app", want: "-app"}, + {name: "separators in the middle are left alone", fragment: "my-app.v2", want: "my-app.v2"}, + {name: "entirely separators", fragment: "---", want: ""}, + {name: "empty", fragment: "", want: ""}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := TrimSeparators(tc.fragment); got != tc.want { + t.Errorf("TrimSeparators(%q) = %v, want %v", tc.fragment, got, tc.want) + } + }) + } +} + +func TestTruncate(t *testing.T) { + testCases := []struct { + name string + s string + maxLength int + want string + }{ + {name: "shorter than the limit", s: "app", maxLength: 10, want: "app"}, + {name: "exactly at the limit", s: "app", maxLength: 3, want: "app"}, + {name: "over the limit", s: "application", maxLength: 4, want: "appl"}, + // The cut landing on a separator is the case that produced an invalid generated name in + // practice: the character after the cut would begin a new label. + {name: "cut lands on a dash", s: "my-application", maxLength: 3, want: "my"}, + {name: "cut lands on a dot", s: "my.application", maxLength: 3, want: "my"}, + {name: "cut lands inside a run of separators", s: "my-._application", maxLength: 5, want: "my"}, + {name: "cut lands just past a run of separators", s: "my-._application", maxLength: 6, want: "my-._a"}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := Truncate(tc.s, tc.maxLength) + if got != tc.want { + t.Errorf("Truncate(%q, %d) = %v, want %v", tc.s, tc.maxLength, got, tc.want) + } + if len(got) > tc.maxLength { + t.Errorf("len(Truncate(%q, %d)) = %d, want at most %d", tc.s, tc.maxLength, len(got), tc.maxLength) + } + }) + } +} + +func TestLabelValue(t *testing.T) { + testCases := []struct { + name string + value string + want string + }{ + {name: "short value is kept as is", value: "app", want: "app"}, + { + name: "value at the limit is kept as is", + value: strings.Repeat("a", validation.LabelValueMaxLength), + want: strings.Repeat("a", validation.LabelValueMaxLength), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := LabelValue(tc.value); got != tc.want { + t.Errorf("LabelValue(%q) = %v, want %v", tc.value, got, tc.want) + } + }) + } +} + +// TestLabelValueShortening covers the values that do not fit, where the result is not a value the +// test can spell out but a set of properties it must hold: it has to be a legal label value, it +// has to stay within the limit, and it has to distinguish inputs that share a prefix. +func TestLabelValueShortening(t *testing.T) { + long := strings.Repeat("a", validation.LabelValueMaxLength+1) + testCases := []string{ + long, + long + "b", + // A value whose truncation point falls on a separator, which must not survive into the + // shortened form. + strings.Repeat("a", labelValuePrefixMaxLength-1) + "-" + strings.Repeat("b", 40), + strings.Repeat("a", labelValuePrefixMaxLength-1) + "." + strings.Repeat("c", 40), + } + + seen := make(map[string]string, len(testCases)) + for _, value := range testCases { + got := LabelValue(value) + if len(got) > validation.LabelValueMaxLength { + t.Errorf("len(LabelValue(%q)) = %d, want at most %d", value, len(got), validation.LabelValueMaxLength) + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("LabelValue(%q) = %v, want a valid label value: %s", value, got, strings.Join(errs, "; ")) + } + if previous, collided := seen[got]; collided { + t.Errorf("LabelValue(%q) = %v, want a value distinct from that of %q", value, got, previous) + } + seen[got] = value + } +} + +// TestSanitize pins the transform that lets a name legal for its own API but not as a Kubernetes +// object name pass through into a generated name: every result must be legal as a DNS-1123 name +// segment, and the characters a name already allows must survive unchanged. +func TestSanitize(t *testing.T) { + testCases := []struct { + name string + value string + want string + }{ + {name: "already legal", value: "my-app", want: "my-app"}, + {name: "uppercase is lowercased", value: "MyApp", want: "myapp"}, + // The RBAC name that motivated the sanitizer: a colon is legal in the name but not in a + // Kubernetes object name. + {name: "rbac name with a colon", value: "system:aggregate-to-admin", want: "system-aggregate-to-admin"}, + {name: "underscores become dashes", value: "my_app", want: "my-app"}, + // A dot becomes a dash like any other separator: kept as a dot, an invalid character beside it + // would leave a label bounded by a dash, which the API server rejects. + {name: "dot becomes a dash", value: "my.app", want: "my-app"}, + {name: "dot next to an illegal character does not leave a bad label", value: "my.:app", want: "my--app"}, + {name: "adjacent dots do not leave an empty label", value: "a..b", want: "a--b"}, + {name: "leading separators are trimmed", value: ":::app", want: "app"}, + {name: "trailing separators are trimmed", value: "app:::", want: "app"}, + {name: "entirely illegal collapses to empty", value: ":::", want: ""}, + {name: "empty", value: "", want: ""}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := Sanitize(tc.value) + if got != tc.want { + t.Errorf("Sanitize(%q) = %v, want %v", tc.value, got, tc.want) + } + if got != "" { + if errs := validation.IsDNS1123Subdomain(got); len(errs) > 0 { + t.Errorf("Sanitize(%q) = %v, want a valid DNS-1123 subdomain: %s", tc.value, got, strings.Join(errs, "; ")) + } + } + }) + } +} + +// TestLabelValueSanitizes covers values that carry characters a label value forbids. The wider +// label-value character set keeps case and underscores that Sanitize would drop, but a colon still +// has to go, and the result must be a value the API server accepts. +func TestLabelValueSanitizes(t *testing.T) { + testCases := []struct { + name string + value string + want string + }{ + {name: "case and underscores are kept", value: "My_App", want: "My_App"}, + {name: "colon becomes a dash", value: "system:aggregate-to-admin", want: "system-aggregate-to-admin"}, + {name: "leading separators are trimmed", value: ":app", want: "app"}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := LabelValue(tc.value) + if got != tc.want { + t.Errorf("LabelValue(%q) = %v, want %v", tc.value, got, tc.want) + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("LabelValue(%q) = %v, want a valid label value: %s", tc.value, got, strings.Join(errs, "; ")) + } + }) + } +} + +// TestTruncateNonPositiveBudget covers a caller whose own arithmetic left nothing for the value. +// The package exists to keep budget mistakes from reaching the API server; it must not turn one +// into a panic inside a reconcile loop either. +func TestTruncateNonPositiveBudget(t *testing.T) { + for _, maxLength := range []int{0, -1, -64} { + if got := Truncate("application", maxLength); got != "" { + t.Errorf("Truncate(%q, %d) = %v, want the empty string", "application", maxLength, got) + } + } +} diff --git a/pkg/v1/controllers/annotationplacement/controller.go b/pkg/v1/controllers/annotationplacement/controller.go new file mode 100644 index 000000000..fdef216b8 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/controller.go @@ -0,0 +1,423 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "errors" + "fmt" + "time" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/tools/events" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +// The reasons of the events this controller records on the annotated resource. Each reason is the +// same whether the generated object is a PlacementPolicy or a ClusterPlacementPolicy -- the reason +// names the action, and the event message names the concrete kind. +// +// The events are recorded on the resource the user annotated rather than on the policy generated +// from it, because the resource is where a user who has just run kubectl annotate is looking, and +// because a rejected annotation generates no policy for an event to be attached to. +const ( + // EventReasonPolicyCreated is recorded when a generated policy is created for a resource. + EventReasonPolicyCreated = "PlacementPolicyCreated" + // EventReasonPolicyUpdated is recorded when an annotation change reaches its generated policy. + EventReasonPolicyUpdated = "PlacementPolicyUpdated" + // EventReasonPolicyDeleted is recorded when the policy generated for a resource is deleted -- + // because the annotation was removed, or because the resource stopped being eligible for + // placement. The event's message names which. + EventReasonPolicyDeleted = "PlacementPolicyDeleted" + // EventReasonInvalidAnnotation is recorded when an annotation cannot be parsed. It is a warning + // rather than an error on the queue: no amount of retrying makes a malformed annotation parse, + // so the event is the only way the user learns that the placement they asked for is not running. + EventReasonInvalidAnnotation = "InvalidClusterSelectorsAnnotation" + // EventReasonPolicyConflict is recorded when a policy already exists at a resource's generated + // name but is not one this controller generated. It is a warning: the controller neither + // overwrites nor deletes the pre-existing policy, so the placement the annotation asks for is not + // running, and the event is how the user learns why. + EventReasonPolicyConflict = "PlacementPolicyConflict" +) + +// errForeignPolicy reports that a policy occupies a resource's generated name but is not one this +// controller generated. +// +// It is returned as an error rather than swallowed with a fixed requeue so that the source is +// retried with the queue's exponential backoff: nothing else brings it back, since the blocking +// policy carries no owner reference to the source and the annotation does not change, yet a +// conflict left in place must not be re-examined at a fixed, busy pace forever. +var errForeignPolicy = errors.New("a policy at the generated name was not generated from the annotation") + +// errRetiredVersion reports that the version a request names is no longer served while its kind +// still is. The request cannot be retried as it stands -- the resource is re-reported under the +// served version as a different request, which is what reconciles it -- so it is dropped rather +// than requeued, and the generated policy is left alone. +var errRetiredVersion = errors.New("the version the resource was reported under is no longer served") + +// Request identifies the resource a reconciliation is for. +// +// A plain namespaced name would not do: this controller reconciles resources of any kind the hub +// agent watches, and the name alone does not say which API to read the resource from. +type Request struct { + schema.GroupVersionKind + types.NamespacedName +} + +// RequestFor returns the request that reconciles the given resource, which must carry its kind. +func RequestFor(object client.Object) Request { + return Request{ + GroupVersionKind: object.GetObjectKind().GroupVersionKind(), + NamespacedName: client.ObjectKeyFromObject(object), + } +} + +// String formats the request for logs. +func (r Request) String() string { + return fmt.Sprintf("%s %s", r.GroupVersionKind.String(), r.NamespacedName.String()) +} + +// Reconciler keeps the placement policy generated from a resource's cluster-selectors annotation in +// sync with that annotation. +// +// Its queue holds requests for resources of any kind the hub agent watches, not for the generated +// policies themselves; an event on a generated policy is mapped back to the resource it came from. +type Reconciler struct { + // Client reads and writes the generated placement policies. + // + // It may be backed by the manager's cache: the generated policies are watched through the same + // cache, so a policy event is never delivered ahead of the state this reads, and a write that + // races a change the cache has yet to see fails with a conflict and is retried. The one thing + // the cache can lag is this controller's own writes, which is why an already-exists error on a + // create is treated as transient rather than as a failure. + client.Client + + // APIReader reads the annotated resources straight from the API server. + // + // The resources are of any kind the hub agent watches, and a cache-backed read of a kind the + // cache has not seen would start an informer for it -- for a kind that cannot be listed or + // watched, one that never syncs. Only annotated resources and the owners of generated policies + // reach this controller's queue, so the direct read is not on the hot path of every watched + // resource in the cluster. + APIReader client.Reader + + // RESTMapper resolves the kind a request names into the resource it is read through, and the + // scope that decides whether the read is namespaced. + RESTMapper meta.RESTMapper + + // Recorder records the outcome of a reconciliation on the annotated resource. + Recorder events.EventRecorder + + // ShouldPlace reports whether a resource is one KubeFleet places at all, mirroring the filter + // the resource watcher applies to its events. For example, a Deployment in a user namespace + // should place (true); a ConfigMap in a skipped namespace like kube-system, or a ReplicaSet a + // Deployment already owns, should not (false). + // + // The reconciler applies it again because it can be reached for a resource the watcher would + // filter out: the watcher reports a resource that stops passing its filter as a deletion, and + // the generated policy watch enqueues whatever a policy names as its owner. A resource that + // fails the check has its generated policy deleted. + // + // Eligibility can change over a resource's life -- an owned ReplicaSet becomes eligible again + // once orphaned, a resource stays ineligible while in a skipped namespace. The transitions that + // matter are edits to the resource itself (its owner references, its labels), so each fires an + // event that re-runs this check; a resource that becomes eligible again and still carries the + // annotation has its policy regenerated on that event. + // + // Left nil, every resource is eligible. + ShouldPlace func(source *unstructured.Unstructured) (bool, error) +} + +// Reconcile brings the generated policy for one resource in line with that resource's annotation. +func (r *Reconciler) Reconcile(ctx context.Context, req Request) (ctrl.Result, error) { + startTime := time.Now() + klog.V(2).InfoS("Reconciling annotation-based placement", "obj", req) + defer func() { + klog.V(2).InfoS("Annotation-based placement reconciliation loop ends", "obj", req, "latency", time.Since(startTime).Milliseconds()) + }() + + source, err := r.sourceObject(ctx, req) + switch { + case errors.Is(err, errRetiredVersion): + // Dropped rather than retried: this request names a version that is gone, and no number of + // retries brings it back. API discovery asks for server-preferred versions, so the resource + // is re-reported under the served one as a fresh request, and that is what reconciles it. + // The generated policy stands untouched in the meantime, since its kind still places. + klog.V(2).InfoS("Dropped a request naming a version that is no longer served", "obj", req) + return ctrl.Result{}, nil + case apierrors.IsNotFound(err), meta.IsNoMatchError(err): + // The resource is gone, or its whole kind is (the CRD was removed; the generated policy + // watch enqueues a stale policy's owner). The generated policy is KubeFleet's own object, + // so it is always deleted here rather than left to garbage collection, which would wait on + // every owner reference -- including the foreign ones the merge deliberately preserves. + _, deleted, err := r.deleteGeneratedPolicy(ctx, req.GroupVersionKind, req.Namespace, req.Name) + switch { + case err != nil: + klog.ErrorS(err, "Failed to delete the policy generated for a resource that is gone", kferrors.Args(err, "obj", req)...) + case deleted: + klog.V(2).InfoS("Deleted the policy generated for a resource that is gone", "obj", req) + } + return ctrl.Result{}, err + case err != nil: + klog.ErrorS(err, "Failed to get the annotated resource", kferrors.Args(err)...) + return ctrl.Result{}, err + } + + if r.ShouldPlace != nil { + eligible, err := r.ShouldPlace(source) + if err != nil { + klog.ErrorS(err, "Failed to decide whether the resource is eligible for placement", "obj", req) + return ctrl.Result{}, err + } + if !eligible { + // A resource KubeFleet does not place cannot keep a generated policy either; without + // this, a resource that stops being eligible (for instance a ReplicaSet adopted by a + // Deployment) would leave its policy behind, invisible to the watcher from then on. + return ctrl.Result{}, r.deletePolicy(ctx, source, "the resource is not eligible for placement") + } + } + + value, annotated := source.GetAnnotations()[kfplacementv1alpha1.ClusterSelectorsAnnotation] + if !annotated { + return ctrl.Result{}, r.deletePolicy(ctx, source, "the "+kfplacementv1alpha1.ClusterSelectorsAnnotation+" annotation was removed") + } + + selectors, err := parseClusterSelectors(value) + if err != nil { + // The annotation is the user's to fix, so the failure is reported to the user and the + // request is dropped. Any policy generated from an earlier, valid annotation is deliberately + // left standing: the desired state is now unknown, and tearing down a running placement is + // a worse answer to a typo than leaving the last one the user did express. + klog.V(2).InfoS("The annotation cannot be parsed", "obj", req, "err", err) + r.Recorder.Eventf(source, nil, corev1.EventTypeWarning, EventReasonInvalidAnnotation, "ParseAnnotation", + "The %s annotation is not valid and no placement policy was generated from it: %s", kfplacementv1alpha1.ClusterSelectorsAnnotation, err) + return ctrl.Result{}, nil + } + return ctrl.Result{}, r.syncPolicy(ctx, source, selectors) +} + +// sourceObject reads the annotated resource a request refers to. +// +// A not-found error and a no-match error (the kind itself is unknown) are returned as they are, so +// that the caller can tell the two apart from a read that failed. +func (r *Reconciler) sourceObject(ctx context.Context, req Request) (*unstructured.Unstructured, error) { + // The resource is read under the version the request names and no other. Falling back to + // whatever version is served now would quietly retarget placement at a version the member + // clusters may not serve, turning a clear failure here into a confusing one later; and it + // would buy nothing, since API discovery asks for server-preferred versions, so a resource + // seen under a retired version is re-reported under the current one and reconciled again. + restMapping, err := r.RESTMapper.RESTMapping(req.GroupKind(), req.Version) + if err != nil { + if meta.IsNoMatchError(err) { + // A no-match means either that the kind is gone or that only this version is. The two + // call for opposite answers -- a gone kind should take its generated policy with it, + // while a retired version should not, since the kind still places -- so they are told + // apart by asking for the kind without a version. Only the truly gone kind is reported + // as a no-match; a retired version is transient, and the re-report that discovery + // produces under the served version reconciles it. + if _, kindErr := r.RESTMapper.RESTMapping(req.GroupKind()); kindErr == nil { + return nil, errRetiredVersion + } + return nil, err + } + return nil, kferrors.NewUnexpectedError(err, "failed to resolve the resource of the annotated object", "obj", req) + } + + source := &unstructured.Unstructured{} + source.SetGroupVersionKind(restMapping.GroupVersionKind) + // Scope comes from the mapping rather than from the request, since the mapping is what the read + // actually goes through. A cluster-scoped read must not name a namespace. + key := client.ObjectKey{Name: req.Name} + if restMapping.Scope.Name() != meta.RESTScopeNameRoot { + key.Namespace = req.Namespace + } + if err := r.APIReader.Get(ctx, key, source); err != nil { + if apierrors.IsNotFound(err) { + return nil, err + } + return nil, kferrors.NewAPIServerError(err, "failed to get the annotated resource", false, "obj", req) + } + return source, nil +} + +// syncPolicy creates or updates the policy generated from a resource's annotation. +func (r *Reconciler) syncPolicy(ctx context.Context, source *unstructured.Unstructured, selectors []kfplacementv1alpha1.ClusterSelector) error { + desired := desiredPolicy(source, selectors) + policy := emptyPolicyForScope(source.GetNamespace()) + policy.SetName(desired.GetName()) + policy.SetNamespace(desired.GetNamespace()) + + // The mutation runs only once the read behind it has succeeded, so whether it ran tells a failed + // read apart from a failed write. + read := false + result, err := controllerutil.CreateOrUpdate(ctx, r.Client, policy, func() error { + read = true + if policy.GetResourceVersion() != "" && !isGeneratedFor(policy, source.GroupVersionKind(), source.GetName(), r.Scheme()) { + // A policy already occupies this resource's generated name, but it is not one this + // controller produced -- a user or another tool authored a policy that happens to collide + // with the deterministic name. Overwriting its spec would silently commandeer it, so it is + // left exactly as found and the conflict is surfaced to the user instead. The name mixes in + // a hash of the resource's identity, so a genuine collision is near impossible and almost + // always means the name was chosen deliberately. + return errForeignPolicy + } + return applyDesiredPolicy(policy, desired, source, r.Scheme()) + }) + switch { + case errors.Is(err, errForeignPolicy): + klog.V(2).InfoS("A policy at the generated name was not generated by this controller; leaving it untouched", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + r.Recorder.Eventf(source, nil, corev1.EventTypeWarning, EventReasonPolicyConflict, "GeneratePolicy", + "A %s named %s already exists and was not generated from the %s annotation; it was left unchanged", generatedPolicyKind(source.GetNamespace()), policy.GetName(), kfplacementv1alpha1.ClusterSelectorsAnnotation) + return kferrors.NewUserError(err, "the generated name is taken", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + case apierrors.IsAlreadyExists(err): + // The cache has yet to see a policy this controller created moments ago, and an unrelated + // second event for the source arrived in that window. The next pass, after the queue's + // backoff, reads the policy the watch has delivered by then. + klog.V(2).InfoS("The generated placement policy was created but is not in the cache yet; retrying", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + return kferrors.NewTransientError(err, "the generated placement policy is not in the cache yet", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + case err != nil: + err = kferrors.NewAPIServerError(err, "failed to create or update the generated placement policy", !read, "obj", klog.KObj(source), "policy", klog.KObj(policy)) + klog.ErrorS(err, "Failed to sync the generated placement policy", kferrors.Args(err)...) + return err + } + + switch result { + case controllerutil.OperationResultCreated: + klog.V(2).InfoS("Created the generated placement policy", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + r.Recorder.Eventf(source, nil, corev1.EventTypeNormal, EventReasonPolicyCreated, "GeneratePolicy", + "Created the %s %s from the %s annotation", generatedPolicyKind(source.GetNamespace()), policy.GetName(), kfplacementv1alpha1.ClusterSelectorsAnnotation) + case controllerutil.OperationResultUpdated: + klog.V(2).InfoS("Updated the generated placement policy", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + r.Recorder.Eventf(source, nil, corev1.EventTypeNormal, EventReasonPolicyUpdated, "GeneratePolicy", + "Updated the %s %s from the %s annotation", generatedPolicyKind(source.GetNamespace()), policy.GetName(), kfplacementv1alpha1.ClusterSelectorsAnnotation) + default: + klog.V(3).InfoS("The generated placement policy is already up to date", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + } + return nil +} + +// deletePolicy removes the policy generated for a resource that should not have one -- because the +// annotation was removed, or because the resource is not eligible for placement -- and tells the +// user which through an event carrying the given cause. +// +// The cause is a plain string, never a format: keeping the only format string in the Eventf call +// below constant is what lets go vet check it. +func (r *Reconciler) deletePolicy(ctx context.Context, source *unstructured.Unstructured, cause string) error { + name, deleted, err := r.deleteGeneratedPolicy(ctx, source.GroupVersionKind(), source.GetNamespace(), source.GetName()) + if err != nil { + klog.ErrorS(err, "Failed to delete the generated placement policy", kferrors.Args(err, "obj", klog.KObj(source), "policy", klog.KRef(source.GetNamespace(), name))...) + return err + } + if !deleted { + // The common case by far: a resource nobody annotated. + return nil + } + klog.V(2).InfoS("Deleted the generated placement policy", "obj", klog.KObj(source), "policy", klog.KRef(source.GetNamespace(), name)) + r.Recorder.Eventf(source, nil, corev1.EventTypeNormal, EventReasonPolicyDeleted, "DeletePolicy", "Deleted the %s %s because %s", generatedPolicyKind(source.GetNamespace()), name, cause) + return nil +} + +// deleteGeneratedPolicy deletes the policy generated for the given resource identity, reporting the +// policy's name and whether this pass performed the deletion. The name is returned so a caller that +// logs or records an event about the deletion need not derive it a second time. +func (r *Reconciler) deleteGeneratedPolicy(ctx context.Context, gvk schema.GroupVersionKind, namespace, name string) (string, bool, error) { + actual := emptyPolicyForScope(namespace) + policyName := generatedPolicyName(gvk, namespace, name) + + err := r.Get(ctx, client.ObjectKey{Namespace: namespace, Name: policyName}, actual) + switch { + case apierrors.IsNotFound(err): + return policyName, false, nil + case err != nil: + return policyName, false, kferrors.NewAPIServerError(err, "failed to get the generated placement policy", true) + } + + if !isGeneratedFor(actual, gvk, name, r.Scheme()) { + // A policy occupies the generated name but this controller did not create it, so deleting it + // would destroy a user's or another tool's object. It is never this controller's to remove; + // the deletion is declined and reported as though nothing was there to delete. + klog.V(2).InfoS("A policy at the generated name was not generated by this controller; declining to delete it", "policy", klog.KRef(namespace, policyName)) + return policyName, false, nil + } + + // The delete carries the resource version the read returned as a precondition, so it removes only + // the exact object this pass read and confirmed was one it generated. Between that read and here + // the policy could be replaced -- deleted and a hand-authored one created at the same name, or + // overwritten in place with its provenance stripped -- and an unconditioned delete, which targets + // the name alone, would then remove whatever now sits there. The precondition turns that into a + // conflict instead; the conflict requeues, and the next pass reads the current object and declines + // it if it is no longer one this controller generated. + resourceVersion := actual.GetResourceVersion() + if err := r.Delete(ctx, actual, client.Preconditions{ResourceVersion: &resourceVersion}); err != nil { + if apierrors.IsNotFound(err) { + // The read above raced a deletion that already happened -- typically this controller's + // own, re-entered through the generated policy watch moments later. Nothing was deleted + // here, and reporting otherwise would log, and on some paths announce to the user, a + // deletion that this pass did not perform. + return policyName, false, nil + } + return policyName, false, kferrors.NewAPIServerError(err, "failed to delete the generated placement policy", false) + } + return policyName, true, nil +} + +// applyDesiredPolicy brings a live generated policy in line with the desired one. +// +// Only what this controller generates is overwritten: the spec, the provenance labels, and the owner +// reference to the annotated resource. Labels and annotations that something else added are left +// alone, so that a generated policy can be labelled by an operator or a GitOps tool without this +// controller and that tool taking turns undoing each other. Whether anything changed is for the +// caller to judge by comparing the object before and after. +func applyDesiredPolicy(actual, desired kfplacementv1alpha1.PlacementPolicyAccessor, source *unstructured.Unstructured, scheme *runtime.Scheme) error { + desired.GetSpec().DeepCopyInto(actual.GetSpec()) + + labels := actual.GetLabels() + if labels == nil { + labels = make(map[string]string, len(desired.GetLabels())) + } + for key, value := range desired.GetLabels() { + labels[key] = value + } + actual.SetLabels(labels) + + // The owner reference is matched on the source's group, kind, and name and replaced in place, + // so a reference left by a source that was deleted and recreated under the same name (a new UID) + // or written under a since-retired version is corrected rather than joined by a second one. + return controllerutil.SetOwnerReference(source, actual, scheme) +} + +// HasClusterSelectorsAnnotation reports whether an object carries the annotation this controller +// acts on. It is what keeps every unrelated resource in the cluster out of this controller's queue: +// whichever event source feeds the controller is expected to call it on every event for every +// watched resource, and to enqueue only the objects it reports true for. +func HasClusterSelectorsAnnotation(object metav1.Object) bool { + _, found := object.GetAnnotations()[kfplacementv1alpha1.ClusterSelectorsAnnotation] + return found +} diff --git a/pkg/v1/controllers/annotationplacement/controller_integration_test.go b/pkg/v1/controllers/annotationplacement/controller_integration_test.go new file mode 100644 index 000000000..22d46bef2 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/controller_integration_test.go @@ -0,0 +1,358 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + eventuallyTimeout = "10s" + eventuallyTimeoutDuration = 10 * time.Second + eventuallyInterval = "250ms" +) + +var configMapCount int + +// annotate sets, changes, or (with an empty value) removes the annotation on an object. +func annotate(object client.Object, value string) { + Expect(hubClient.Get(ctx, client.ObjectKeyFromObject(object), object)).Should(Succeed()) + annotations := object.GetAnnotations() + if annotations == nil { + annotations = map[string]string{} + } + if value == "" { + delete(annotations, kfplacementv1alpha1.ClusterSelectorsAnnotation) + } else { + annotations[kfplacementv1alpha1.ClusterSelectorsAnnotation] = value + } + object.SetAnnotations(annotations) + Expect(hubClient.Update(ctx, object)).Should(Succeed()) +} + +// reconcile runs one pass of the reconciler over an object, as the resource watcher would. +func reconcile(gvk schema.GroupVersionKind, object client.Object) error { + _, err := reconciler.Reconcile(ctx, requestFor(gvk, object.GetNamespace(), object.GetName())) + return err +} + +// generatedPolicyFor reads back the policy generated for an object, whichever scope it has. +func generatedPolicyFor(gvk schema.GroupVersionKind, object client.Object) (kfplacementv1alpha1.PlacementPolicyAccessor, error) { + namespace := object.GetNamespace() + policy := emptyPolicyForScope(namespace) + name := generatedPolicyName(gvk, namespace, object.GetName()) + err := hubClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, policy) + return policy, err +} + +// drainEvents returns the reasons of every event recorded so far, clearing the recorder. +func drainEvents() []string { + reasons := []string{} + for { + select { + case event := <-eventRecorder.Events: + fields := strings.SplitN(event, " ", 3) + if len(fields) >= 2 { + reasons = append(reasons, fields[1]) + } + default: + return reasons + } + } +} + +// The API server itself enforces the count bounds on both forms of the int-or-string: the Pattern +// marker covers the string form, and the CEL rule covers the integer form, which a pattern alone +// leaves unbounded. The parser mirrors the same bounds for annotations; these specs pin the API +// side against a real API server, where a plain schema reading cannot. +var _ = Describe("the count bounds of a hand-authored policy", func() { + newPolicy := func(count intstr.IntOrString) *kfplacementv1alpha1.PlacementPolicy { + configMapCount++ + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("hand-authored-%d", configMapCount), + Namespace: "default", + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{{Count: &count}}, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIVersion: "v1", Kind: "ConfigMap", Name: "app", + }}, + }, + } + } + + DescribeTable("integer and string forms share the same bounds", + func(count intstr.IntOrString, wantAccepted bool) { + policy := newPolicy(count) + err := hubClient.Create(ctx, policy) + if err == nil { + // Whatever the verdict was meant to be, an object that made it in must not + // outlive the spec; a regression that admits an out-of-range count would + // otherwise leave its evidence lying around for the rest of the suite. + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, policy))).Should(Succeed()) + }) + } + if wantAccepted { + Expect(err).Should(Succeed()) + return + } + Expect(apierrors.IsInvalid(err)).Should(BeTrue(), "got %v, want an invalid error", err) + }, + Entry("count 1 as an integer is accepted", intstr.FromInt32(1), true), + Entry("count 999 as an integer is accepted", intstr.FromInt32(999), true), + Entry("count 999 as a string is accepted", intstr.FromString("999"), true), + Entry("count All is accepted", intstr.FromString("All"), true), + // The integer entries below are the reason the CEL rule exists: before it, they were + // accepted while their quoted twins were rejected. + Entry("count 1000 as an integer is rejected", intstr.FromInt32(1000), false), + Entry("count 0 as an integer is rejected", intstr.FromInt32(0), false), + Entry("count -1 as an integer is rejected", intstr.FromInt32(-1), false), + Entry("count 1000 as a string is rejected", intstr.FromString("1000"), false), + Entry("count 0 as a string is rejected", intstr.FromString("0"), false), + ) + + // A digit string too long for an int64 used to reach the int() conversion inside the + // minCount<=count rule, adding an opaque evaluation error beside the length violation. The + // count field's own validation must be the whole story now: MaxLength rejects the string + // first, and the rule's digit guard keeps int() out of reach regardless. + It("should report an over-long count with the count field's own message alone", func() { + policy := newPolicy(intstr.FromString("99999999999999999999")) + policy.GetSpec().ClusterSelectors[0].MinCount = ptr.To(int32(1)) + + err := hubClient.Create(ctx, policy) + Expect(apierrors.IsInvalid(err)).Should(BeTrue(), "got %v, want an invalid error", err) + Expect(err.Error()).Should(ContainSubstring("Too long"), "the count field's length bound must report the malformed count") + Expect(err.Error()).ShouldNot(ContainSubstring("minCount"), "the minCount rule must stay out of a problem that is not its own") + }) +}) + +var _ = Describe("annotation based placement", func() { + Context("a namespaced resource", Ordered, func() { + var configMap *corev1.ConfigMap + + BeforeAll(func() { + configMapCount++ + configMap = &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("app-%d", configMapCount), + Namespace: "default", + }, + } + Expect(hubClient.Create(ctx, configMap)).Should(Succeed()) + drainEvents() + }) + + AfterAll(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, configMap))).Should(Succeed()) + }) + + It("should generate no policy while the resource carries no annotation", func() { + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + Expect(drainEvents()).Should(BeEmpty()) + }) + + It("should generate a placement policy when the annotation is set", func() { + annotate(configMap, "env=staging,count=All") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + + spec := policy.GetSpec() + Expect(spec.ClusterSelectors).Should(HaveLen(1)) + Expect(spec.ClusterSelectors[0].Count).Should(Equal(ptrIntOrString(countAll))) + Expect(spec.ClusterSelectors[0].Terms[0].MatchLabels).Should(Equal(map[string]string{"env": "staging"})) + Expect(spec.ResourceSelectors).Should(Equal([]kfplacementv1alpha1.ResourceSelector{{ + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMap.Name, + }})) + + Expect(policy.GetNamespace()).Should(Equal(configMap.Namespace), "a namespaced resource must generate a policy in its own namespace") + Expect(policy.GetLabels()).Should(HaveKeyWithValue(kfplacementv1alpha1.ParentKindLabel, "ConfigMap")) + Expect(policy.GetLabels()).Should(HaveKeyWithValue(kfplacementv1alpha1.ParentNameLabel, configMap.Name)) + Expect(policy.GetLabels()).Should(HaveKeyWithValue(kfplacementv1alpha1.ParentAPIGroupLabel, "")) + + Expect(policy.GetOwnerReferences()).Should(HaveLen(1)) + owner := policy.GetOwnerReferences()[0] + Expect(owner.UID).Should(Equal(configMap.UID)) + Expect(owner.Controller).Should(BeNil(), "a generated policy must not claim controller ownership") + Expect(owner.BlockOwnerDeletion).Should(BeNil(), "a generated policy must not block deletion of the resource it came from") + + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyCreated})) + }) + + It("should not touch the policy when nothing changed", func() { + before, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + after, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + // A resource version that moved means the reconciler wrote to the API server without + // anything having changed, which every pass would then repeat. + Expect(after.GetResourceVersion()).Should(Equal(before.GetResourceVersion())) + Expect(drainEvents()).Should(BeEmpty()) + }) + + It("should update the policy when the annotation changes", func() { + annotate(configMap, "env=canary,region=eastus,count=3") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + spec := policy.GetSpec() + Expect(spec.ClusterSelectors).Should(HaveLen(1)) + Expect(spec.ClusterSelectors[0].Count).Should(Equal(ptrIntOrString("3"))) + Expect(spec.ClusterSelectors[0].Terms[0].MatchLabels).Should(Equal(map[string]string{ + "env": "canary", + corev1.LabelTopologyRegion: "eastus", + })) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyUpdated})) + }) + + It("should restore the policy when someone edits it", func() { + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + policy.GetSpec().ClusterSelectors = nil + Expect(hubClient.Update(ctx, policy)).Should(Succeed()) + + // In the running agent this reconcile is triggered by the watch on the generated + // policies themselves; an edit produces no event on the ConfigMap. + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + restored, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + Expect(restored.GetSpec().ClusterSelectors).Should(HaveLen(1)) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyUpdated})) + }) + + It("should recreate the policy when someone deletes it", func() { + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + Expect(hubClient.Delete(ctx, policy)).Should(Succeed()) + + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + _, err = generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed(), "the deleted policy must be generated again") + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyCreated})) + }) + + It("should keep the policy and warn when the annotation becomes invalid", func() { + annotate(configMap, "env") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed(), "a malformed annotation must not be retried") + + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed(), "the policy from the last valid annotation must be left in place") + Expect(drainEvents()).Should(Equal([]string{EventReasonInvalidAnnotation})) + }) + + It("should delete the policy when the annotation is removed", func() { + annotate(configMap, "") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyDeleted})) + }) + + It("should delete the policy once the resource itself is gone", func() { + // Deleting explicitly, rather than leaving the policy to garbage collection, is what + // keeps an owner reference some other party added from holding the policy up forever. + // It also means envtest, which runs no garbage collector, can observe the cleanup. + annotate(configMap, "env=staging") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + drainEvents() + + Expect(hubClient.Delete(ctx, configMap)).Should(Succeed()) + + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + _, err = generatedPolicyFor(configMapGVK, configMap) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + // No event: the resource an event would be recorded on no longer exists. + Expect(drainEvents()).Should(BeEmpty()) + }) + }) + + Context("a cluster scoped resource", Ordered, func() { + var namespace *corev1.Namespace + + BeforeAll(func() { + configMapCount++ + namespace = &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("team-%d", configMapCount)}, + } + Expect(hubClient.Create(ctx, namespace)).Should(Succeed()) + drainEvents() + }) + + AfterAll(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, namespace))).Should(Succeed()) + }) + + It("should generate a cluster scoped policy", func() { + annotate(namespace, "env=staging") + Expect(reconcile(namespaceGVK, namespace)).Should(Succeed()) + + policy, err := generatedPolicyFor(namespaceGVK, namespace) + Expect(err).Should(Succeed()) + Expect(policy).Should(BeAssignableToTypeOf(&kfplacementv1alpha1.ClusterPlacementPolicy{}), + "a cluster scoped resource must generate a ClusterPlacementPolicy, since a namespaced policy owned by it would never be collected") + Expect(policy.GetNamespace()).Should(BeEmpty()) + Expect(policy.GetSpec().ClusterSelectors).Should(HaveLen(1)) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyCreated})) + }) + + It("should delete the cluster scoped policy when the annotation is removed", func() { + annotate(namespace, "") + Expect(reconcile(namespaceGVK, namespace)).Should(Succeed()) + + _, err := generatedPolicyFor(namespaceGVK, namespace) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyDeleted})) + }) + }) +}) + +func ptrIntOrString(value string) *intstr.IntOrString { + parsed := intstr.Parse(value) + return &parsed +} diff --git a/pkg/v1/controllers/annotationplacement/controller_test.go b/pkg/v1/controllers/annotationplacement/controller_test.go new file mode 100644 index 000000000..1c1d4c506 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/controller_test.go @@ -0,0 +1,1055 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "errors" + "slices" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + "k8s.io/apimachinery/pkg/api/equality" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/tools/events" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var ( + deploymentGVR = schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} + namespaceGVR = schema.GroupVersionResource{Group: "", Version: "v1", Resource: "namespaces"} +) + +const ( + testNamespace = "prod" + testName = "web" + + oneSelector = "env=staging" + twoSelectors = "env=staging,count=All;env=canary,region=eastus" +) + +// newSource builds the annotated resource as the API server would return it. +func newSource(gvk schema.GroupVersionKind, namespace, name string, annotations map[string]string) *unstructured.Unstructured { + source := sourceObject(gvk, namespace, name) + if annotations != nil { + source.SetAnnotations(annotations) + } + return source +} + +// requestFor builds the request the resource watcher would enqueue for a resource. +func requestFor(gvk schema.GroupVersionKind, namespace, name string) Request { + return Request{ + GroupVersionKind: gvk, + NamespacedName: client.ObjectKey{Namespace: namespace, Name: name}, + } +} + +func newScheme(t *testing.T) *runtime.Scheme { + t.Helper() + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + return scheme +} + +// newRESTMapper knows the served kinds these tests read: a namespaced Deployment and a cluster +// scoped Namespace. A kind or a version outside it resolves to a no-match error, as a kind whose +// CRD was removed would at the API server. +func newRESTMapper() meta.RESTMapper { + mapper := meta.NewDefaultRESTMapper([]schema.GroupVersion{deploymentGVK.GroupVersion(), namespaceGVK.GroupVersion()}) + mapper.AddSpecific(deploymentGVK, deploymentGVR, deploymentGVR, meta.RESTScopeNamespace) + mapper.AddSpecific(namespaceGVK, namespaceGVR, namespaceGVR, meta.RESTScopeRoot) + return mapper +} + +// newReconciler wires a reconciler whose API server holds the given annotated resources and +// policies. One fake client backs both the policy client and the API reader, so the two are always +// consistent, which is what every test here wants. +func newReconciler(t *testing.T, funcs interceptor.Funcs, objects ...client.Object) (*Reconciler, *events.FakeRecorder) { + t.Helper() + recorder := events.NewFakeRecorder(10) + c := fake.NewClientBuilder().WithScheme(newScheme(t)).WithObjects(objects...).WithInterceptorFuncs(funcs).Build() + return &Reconciler{ + Client: c, + APIReader: c, + RESTMapper: newRESTMapper(), + Recorder: recorder, + }, recorder +} + +// recordedReasons drains the recorder and returns the reason of every event it holds. +func recordedReasons(recorder *events.FakeRecorder) []string { + reasons := make([]string, 0, len(recorder.Events)) + for { + select { + case event := <-recorder.Events: + // A recorded event reads "Normal ". + fields := strings.SplitN(event, " ", 3) + if len(fields) < 2 { + reasons = append(reasons, event) + continue + } + reasons = append(reasons, fields[1]) + default: + return reasons + } + } +} + +// policyFrom reads the generated policy for a resource back out of the client, or reports that none +// exists. +func policyFrom(ctx context.Context, t *testing.T, r *Reconciler, source *unstructured.Unstructured) (kfplacementv1alpha1.PlacementPolicyAccessor, bool) { + t.Helper() + namespace := source.GetNamespace() + policy := emptyPolicyForScope(namespace) + name := generatedPolicyName(source.GroupVersionKind(), namespace, source.GetName()) + err := r.Get(ctx, client.ObjectKey{Namespace: namespace, Name: name}, policy) + switch { + case apierrors.IsNotFound(err): + return nil, false + case err != nil: + t.Fatalf("Get(%s) = %v, want no error", name, err) + } + return policy, true +} + +// policyIgnoreOpts drop the fields the API server owns, which no expectation here can predict. +var policyIgnoreOpts = cmp.Options{ + cmpopts.IgnoreFields(metav1.TypeMeta{}, "Kind", "APIVersion"), + cmpopts.IgnoreFields(metav1.ObjectMeta{}, "ResourceVersion", "Generation", "CreationTimestamp", "ManagedFields"), +} + +func mustParse(t *testing.T, value string) []kfplacementv1alpha1.ClusterSelector { + t.Helper() + selectors, err := parseClusterSelectors(value) + if err != nil { + t.Fatalf("parseClusterSelectors(%q) = %v, want no error", value, err) + } + return selectors +} + +func TestReconcile(t *testing.T) { + // The desired policy is built with desiredPolicy, which policy_test.go covers on its own; what + // is under test here is whether the reconciler puts that object in the cluster, takes it away + // again, and says so. + annotatedSource := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + bareSource := newSource(deploymentGVK, testNamespace, testName, nil) + clusterScopedSource := newSource(namespaceGVK, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + + testCases := []struct { + name string + // source is the resource the API server holds, and the resource the request points at. + source *unstructured.Unstructured + // existing is the annotation value a policy was already generated from, if any. + existing string + wantPolicy bool + wantValue string + wantReasons []string + }{ + { + name: "an annotated resource with no policy yet gets one", + source: annotatedSource, + wantPolicy: true, + wantValue: oneSelector, + wantReasons: []string{EventReasonPolicyCreated}, + }, + { + name: "a policy that already matches the annotation is left alone", + source: annotatedSource, + existing: oneSelector, + wantPolicy: true, + wantValue: oneSelector, + wantReasons: nil, + }, + { + name: "a changed annotation reaches the policy", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: twoSelectors}), + existing: oneSelector, + wantPolicy: true, + wantValue: twoSelectors, + wantReasons: []string{EventReasonPolicyUpdated}, + }, + { + name: "removing the annotation deletes the policy", + source: bareSource, + existing: oneSelector, + wantPolicy: false, + wantReasons: []string{EventReasonPolicyDeleted}, + }, + { + name: "a resource that was never annotated is left alone", + source: bareSource, + wantPolicy: false, + wantReasons: nil, + }, + { + name: "a cluster scoped resource generates a cluster scoped policy", + source: clusterScopedSource, + wantPolicy: true, + wantValue: oneSelector, + wantReasons: []string{EventReasonPolicyCreated}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + objects := []client.Object{tc.source} + if tc.existing != "" { + objects = append(objects, desiredPolicy(tc.source, mustParse(t, tc.existing))) + } + r, recorder := newReconciler(t, interceptor.Funcs{}, objects...) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + + got, found := policyFrom(ctx, t, r, tc.source) + if found != tc.wantPolicy { + t.Fatalf("Reconcile(%v) left a generated policy = %v, want %v", req, found, tc.wantPolicy) + } + if tc.wantPolicy { + want := desiredPolicy(tc.source, mustParse(t, tc.wantValue)) + if diff := cmp.Diff(got, want, policyIgnoreOpts); diff != "" { + t.Errorf("Reconcile(%v) generated policy mismatch (-got, +want):\n%s", req, diff) + } + } + if diff := cmp.Diff(recordedReasons(recorder), tc.wantReasons, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } +} + +// TestReconcileInvalidAnnotation covers the case the parser rejects: the user hears about it, the +// request is not retried, and a policy generated from an earlier valid annotation stays up. +func TestReconcileInvalidAnnotation(t *testing.T) { + ctx := context.Background() + valid := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(valid, mustParse(t, oneSelector)) + + broken := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: "env"}) + r, recorder := newReconciler(t, interceptor.Funcs{}, broken, existing) + + req := RequestFor(broken) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error, so that a malformed annotation is not retried", req, err) + } + + got, found := policyFrom(ctx, t, r, broken) + if !found { + t.Fatalf("Reconcile(%v) removed the policy generated from the previous annotation, want it left in place", req) + } + if diff := cmp.Diff(got, existing, policyIgnoreOpts); diff != "" { + t.Errorf("Reconcile(%v) changed the policy generated from the previous annotation (-got, +want):\n%s", req, diff) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonInvalidAnnotation}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } +} + +// TestReconcileDeletedResource covers a request for a resource that is already gone. The reconciler +// deletes the generated policy itself rather than deferring to garbage collection, which removes a +// dependent only once every owner reference on it is gone -- and the merge deliberately preserves +// owner references that other parties added, any live one of which would keep the policy standing. +// No event is recorded, since the resource an event would be recorded on no longer exists. +func TestReconcileDeletedResource(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(source, mustParse(t, oneSelector)) + // The other party whose owner reference would hold the policy back from garbage collection. + otherOwner := metav1.OwnerReference{APIVersion: "example.com/v1", Kind: "Widget", Name: "unrelated", UID: "00000000-0000-0000-0000-0000000000ff"} + existing.SetOwnerReferences(append(existing.GetOwnerReferences(), otherOwner)) + + r, recorder := newReconciler(t, interceptor.Funcs{}, existing) + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if _, found := policyFrom(ctx, t, r, source); found { + t.Errorf("Reconcile(%v) left the generated policy standing, want it deleted", req) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none", req, got) + } +} + +// TestReconcileIneligibleResource covers a resource that exists but fails the placement eligibility +// test -- for instance one in a skipped namespace, or a ReplicaSet that a Deployment has adopted. +// Its generated policy is deleted: the resource watcher reports such a resource as deleted and then +// never again, so this reconciliation is the last chance to clean up. +func TestReconcileIneligibleResource(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(source, mustParse(t, oneSelector)) + + r, recorder := newReconciler(t, interceptor.Funcs{}, source, existing) + r.ShouldPlace = func(*unstructured.Unstructured) (bool, error) { return false, nil } + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if _, found := policyFrom(ctx, t, r, source); found { + t.Errorf("Reconcile(%v) left the generated policy standing, want it deleted", req) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonPolicyDeleted}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + + // A second pass over a resource that never generated anything must stay silent. + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none on the second pass", req, got) + } +} + +// TestReconcileEligibilityError covers the eligibility test itself failing, which must surface as a +// retryable error rather than as either verdict. +func TestReconcileEligibilityError(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + r, _ := newReconciler(t, interceptor.Funcs{}, source) + wantErr := errors.New("the eligibility test is unwell") + r.ShouldPlace = func(*unstructured.Unstructured) (bool, error) { return false, wantErr } + + req := RequestFor(source) + if _, err := r.Reconcile(context.Background(), req); !errors.Is(err, wantErr) { + t.Errorf("Reconcile(%v) = %v, want an error wrapping %v", req, err, wantErr) + } + if _, found := policyFrom(context.Background(), t, r, source); found { + t.Errorf("Reconcile(%v) acted on the policy despite the eligibility error", req) + } +} + +// TestReconcileAPIServerErrors covers the paths where a read or a write fails. Each has to come +// back as a retryable error rather than as a silent success, because the annotation and the cluster +// have disagreed at that point and only another pass can settle it. +func TestReconcileAPIServerErrors(t *testing.T) { + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + bare := newSource(deploymentGVK, testNamespace, testName, nil) + existing := desiredPolicy(annotated, mustParse(t, oneSelector)) + failure := apierrors.NewInternalError(errors.New("the api server is unwell")) + + testCases := []struct { + name string + source *unstructured.Unstructured + existing []client.Object + interceptor interceptor.Funcs + }{ + { + name: "the annotated resource cannot be read", + source: annotated, + interceptor: interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, isSource := obj.(*unstructured.Unstructured); isSource { + return failure + } + return c.Get(ctx, key, obj, opts...) + }, + }, + }, + { + name: "the policy cannot be read", + source: annotated, + interceptor: interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, isSource := obj.(*unstructured.Unstructured); isSource { + return c.Get(ctx, key, obj, opts...) + } + return failure + }, + }, + }, + { + name: "the policy cannot be created", + source: annotated, + interceptor: interceptor.Funcs{ + Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { + return failure + }, + }, + }, + { + name: "the policy cannot be updated", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: twoSelectors}), + existing: []client.Object{existing}, + interceptor: interceptor.Funcs{ + Update: func(context.Context, client.WithWatch, client.Object, ...client.UpdateOption) error { + return failure + }, + }, + }, + { + name: "the policy cannot be deleted", + source: bare, + existing: []client.Object{existing}, + interceptor: interceptor.Funcs{ + Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { + return failure + }, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + r, recorder := newReconciler(t, tc.interceptor, append([]client.Object{tc.source}, tc.existing...)...) + + req := RequestFor(tc.source) + _, err := r.Reconcile(context.Background(), req) + if !errors.Is(err, failure) { + t.Errorf("Reconcile(%v) = %v, want an error wrapping %v", req, err, failure) + } + // A write that failed must not be announced as though it had happened. + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none", req, got) + } + }) + } +} + +// TestReconcileRetriesWhenCreateRacesTheCache covers a create that finds the policy already there: +// the cache has yet to see the policy this controller created a moment ago, and a second, unrelated +// event for the source arrived in that window. The pass must come back as a retryable error, not +// as a failure, and must not announce a creation it did not perform. +func TestReconcileRetriesWhenCreateRacesTheCache(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + alreadyExists := apierrors.NewAlreadyExists(schema.GroupResource{Group: kfplacementv1alpha1.GroupVersion.Group, Resource: "placementpolicies"}, generatedPolicyName(deploymentGVK, testNamespace, testName)) + raced := interceptor.Funcs{ + Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { + return alreadyExists + }, + } + r, recorder := newReconciler(t, raced, source) + + req := RequestFor(source) + if _, err := r.Reconcile(context.Background(), req); !errors.Is(err, alreadyExists) { + t.Errorf("Reconcile(%v) = %v, want an error wrapping %v so the source is retried", req, err, alreadyExists) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none for a creation this pass did not perform", req, got) + } +} + +// TestDeleteRacesAnotherDeletion covers a cached read handing the reconciler a policy that is +// already gone by the time it deletes -- typically its own deletion re-entered through the +// generated policy watch. The pass must not claim, in logs or events, a deletion it did not do. +func TestDeleteRacesAnotherDeletion(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, nil) + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(annotated, mustParse(t, oneSelector)) + + // The interceptor stands in for the stale cache: the Get finds the policy, the Delete + // discovers someone else got there first. + raced := interceptor.Funcs{ + Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { + return apierrors.NewNotFound(schema.GroupResource{Group: kfplacementv1alpha1.GroupVersion.Group, Resource: "placementpolicies"}, existing.GetName()) + }, + } + r, recorder := newReconciler(t, raced, source, existing) + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none for a deletion this pass did not perform", req, got) + } +} + +// TestReconcileDoesNotDeleteAReplacementPolicy covers the window between the read that confirms a +// policy is this controller's and the delete that removes it. If the policy is replaced in that +// window -- overwritten with its provenance stripped, or deleted and a hand-authored one created at +// the same name -- an unconditioned delete would remove the replacement, since it targets the name +// alone. The resource-version precondition must turn that into a conflict, and the retry must then +// read the current object and decline it. +func TestReconcileDoesNotDeleteAReplacementPolicy(t *testing.T) { + ctx := context.Background() + bare := newSource(deploymentGVK, testNamespace, testName, nil) + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + ours := desiredPolicy(annotated, mustParse(t, oneSelector)) + + // A spec no generated policy would carry, marking the object that takes the name. + foreignSpec := kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIGroup: "example.com", APIVersion: "v1", Kind: "Widget", Name: "hand-authored"}}, + } + + replaced := false + raced := interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if !replaced { + replaced = true + // Stand in for the replacement: the live object loses its provenance and its resource + // version moves on, so it is no longer the one the precondition names. + live := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(ctx, client.ObjectKeyFromObject(obj), live); err != nil { + return err + } + live.SetLabels(nil) + live.SetOwnerReferences(nil) + live.Spec = foreignSpec + if err := c.Update(ctx, live); err != nil { + return err + } + } + return c.Delete(ctx, obj, opts...) + }, + } + r, recorder := newReconciler(t, raced, bare, ours) + + req := RequestFor(bare) + // First pass: the policy is replaced after it is read, so the guarded delete conflicts and the + // pass returns a retryable error rather than removing the replacement. + if _, err := r.Reconcile(ctx, req); err == nil { + t.Fatalf("Reconcile(%v) = nil, want a conflict error so the delete of a replaced policy is retried", req) + } + got, found := policyFrom(ctx, t, r, bare) + if !found { + t.Fatalf("Reconcile(%v) deleted the replacement policy, want it left in place", req) + } + if diff := cmp.Diff(got.(*kfplacementv1alpha1.PlacementPolicy).Spec, foreignSpec); diff != "" { + t.Errorf("Reconcile(%v) changed the replacement policy (-got, +want):\n%s", req, diff) + } + + // Second pass: the reconciler reads the current, now foreign object, declines it, and records no + // deletion it did not perform. + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error once the replacement is recognized as foreign", req, err) + } + if _, found := policyFrom(ctx, t, r, bare); !found { + t.Errorf("Reconcile(%v) deleted the foreign replacement on retry, want it left in place", req) + } + if got := recordedReasons(recorder); slices.Contains(got, EventReasonPolicyDeleted) { + t.Errorf("Reconcile(%v) recorded events = %v, want no deletion of a policy it did not generate", req, got) + } +} + +func TestApplyDesiredPolicy(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + desired := desiredPolicy(source, mustParse(t, oneSelector)) + scheme := newScheme(t) + + otherOwner := metav1.OwnerReference{ + APIVersion: "example.com/v1", + Kind: "Widget", + Name: "unrelated", + UID: "00000000-0000-0000-0000-0000000000ff", + } + + testCases := []struct { + name string + // mutate turns the desired policy into the live one the reconciler would read back. + mutate func(kfplacementv1alpha1.PlacementPolicyAccessor) + wantChanged bool + // check asserts on whatever the merge is supposed to preserve. + check func(*testing.T, kfplacementv1alpha1.PlacementPolicyAccessor) + }{ + { + name: "an untouched policy needs no update", + mutate: func(kfplacementv1alpha1.PlacementPolicyAccessor) {}, + wantChanged: false, + }, + { + name: "a drifted spec is restored", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors = nil + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if diff := cmp.Diff(policy.GetSpec(), desired.GetSpec()); diff != "" { + t.Errorf("applyDesiredPolicy() spec mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "a missing provenance label is restored", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + labels := policy.GetLabels() + delete(labels, kfplacementv1alpha1.ParentKindLabel) + policy.SetLabels(labels) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if diff := cmp.Diff(policy.GetLabels(), desired.GetLabels()); diff != "" { + t.Errorf("applyDesiredPolicy() labels mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "a policy stripped of every label gets the provenance labels back", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.SetLabels(nil) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if diff := cmp.Diff(policy.GetLabels(), desired.GetLabels()); diff != "" { + t.Errorf("applyDesiredPolicy() labels mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "labels added by something else are kept", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + labels := policy.GetLabels() + labels["example.com/managed-by"] = "gitops" + delete(labels, kfplacementv1alpha1.ParentKindLabel) + policy.SetLabels(labels) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if got := policy.GetLabels()["example.com/managed-by"]; got != "gitops" { + t.Errorf("applyDesiredPolicy() label example.com/managed-by = %q, want %q", got, "gitops") + } + }, + }, + { + name: "an owner reference that drifted is corrected in place", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + drifted := parentOwnerReference(source) + drifted.APIVersion = "apps/v1beta1" + policy.SetOwnerReferences([]metav1.OwnerReference{drifted}) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + want := []metav1.OwnerReference{parentOwnerReference(source)} + if diff := cmp.Diff(policy.GetOwnerReferences(), want); diff != "" { + t.Errorf("applyDesiredPolicy() owner references mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "a missing owner reference is restored alongside any other", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.SetOwnerReferences([]metav1.OwnerReference{otherOwner}) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + want := []metav1.OwnerReference{otherOwner, parentOwnerReference(source)} + if diff := cmp.Diff(policy.GetOwnerReferences(), want); diff != "" { + t.Errorf("applyDesiredPolicy() owner references mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + // The source was deleted and recreated under the same name, so its reference carries the + // old UID. It must be updated in place, not left dangling while a second one is appended -- + // otherwise the list grows by one on every such cycle. + name: "an owner reference left by a recreated source is replaced, not appended", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + stale := parentOwnerReference(source) + stale.UID = "00000000-0000-0000-0000-00000000dead" + policy.SetOwnerReferences([]metav1.OwnerReference{stale}) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + want := []metav1.OwnerReference{parentOwnerReference(source)} + if diff := cmp.Diff(policy.GetOwnerReferences(), want); diff != "" { + t.Errorf("applyDesiredPolicy() owner references mismatch (-got, +want):\n%s", diff) + } + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + actual := desired.DeepCopyObject().(kfplacementv1alpha1.PlacementPolicyAccessor) + tc.mutate(actual) + before := actual.DeepCopyObject() + + if err := applyDesiredPolicy(actual, desired, source, scheme); err != nil { + t.Fatalf("applyDesiredPolicy() = %v, want no error", err) + } + if got := !equality.Semantic.DeepEqual(before, actual); got != tc.wantChanged { + t.Errorf("applyDesiredPolicy() changed the policy = %v, want %v", got, tc.wantChanged) + } + if tc.check != nil { + tc.check(t, actual) + } + // Whatever the merge did, a second pass over its own output must find nothing left to + // do; otherwise the reconciler would issue an update on every single pass. + settled := actual.DeepCopyObject() + if err := applyDesiredPolicy(actual, desired, source, scheme); err != nil { + t.Fatalf("applyDesiredPolicy() = %v on the second pass, want no error", err) + } + if !equality.Semantic.DeepEqual(settled, actual) { + t.Errorf("applyDesiredPolicy() changed the policy on the second pass, want it settled") + } + }) + } +} + +// TestEventMessagesNameTheGeneratedKind pins that an event's message names the kind actually +// generated: the two scopes generate different kinds, and a message claiming a PlacementPolicy for +// what is really a ClusterPlacementPolicy sends the user's kubectl to a resource that is not there. +func TestEventMessagesNameTheGeneratedKind(t *testing.T) { + testCases := []struct { + name string + source *unstructured.Unstructured + wantKind string + }{ + { + name: "a namespaced source names PlacementPolicy", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + wantKind: "PlacementPolicy", + }, + { + name: "a cluster scoped source names ClusterPlacementPolicy", + source: newSource(namespaceGVK, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + wantKind: "ClusterPlacementPolicy", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + + select { + case event := <-recorder.Events: + // The kind is matched with its surrounding spaces: "ClusterPlacementPolicy" + // contains "PlacementPolicy" as a bare substring, so an unanchored check would + // pass the namespaced case even if the wrong kind were named. + if !strings.Contains(event, " the "+tc.wantKind+" ") { + t.Errorf("Reconcile(%v) recorded event %q, want it to name the kind %q", req, event, tc.wantKind) + } + default: + t.Fatalf("Reconcile(%v) recorded no event, want one naming the kind %q", req, tc.wantKind) + } + }) + } +} + +// TestReconcileUnknownKind covers a request whose kind the API server does not know, which the +// generated policy watch can produce by enqueuing whatever a policy names as its owner. The kind +// being gone means the source's own CRD was removed, so any policy generated from it is stale and is +// deleted; when none exists the request is simply dropped, since no retry can make the kind exist +// and an error would keep it backing off forever. +func TestReconcileUnknownKind(t *testing.T) { + widgetGVK := schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: "Widget"} + // The resource whose kind is gone. It is only used to derive the generated policy the reconciler + // must find and delete; the REST mapper deliberately does not know its kind. + widget := newSource(widgetGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + stale := desiredPolicy(widget, mustParse(t, oneSelector)) + + testCases := []struct { + name string + existing []client.Object + wantPolicy bool + }{ + {name: "no policy to clean up, request is dropped", existing: nil, wantPolicy: false}, + {name: "a stale policy left by the gone kind is deleted", existing: []client.Object{stale}, wantPolicy: false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.existing...) + req := RequestFor(widget) + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Errorf("Reconcile(%v) = %v, want no error, so that a kind that does not exist is not retried", req, err) + } + if _, found := policyFrom(context.Background(), t, r, widget); found != tc.wantPolicy { + t.Errorf("Reconcile(%v) left a generated policy = %v, want %v", req, found, tc.wantPolicy) + } + // No event is recorded either way: the resource an event would attach to is gone. + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none", req, got) + } + }) + } +} + +// TestReconcileKeepsPolicyWhenOnlyVersionRetired covers a request whose recorded version is no +// longer served while the kind lives on under another. The generated policy watch can enqueue such +// a request, and treating the retired version as a gone kind would delete a policy that is still +// wanted. The resource is deliberately not re-read under the served version -- that version may not +// exist on the member clusters -- so the round is retried instead, and API discovery re-reports the +// resource under the served version. +func TestReconcileKeepsPolicyWhenOnlyVersionRetired(t *testing.T) { + testCases := []struct { + name string + source *unstructured.Unstructured + // req names the source under a version the REST mapper no longer knows. + req Request + }{ + { + name: "a namespaced source served under apps/v1 is reached through apps/v2", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + req: requestFor(schema.GroupVersionKind{Group: "apps", Version: "v2", Kind: "Deployment"}, testNamespace, testName), + }, + { + name: "a cluster scoped source served under v1 is reached through v2", + source: newSource(namespaceGVK, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + req: requestFor(schema.GroupVersionKind{Group: "", Version: "v2", Kind: "Namespace"}, "", testName), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source) + + // An existing policy stands in for one generated before the version was retired; the + // round must leave it alone rather than take it for the policy of a gone kind. + existing := desiredPolicy(tc.source, mustParse(t, oneSelector)) + if err := r.Create(ctx, existing); err != nil { + t.Fatalf("Create(%v) = %v, want no error", klog.KObj(existing), err) + } + + // Dropped, not retried: no error and no requeue, since the resource is re-reported + // under the served version as a different request. + res, err := r.Reconcile(ctx, tc.req) + if err != nil { + t.Errorf("Reconcile(%v) = %v, want the retired version dropped rather than retried", tc.req, err) + } + if res.RequeueAfter != 0 { + t.Errorf("Reconcile(%v) = %+v, want no requeue for a version that will never come back", tc.req, res) + } + if _, found := policyFrom(ctx, t, r, tc.source); !found { + t.Errorf("Reconcile(%v) deleted the generated policy, want it kept while the kind is still served", tc.req) + } + if diff := cmp.Diff(recordedReasons(recorder), []string(nil), cmpopts.EquateEmpty()); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", tc.req, diff) + } + }) + } +} + +// TestReconcileForeignPolicyAtGeneratedName covers a policy that already occupies a resource's +// generated name but was authored by someone else -- it carries none of this controller's provenance +// labels. The controller must neither overwrite it when the annotation asks for a policy nor delete +// it when the annotation is removed; a bare name match would do both. +func TestReconcileForeignPolicyAtGeneratedName(t *testing.T) { + // A policy sitting at the generated name, distinguishable by a spec this controller would never + // produce and by the absence of the provenance labels. + foreignSpec := func() kfplacementv1alpha1.PlacementPolicySpec { + return kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIGroup: "example.com", APIVersion: "v1", Kind: "Widget", Name: "hand-authored"}}, + } + } + newForeign := func() *kfplacementv1alpha1.PlacementPolicy { + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: generatedPolicyName(deploymentGVK, testNamespace, testName), + Namespace: testNamespace, + }, + Spec: foreignSpec(), + } + } + + testCases := []struct { + name string + source *unstructured.Unstructured + wantErr error + wantReasons []string + }{ + { + name: "the annotation asks for a policy, the foreign one is not overwritten", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + // The source is retried with backoff -- nothing else brings it back once the conflict is + // cleared -- and the user hears why the placement they asked for is not running. + wantErr: errForeignPolicy, + wantReasons: []string{EventReasonPolicyConflict}, + }, + { + name: "the annotation is absent, the foreign one is not deleted", + // Nothing asked for a policy here, so declining to delete a policy that was never this + // controller's is silent -- the same as a resource that never generated anything. + source: newSource(deploymentGVK, testNamespace, testName, nil), + wantReasons: nil, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source, newForeign()) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(ctx, req); !errors.Is(err, tc.wantErr) { + t.Fatalf("Reconcile(%v) = %v, want %v", req, err, tc.wantErr) + } + + got, found := policyFrom(ctx, t, r, tc.source) + if !found { + t.Fatalf("Reconcile(%v) removed the foreign policy, want it left in place", req) + } + gotSpec := got.(*kfplacementv1alpha1.PlacementPolicy).Spec + if diff := cmp.Diff(gotSpec, foreignSpec()); diff != "" { + t.Errorf("Reconcile(%v) changed the foreign policy's spec (-got, +want):\n%s", req, diff) + } + if labels := got.GetLabels(); len(labels) != 0 { + t.Errorf("Reconcile(%v) added labels %v to the foreign policy, want it left untouched", req, labels) + } + if diff := cmp.Diff(recordedReasons(recorder), tc.wantReasons, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } +} + +// TestReconcileRepairsDriftedProvenance covers a policy this controller generated whose provenance +// labels were later edited away. Its owner reference still identifies it as ours, so it must be +// repaired while the annotation stands and deleted once the annotation is removed -- never stranded +// as though it were foreign, which would leave its placement running with nothing able to reconcile +// or remove it. +func TestReconcileRepairsDriftedProvenance(t *testing.T) { + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + selectors := mustParse(t, oneSelector) + // A generated policy whose provenance labels have drifted -- one label is gone -- while the owner + // reference this controller set is untouched, so the policy is still recognizable as ours. + drifted := func() client.Object { + policy := desiredPolicy(annotated, selectors) + labels := policy.GetLabels() + delete(labels, kfplacementv1alpha1.ParentKindLabel) + policy.SetLabels(labels) + return policy + } + + testCases := []struct { + name string + source *unstructured.Unstructured + wantPolicy bool + wantReason string + }{ + { + name: "the annotation stands, so the drifted labels are repaired", + source: annotated, + wantPolicy: true, + wantReason: EventReasonPolicyUpdated, + }, + { + name: "the annotation is gone, so the policy is deleted", + source: newSource(deploymentGVK, testNamespace, testName, nil), + wantPolicy: false, + wantReason: EventReasonPolicyDeleted, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source, drifted()) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + + got, found := policyFrom(ctx, t, r, annotated) + if found != tc.wantPolicy { + t.Fatalf("Reconcile(%v) left a generated policy = %v, want %v", req, found, tc.wantPolicy) + } + if tc.wantPolicy { + if diff := cmp.Diff(got.GetLabels(), desiredPolicy(annotated, selectors).GetLabels()); diff != "" { + t.Errorf("Reconcile(%v) did not restore the drifted provenance labels (-got, +want):\n%s", req, diff) + } + } + if diff := cmp.Diff(recordedReasons(recorder), []string{tc.wantReason}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } +} + +// TestReconcileResumesOnceForeignPolicyClears covers a source whose generated name is occupied by a +// foreign policy. Removing that policy fires no event that reaches the source -- it carries no owner +// reference to it -- and the annotation does not change, so the reconciler reports the conflict as +// an error to be retried with backoff; the pass after the conflict clears must then create the +// requested policy. +func TestReconcileResumesOnceForeignPolicyClears(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + foreign := &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: generatedPolicyName(deploymentGVK, testNamespace, testName), + Namespace: testNamespace, + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIGroup: "example.com", APIVersion: "v1", Kind: "Widget", Name: "hand-authored"}}, + }, + } + r, recorder := newReconciler(t, interceptor.Funcs{}, source, foreign) + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); !errors.Is(err, errForeignPolicy) { + t.Fatalf("Reconcile(%v) = %v, want %v so the blocked source is retried", req, err, errForeignPolicy) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonPolicyConflict}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + + // The user removes the blocking policy. The next pass creates the requested policy. + if err := r.Delete(ctx, foreign); err != nil { + t.Fatalf("Delete(foreign) = %v, want no error", err) + } + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error after the conflict cleared", req, err) + } + if _, found := policyFrom(ctx, t, r, source); !found { + t.Errorf("Reconcile(%v) did not create the policy after the conflict cleared", req) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonPolicyCreated}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } +} + +func TestHasClusterSelectorsAnnotation(t *testing.T) { + testCases := []struct { + name string + annotations map[string]string + want bool + }{ + {name: "no annotations at all", annotations: nil, want: false}, + {name: "other annotations only", annotations: map[string]string{"example.com/other": "value"}, want: false}, + {name: "the annotation is present", annotations: map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}, want: true}, + // An empty value is still a request for annotation-based placement; the parser is what + // rejects it, and it can only do that if the resource reaches the queue at all. + {name: "the annotation is present but empty", annotations: map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: ""}, want: true}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, tc.annotations) + if got := HasClusterSelectorsAnnotation(source); got != tc.want { + t.Errorf("HasClusterSelectorsAnnotation(%v) = %v, want %v", tc.annotations, got, tc.want) + } + }) + } +} diff --git a/pkg/v1/controllers/annotationplacement/manager_integration_test.go b/pkg/v1/controllers/annotationplacement/manager_integration_test.go new file mode 100644 index 000000000..8dd98c174 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/manager_integration_test.go @@ -0,0 +1,163 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/tools/events" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// These specs run the controller the way the hub agent would: registered with a manager, fed +// annotated resources through the channel the resource watcher writes to, and reading generated +// policies from the manager's cache. They are what proves the watch on the generated policies -- +// the reconciler's own tests call Reconcile directly and never see a policy event. +var _ = Describe("annotation based placement under a manager", Ordered, func() { + var ( + managerCtx context.Context + stopManager context.CancelFunc + managerDone chan struct{} + sourceQueue *SourceQueue + configMap *corev1.ConfigMap + managedEvents *events.FakeRecorder + ) + + // report hands the resource watcher's view of a resource to the controller, as the hub agent's + // change detector would on every event for an annotated resource. + report := func(object client.Object) { + source := &unstructured.Unstructured{} + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(object) + Expect(err).Should(Succeed()) + source.SetUnstructuredContent(content) + source.SetGroupVersionKind(configMapGVK) + sourceQueue.Add(source) + } + + managedPolicy := func() (kfplacementv1alpha1.PlacementPolicyAccessor, error) { + policy := &kfplacementv1alpha1.PlacementPolicy{} + key := client.ObjectKey{Namespace: configMap.Namespace, Name: generatedPolicyName(configMapGVK, configMap.Namespace, configMap.Name)} + return policy, hubClient.Get(ctx, key, policy) + } + + BeforeAll(func() { + managerCtx, stopManager = context.WithCancel(ctx) + mgr, err := ctrl.NewManager(restConfig, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{BindAddress: "0"}, + }) + Expect(err).Should(Succeed()) + + sourceQueue = &SourceQueue{} + managedEvents = events.NewFakeRecorder(100) + managed := &Reconciler{ + Client: mgr.GetClient(), + APIReader: mgr.GetAPIReader(), + RESTMapper: mgr.GetRESTMapper(), + Recorder: managedEvents, + } + Expect(managed.SetupWithManager(mgr, sourceQueue)).Should(Succeed()) + managerDone = make(chan struct{}) + go func() { + defer GinkgoRecover() + defer close(managerDone) + Expect(mgr.Start(managerCtx)).Should(Succeed()) + }() + + configMapCount++ + configMap = &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("managed-%d", configMapCount), + Namespace: "default", + Annotations: map[string]string{ + kfplacementv1alpha1.ClusterSelectorsAnnotation: "env=staging,count=All", + }, + }, + } + Expect(hubClient.Create(ctx, configMap)).Should(Succeed()) + }) + + AfterAll(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, configMap))).Should(Succeed()) + stopManager() + // Start's own assertion must land while this spec tree is still running. + Eventually(managerDone, eventuallyTimeout).Should(BeClosed()) + }) + + It("should generate a policy for a resource the watcher reports", func() { + report(configMap) + Eventually(func() error { + _, err := managedPolicy() + return err + }, eventuallyTimeout, eventuallyInterval).Should(Succeed()) + }) + + It("should restore a generated policy someone edits, with no event on the resource", func() { + policy, err := managedPolicy() + Expect(err).Should(Succeed()) + policy.GetSpec().ClusterSelectors = nil + Expect(hubClient.Update(ctx, policy)).Should(Succeed()) + + Eventually(func() ([]kfplacementv1alpha1.ClusterSelector, error) { + policy, err := managedPolicy() + if err != nil { + return nil, err + } + return policy.GetSpec().ClusterSelectors, nil + }, eventuallyTimeout, eventuallyInterval).Should(HaveLen(1), "the watch on generated policies must bring an edited policy back") + }) + + It("should recreate a generated policy someone deletes, with no event on the resource", func() { + policy, err := managedPolicy() + Expect(err).Should(Succeed()) + deletedUID := policy.GetUID() + Expect(hubClient.Delete(ctx, policy)).Should(Succeed()) + + Eventually(func() (bool, error) { + policy, err := managedPolicy() + if err != nil { + return false, err + } + return policy.GetUID() != deletedUID, nil + }, eventuallyTimeout, eventuallyInterval).Should(BeTrue(), "the watch on generated policies must bring a deleted policy back") + }) + + It("should delete the policy once the watcher reports the annotation gone", func() { + Expect(hubClient.Get(ctx, client.ObjectKeyFromObject(configMap), configMap)).Should(Succeed()) + configMap.Annotations = nil + Expect(hubClient.Update(ctx, configMap)).Should(Succeed()) + report(configMap) + + Eventually(func() bool { + _, err := managedPolicy() + return apierrors.IsNotFound(err) + }, eventuallyTimeout, eventuallyInterval).Should(BeTrue()) + }) +}) diff --git a/pkg/v1/controllers/annotationplacement/parser.go b/pkg/v1/controllers/annotationplacement/parser.go new file mode 100644 index 000000000..cf6a2c9c8 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/parser.go @@ -0,0 +1,230 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package annotationplacement implements FEP-0001 annotation-based placement: it keeps a placement +// policy object in sync with the kubefleet.dev/cluster-selectors annotation set on a Kubernetes +// resource, so that simple placement scenarios can be expressed without authoring a placement +// policy by hand. +package annotationplacement + +import ( + "errors" + "fmt" + "strconv" + "strings" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" + kferrors "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +const ( + // selectorSeparator separates the cluster selectors within the annotation value. + selectorSeparator = ";" + // matcherSeparator separates the label matchers, and the count directive, within one selector. + matcherSeparator = "," + // keyValueSeparator separates a label matcher's key from its value. + keyValueSeparator = "=" + + // countKey is the reserved matcher key that sets the desired cluster count of a selector. + countKey = "count" + // countAll is the count value that selects every cluster matching the selector's terms. + countAll = "All" + + // regionShorthand and aliasShorthand are the label key shorthands FEP-0001 reserves. + regionShorthand = "region" + aliasShorthand = "alias" +) + +// The limits below mirror the validation markers on the placement policy API, so that a rejected +// annotation is reported against the annotation itself rather than surfacing later as an opaque +// failure to create the policy object. +const ( + // maxSelectors mirrors MaxItems on PlacementPolicySpec.ClusterSelectors. + maxSelectors = 10 + // maxMatchLabels mirrors MaxProperties on ClusterLabelAndPropertySelectorTerm.MatchLabels. + maxMatchLabels = 10 + // maxCount mirrors the ceiling on ClusterSelector.Count, which the API enforces once per form: + // the Pattern marker bounds the string form, and a CEL rule bounds the integer form, since a + // pattern does not constrain the integer side of an int-or-string. + maxCount = 999 +) + +// defaultCount is the desired cluster count of a selector that does not carry a count directive. It +// matches the API's own default for the field, and is set explicitly rather than left to the API +// server so that the policy object built from an annotation compares equal to the one the API +// server stores; otherwise every reconciliation would observe a difference and issue an update. +const defaultCount = 1 + +// shorthandLabelKeys maps the label key shorthands reserved by FEP-0001 to the keys they expand to. +var shorthandLabelKeys = map[string]string{ + regionShorthand: corev1.LabelTopologyRegion, + aliasShorthand: placementv1beta1.ClusterAliasLabel, +} + +// parseClusterSelectors converts the value of the kubefleet.dev/cluster-selectors annotation into +// the cluster selectors of a placement policy. +// +// Errors returned by this function are always caused by the annotation's own contents, so they are +// categorized as user errors: they are never transient, and a caller should surface them to the +// user rather than retry. The message stays human-readable, since it is what the event recorded on +// the annotated resource shows: the category is for callers, the text is for whoever wrote the +// annotation. +func parseClusterSelectors(value string) ([]kfplacementv1alpha1.ClusterSelector, error) { + if strings.TrimSpace(value) == "" { + return nil, kferrors.NewUserError(nil, "the annotation value is empty; it must list at least one cluster selector") + } + + segments := strings.Split(value, selectorSeparator) + if len(segments) > maxSelectors { + return nil, kferrors.NewUserError(nil, fmt.Sprintf("the annotation lists %d cluster selectors, more than the supported maximum of %d", len(segments), maxSelectors)) + } + + selectors := make([]kfplacementv1alpha1.ClusterSelector, 0, len(segments)) + for idx, segment := range segments { + selector, err := parseSelector(segment) + if err != nil { + // The index is 1-based: it is read by whoever wrote the annotation, who counts the + // selectors in it by eye. + return nil, kferrors.NewUserError(err, fmt.Sprintf("cluster selector %d (%q) is invalid", idx+1, strings.TrimSpace(segment))) + } + selectors = append(selectors, selector) + } + return selectors, nil +} + +// parseSelector converts one semicolon-delimited segment of the annotation into a cluster selector. +func parseSelector(segment string) (kfplacementv1alpha1.ClusterSelector, error) { + var selector kfplacementv1alpha1.ClusterSelector + + matchLabels := make(map[string]string) + var count *intstr.IntOrString + + for _, matcher := range strings.Split(segment, matcherSeparator) { + matcher = strings.TrimSpace(matcher) + if matcher == "" { + return selector, fmt.Errorf("it has an empty label matcher") + } + + rawKey, rawValue, found := strings.Cut(matcher, keyValueSeparator) + if !found { + return selector, fmt.Errorf("the label matcher %q is not in the KEY=VALUE format", matcher) + } + rawKey, rawValue = strings.TrimSpace(rawKey), strings.TrimSpace(rawValue) + + if rawKey == countKey { + if count != nil { + return selector, fmt.Errorf("it sets %s more than once", countKey) + } + parsed, err := parseCount(rawValue) + if err != nil { + return selector, err + } + count = parsed + continue + } + + key, err := parseLabelKey(rawKey) + if err != nil { + return selector, err + } + if _, duplicate := matchLabels[key]; duplicate { + return selector, fmt.Errorf("it matches on the label key %q more than once", key) + } + if err := validateLabelValue(key, rawValue); err != nil { + return selector, err + } + matchLabels[key] = rawValue + } + + if len(matchLabels) > maxMatchLabels { + return selector, fmt.Errorf("it matches on %d label keys, more than the supported maximum of %d", len(matchLabels), maxMatchLabels) + } + + if count == nil { + fallback := intstr.FromInt32(defaultCount) + count = &fallback + } + selector.Count = count + // A selector that carries only a count directive has no terms, which the API reads as a match on + // every cluster; this is deliberate, as it lets `count=All` on its own select the whole fleet. + if len(matchLabels) > 0 { + selector.Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: matchLabels}, + } + } + return selector, nil +} + +// parseCount converts the value of a count directive into the desired cluster count of a selector. +func parseCount(value string) (*intstr.IntOrString, error) { + if value == countAll { + all := intstr.FromString(countAll) + return &all, nil + } + if strings.EqualFold(value, countAll) { + return nil, fmt.Errorf("the %s value %q is not recognized; it must be spelled exactly %q", countKey, value, countAll) + } + + // ParseUint accepts exactly a run of decimal digits: no sign, no whitespace, no exponent. + parsed, err := strconv.ParseUint(value, 10, 32) + switch { + case errors.Is(err, strconv.ErrRange): + return nil, fmt.Errorf("the %s value %q is out of the supported range (1-%d)", countKey, value, maxCount) + case err != nil: + return nil, fmt.Errorf("the %s value %q is neither a positive integer nor %q", countKey, value, countAll) + } + if parsed < 1 || parsed > maxCount { + return nil, fmt.Errorf("the %s value %d is out of the supported range (1-%d)", countKey, parsed, maxCount) + } + + count := intstr.FromInt32(int32(parsed)) + return &count, nil +} + +// parseLabelKey expands a label key shorthand, if one is used, and verifies that the result is a +// key that Kubernetes accepts. +func parseLabelKey(key string) (string, error) { + if expanded, isShorthand := shorthandLabelKeys[key]; isShorthand { + return expanded, nil + } + if key == "" { + return "", fmt.Errorf("it has a label matcher with an empty label key") + } + if errs := validation.IsQualifiedName(key); len(errs) > 0 { + return "", fmt.Errorf("the label key %q is not valid: %s", key, strings.Join(errs, "; ")) + } + return key, nil +} + +// validateLabelValue verifies that a label matcher's value is one that Kubernetes accepts. +// +// Kubernetes considers the empty string a valid label value, but this surface rejects it: an empty +// value in a shorthand annotation is far more often a typo than a deliberate match on a label set +// to the empty string, and the placement policy API remains available for the latter. +func validateLabelValue(key, value string) error { + if value == "" { + return fmt.Errorf("the label key %q is matched against an empty value; use a placement policy object to match on an empty label value", key) + } + if errs := validation.IsValidLabelValue(value); len(errs) > 0 { + return fmt.Errorf("the value %q of the label key %q is not valid: %s", value, key, strings.Join(errs, "; ")) + } + return nil +} diff --git a/pkg/v1/controllers/annotationplacement/parser_test.go b/pkg/v1/controllers/annotationplacement/parser_test.go new file mode 100644 index 000000000..1c896bc5e --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/parser_test.go @@ -0,0 +1,485 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "os" + "regexp" + "strconv" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + "sigs.k8s.io/yaml" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + regionLabel = "topology.kubernetes.io/region" + aliasLabel = "kubefleet.dev/cluster-alias" + + // qualifiedNameMaxLength is the longest name segment of a label key that Kubernetes accepts. + // It mirrors the unexported limit that validation.IsQualifiedName enforces. + qualifiedNameMaxLength = 63 + + // crdPath is the placement policy CRD, read by TestLimitsMatchAPIValidation. + crdPath = "../../../../config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml" +) + +// selectorOf builds the cluster selector that the parser is expected to produce for one segment of +// the annotation; a nil matchLabels stands for a selector that carries no terms. +func selectorOf(count intstr.IntOrString, matchLabels map[string]string) kfplacementv1alpha1.ClusterSelector { + selector := kfplacementv1alpha1.ClusterSelector{Count: &count} + if matchLabels != nil { + selector.Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: matchLabels}, + } + } + return selector +} + +func TestParseClusterSelectors(t *testing.T) { + one, three, all := intstr.FromInt32(1), intstr.FromInt32(3), intstr.FromString("All") + + testCases := []struct { + name string + value string + want []kfplacementv1alpha1.ClusterSelector + }{ + { + name: "single label matcher with the region shorthand", + value: "region=eastus", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{regionLabel: "eastus"}), + }, + }, + { + // The first example given in FEP-0001. + name: "one cluster per region", + value: "region=eastus;region=westus;region=centralus", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{regionLabel: "eastus"}), + selectorOf(one, map[string]string{regionLabel: "westus"}), + selectorOf(one, map[string]string{regionLabel: "centralus"}), + }, + }, + { + // The second example given in FEP-0001. + name: "named clusters through the alias shorthand", + value: "alias=bravelion;alias=smartfish", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{aliasLabel: "bravelion"}), + selectorOf(one, map[string]string{aliasLabel: "smartfish"}), + }, + }, + { + // The third example given in FEP-0001. + name: "mixed counts and multiple matchers per selector", + value: "env=staging,count=All;env=canary,region=eastus,count=1", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(all, map[string]string{"env": "staging"}), + selectorOf(one, map[string]string{"env": "canary", regionLabel: "eastus"}), + }, + }, + { + name: "count on its own selects the whole fleet", + value: "count=All", + want: []kfplacementv1alpha1.ClusterSelector{selectorOf(all, nil)}, + }, + { + name: "count directive may precede the label matchers", + value: "count=3,env=prod", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(three, map[string]string{"env": "prod"}), + }, + }, + { + name: "fully qualified label keys are passed through", + value: fmt.Sprintf("%s=eastus", regionLabel), + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{regionLabel: "eastus"}), + }, + }, + { + // Shorthands are expanded on an exact match only; Kubernetes label keys are + // case-sensitive, so a differently cased key is a different label. + name: "shorthand expansion is case-sensitive", + value: "Region=eastus", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{"Region": "eastus"}), + }, + }, + { + name: "surrounding whitespace is ignored", + value: " region = eastus , count = 3 ; env=prod ", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(three, map[string]string{regionLabel: "eastus"}), + selectorOf(one, map[string]string{"env": "prod"}), + }, + }, + { + name: "count at the top of the supported range", + value: fmt.Sprintf("env=prod,count=%d", maxCount), + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(intstr.FromInt32(maxCount), map[string]string{"env": "prod"}), + }, + }, + { + name: "label value at the longest length Kubernetes accepts", + value: "env=" + strings.Repeat("a", validation.LabelValueMaxLength), + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{"env": strings.Repeat("a", validation.LabelValueMaxLength)}), + }, + }, + { + name: "label matchers at the supported maximum", + value: func() string { + matchers := make([]string, 0, maxMatchLabels) + for i := 0; i < maxMatchLabels; i++ { + matchers = append(matchers, fmt.Sprintf("key%d=value", i)) + } + return strings.Join(matchers, matcherSeparator) + }(), + want: func() []kfplacementv1alpha1.ClusterSelector { + matchLabels := make(map[string]string, maxMatchLabels) + for i := 0; i < maxMatchLabels; i++ { + matchLabels[fmt.Sprintf("key%d", i)] = "value" + } + return []kfplacementv1alpha1.ClusterSelector{selectorOf(one, matchLabels)} + }(), + }, + { + // The count directive is reserved on an exact match only, like the shorthands above. + name: "the count keyword is case-sensitive", + value: "Count=5", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{"Count": "5"}), + }, + }, + { + name: "selector count at the supported maximum", + value: strings.TrimSuffix(strings.Repeat("region=eastus;", maxSelectors), ";"), + want: func() []kfplacementv1alpha1.ClusterSelector { + selectors := make([]kfplacementv1alpha1.ClusterSelector, 0, maxSelectors) + for i := 0; i < maxSelectors; i++ { + selectors = append(selectors, selectorOf(one, map[string]string{regionLabel: "eastus"})) + } + return selectors + }(), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := parseClusterSelectors(tc.value) + if err != nil { + t.Fatalf("parseClusterSelectors(%q) = %v, want no error", tc.value, err) + } + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("parseClusterSelectors(%q) mismatch (-got, +want):\n%s", tc.value, diff) + } + }) + } +} + +// formatLimit renders a CRD size limit, which is absent as often as it is set. +func formatLimit(limit *int64) string { + if limit == nil { + return "no limit" + } + return strconv.FormatInt(*limit, 10) +} + +// TestLimitsMatchAPIValidation ties the parser's limits to the validation markers on the placement +// policy CRD. +// +// The parser turns an annotation that exceeds a limit into an error reported against the annotation +// itself. Were the API to tighten a limit on its own, the parser would instead keep building policy +// objects that the API server rejects, and the failure would surface as an endless retry far away +// from the annotation that caused it. +func TestLimitsMatchAPIValidation(t *testing.T) { + rawCRD, err := os.ReadFile(crdPath) + if err != nil { + t.Fatalf("ReadFile(%q) = %v, want no error", crdPath, err) + } + var crd apiextensionsv1.CustomResourceDefinition + if err := yaml.Unmarshal(rawCRD, &crd); err != nil { + t.Fatalf("Unmarshal(%q) = %v, want no error", crdPath, err) + } + + var schema *apiextensionsv1.JSONSchemaProps + for idx := range crd.Spec.Versions { + if crd.Spec.Versions[idx].Name == kfplacementv1alpha1.GroupVersion.Version { + schema = crd.Spec.Versions[idx].Schema.OpenAPIV3Schema + break + } + } + if schema == nil { + t.Fatalf("the %s CRD has no %s version", crd.Name, kfplacementv1alpha1.GroupVersion.Version) + } + + clusterSelectors := schema.Properties["spec"].Properties["clusterSelectors"] + if got := clusterSelectors.MaxItems; got == nil || *got != maxSelectors { + t.Errorf("the CRD caps clusterSelectors at %s, want maxSelectors (%d)", formatLimit(got), maxSelectors) + } + // Every step below walks into the schema, so a renamed or restructured field must stop the test + // with a message that names the missing step rather than panic on a nil pointer. + if clusterSelectors.Items == nil || clusterSelectors.Items.Schema == nil { + t.Fatalf("the CRD schema of clusterSelectors carries no item schema") + } + + selector := clusterSelectors.Items.Schema.Properties + terms := selector["terms"] + if terms.Items == nil || terms.Items.Schema == nil { + t.Fatalf("the CRD schema of the selector terms carries no item schema") + } + matchLabels := terms.Items.Schema.Properties["matchLabels"] + if got := matchLabels.MaxProperties; got == nil || *got != maxMatchLabels { + t.Errorf("the CRD caps matchLabels at %s, want maxMatchLabels (%d)", formatLimit(got), maxMatchLabels) + } + + // The count field is an int-or-string, and the API bounds each form separately: a pattern for + // the string form, checked here by exercising it, and a CEL rule for the integer form, checked + // here by asserting the rule spells out the same bounds the parser enforces. A pattern alone + // would leave the integer form unbounded. + count := selector["count"] + countPattern, err := regexp.Compile(count.Pattern) + if err != nil { + t.Fatalf("Compile(%q) = %v, want no error", count.Pattern, err) + } + if got := strconv.Itoa(maxCount); !countPattern.MatchString(got) { + t.Errorf("the CRD pattern %q rejects maxCount (%s), want it accepted", count.Pattern, got) + } + if got := strconv.Itoa(maxCount + 1); countPattern.MatchString(got) { + t.Errorf("the CRD pattern %q accepts %s, want maxCount (%d) to be the ceiling", count.Pattern, got, maxCount) + } + if !countPattern.MatchString(countAll) { + t.Errorf("the CRD pattern %q rejects %q, want it accepted", count.Pattern, countAll) + } + + // The rule's spelling is free to change; what must not drift is the ceiling it encodes, in + // both the rule and the message a user is shown. The behavior itself is pinned separately by + // the integration suite, which submits out-of-range counts to a real API server. + wantCeiling := fmt.Sprintf("%d", maxCount) + integerRuleFound := false + for _, validation := range count.XValidations { + if !strings.Contains(validation.Rule, wantCeiling) { + continue + } + integerRuleFound = true + if !strings.Contains(validation.Message, wantCeiling) { + t.Errorf("the CRD rejects an out-of-range count with the message %q, want it to name the ceiling (%s)", validation.Message, wantCeiling) + } + } + if !integerRuleFound { + rules := make([]string, 0, len(count.XValidations)) + for _, validation := range count.XValidations { + rules = append(rules, validation.Rule) + } + t.Errorf("the CRD validates count with the rules %q, want one to bound the integer form at maxCount (%s)", rules, wantCeiling) + } + + if count.Default == nil { + t.Fatalf("the CRD schema of the selector count carries no default") + } + wantDefault := strconv.Itoa(defaultCount) + if got := string(count.Default.Raw); got != wantDefault { + t.Errorf("the CRD defaults count to %s, want defaultCount (%s)", got, wantDefault) + } +} + +func TestParseClusterSelectorsInvalid(t *testing.T) { + testCases := []struct { + name string + // value is the annotation value under test. + value string + // wantErrContains is a fragment the error message must carry, so that the annotation's + // author can tell which part of the value was rejected and why. + wantErrContains string + }{ + { + name: "empty value", + value: "", + wantErrContains: "empty", + }, + { + name: "whitespace-only value", + value: " ", + wantErrContains: "empty", + }, + { + name: "more selectors than supported", + value: strings.TrimSuffix(strings.Repeat("region=eastus;", maxSelectors+1), ";"), + wantErrContains: "more than the supported maximum", + }, + { + // Selectors are counted from one in the error, since whoever wrote the annotation + // counts them by eye. + name: "empty selector between two others", + value: "region=eastus;;region=westus", + wantErrContains: "cluster selector 2", + }, + { + name: "empty selector reports the reason as well as the position", + value: "region=eastus;;region=westus", + wantErrContains: "empty label matcher", + }, + { + name: "trailing separator leaves an empty selector", + value: "region=eastus;", + wantErrContains: "empty label matcher", + }, + { + name: "trailing matcher separator", + value: "region=eastus,", + wantErrContains: "empty label matcher", + }, + { + name: "matcher without a value", + value: "region", + wantErrContains: "KEY=VALUE", + }, + { + name: "matcher with an empty key", + value: "=eastus", + wantErrContains: "empty label key", + }, + { + name: "label key that Kubernetes rejects", + value: "not a key=eastus", + wantErrContains: "is not valid", + }, + { + name: "label value that Kubernetes rejects", + value: "env=not a value", + wantErrContains: "is not valid", + }, + { + name: "label matched against an empty value", + value: "env=", + wantErrContains: "empty value", + }, + { + // A value one byte past what Kubernetes accepts. Left unchecked, this would become a + // policy object that the API server rejects, turning a typo into an endless retry. + name: "label value one byte too long", + value: "env=" + strings.Repeat("a", validation.LabelValueMaxLength+1), + wantErrContains: "is not valid", + }, + { + name: "label key one byte too long", + value: strings.Repeat("k", qualifiedNameMaxLength+1) + "=prod", + wantErrContains: "is not valid", + }, + { + // strings.Cut splits on the first separator only, so the value keeps the rest; it is + // then rejected, since a label value cannot contain a separator. + name: "value containing the key-value separator", + value: "env=prod=east", + wantErrContains: "is not valid", + }, + { + name: "same label key matched twice", + value: "env=staging,env=canary", + wantErrContains: "more than once", + }, + { + // The shorthand and the key it expands to are the same label, which would otherwise + // silently drop one of the two matchers. + name: "shorthand collides with its expanded key", + value: fmt.Sprintf("region=eastus,%s=westus", regionLabel), + wantErrContains: "more than once", + }, + { + name: "count set twice", + value: "env=prod,count=1,count=2", + wantErrContains: "more than once", + }, + { + name: "count of zero", + value: "env=prod,count=0", + wantErrContains: "out of the supported range", + }, + { + name: "count above the supported range", + value: fmt.Sprintf("env=prod,count=%d", maxCount+1), + wantErrContains: "out of the supported range", + }, + { + // All digits, but too large to hold; this must be reported like any other out-of-range + // count rather than escaping as a conversion failure. + name: "count too large to be represented", + value: "env=prod,count=99999999999999999999", + wantErrContains: "out of the supported range", + }, + { + name: "negative count", + value: "env=prod,count=-1", + wantErrContains: "neither a positive integer", + }, + { + name: "explicitly signed count", + value: "env=prod,count=+5", + wantErrContains: "neither a positive integer", + }, + { + name: "count that is not a number", + value: "env=prod,count=many", + wantErrContains: "neither a positive integer", + }, + { + // A near miss on the All sentinel is worth its own message; the API accepts the exact + // spelling only. + name: "lowercased All sentinel", + value: "env=prod,count=all", + wantErrContains: `spelled exactly "All"`, + }, + { + name: "uppercased All sentinel", + value: "env=prod,count=ALL", + wantErrContains: `spelled exactly "All"`, + }, + { + name: "more label matchers than supported in one selector", + value: func() string { + matchers := make([]string, 0, maxMatchLabels+1) + for i := 0; i <= maxMatchLabels; i++ { + matchers = append(matchers, fmt.Sprintf("key%d=value", i)) + } + return strings.Join(matchers, matcherSeparator) + }(), + wantErrContains: "more than the supported maximum", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := parseClusterSelectors(tc.value) + if err == nil { + t.Fatalf("parseClusterSelectors(%q) = %v, want an error", tc.value, got) + } + if !strings.Contains(err.Error(), tc.wantErrContains) { + t.Errorf("parseClusterSelectors(%q) = %v, want an error containing %q", tc.value, err, tc.wantErrContains) + } + }) + } +} diff --git a/pkg/v1/controllers/annotationplacement/policy.go b/pkg/v1/controllers/annotationplacement/policy.go new file mode 100644 index 000000000..733b7ddb3 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/policy.go @@ -0,0 +1,258 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "strings" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/naming" +) + +const ( + // maxKindSegmentLength caps how much of the kind appears in a generated name. Every kind a user + // is likely to annotate is far shorter; the cap exists so that a pathologically long custom + // kind cannot crowd out the part of the name that identifies the resource. + maxKindSegmentLength = 40 + + // separatorCount is the number of separators a generated name spends joining its three parts. + separatorCount = 2 +) + +// The values the placement policy API defaults these fields to. +// +// A generated policy sets them explicitly, rather than letting the API server fill them in, so +// that the object built here equals the one the API server stores. Left unset, every field the API +// defaults would read as a difference on every pass and provoke an update that changes nothing. +// +// whenUnfulfilled deliberately follows the API default of requesting a cluster: asking the platform +// for a cluster that does not exist yet is the point of these APIs, and the annotation is meant to +// be the lowest-friction way to do so. The grammar has no slot for the field, so guarding against +// a selector nothing can ever satisfy (a mistyped region, say) is the claim fulfillment layer's +// job -- quota, approval, expiry -- not a softer default here. +const ( + defaultResourceRevisionHistoryLimit int32 = 3 + + defaultWhenUnfulfilled = kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim +) + +// generatedPolicyName derives the name of the placement policy generated for a resource. +// +// The name is deterministic, so that reconciling the same resource twice converges on one policy +// instead of accumulating them, and it is readable, so that the policy can be recognized in a +// listing. Neither property may come at the cost of the third: the hash is taken over the +// resource's whole identity, including the parts that do not appear in the name at all, so two +// resources cannot generate one name even when truncation erases what distinguishes them. +func generatedPolicyName(gvk schema.GroupVersionKind, namespace, name string) string { + // The kind and the name are sanitized before they appear in the name: a resource legal for its + // own API can carry characters a DNS-1123 object name forbids (an RBAC name like + // system:aggregate-to-admin), and copying them through would produce a policy the API server + // rejects, hot-looping the reconciler. The hash below is taken over the unsanitized identity, so + // sanitizing the visible parts costs no uniqueness. + kindSegment := naming.Truncate(naming.Sanitize(gvk.Kind), maxKindSegmentLength) + + // Whatever the kind and the hash do not use is available to the resource's own name. + nameBudget := validation.DNS1123SubdomainMaxLength - len(kindSegment) - naming.HashLength - separatorCount + nameSegment := naming.Truncate(naming.Sanitize(name), nameBudget) + + // Empty segments are dropped rather than joined. An object always has a name, and one read + // from the API server always has a kind, but joining an empty leading segment would put a + // separator at the front of the name, where the API server does not allow one. + segments := make([]string, 0, 3) + for _, segment := range []string{kindSegment, nameSegment} { + if segment != "" { + segments = append(segments, segment) + } + } + identity := fmt.Sprintf("%s/%s/%s/%s", gvk.Group, gvk.Kind, namespace, name) + segments = append(segments, naming.Hash(identity)) + + return strings.Join(segments, "-") +} + +// parentLabels returns the labels recording which resource a generated policy came from. +// +// Every value is shortened to fit a label value, including the API group: a custom resource's +// group is validated as a DNS-1123 subdomain and so may run to 253 bytes, well past what a label +// value holds. A kind cannot exceed the limit today, but it is shortened alongside the others +// rather than relying on a bound that belongs to a different validation rule than this one. +func parentLabels(gvk schema.GroupVersionKind, name string) map[string]string { + return map[string]string{ + kfplacementv1alpha1.ParentAPIGroupLabel: naming.LabelValue(gvk.Group), + kfplacementv1alpha1.ParentKindLabel: naming.LabelValue(gvk.Kind), + kfplacementv1alpha1.ParentNameLabel: naming.LabelValue(name), + } +} + +// isGeneratedFor reports whether a live policy is one this controller generated for the given +// resource identity. It is what stops the controller from overwriting or deleting a policy a user or +// another tool authored at the deterministic name this resource happens to generate. +// +// Ownership is recognized from either marker this controller stamps -- the owner reference back to +// the source, or the provenance labels -- and does not require both. Both are things applyDesiredPolicy +// repairs, so demanding both be intact would let a single edited label or stripped owner reference +// classify one of this controller's own policies as foreign: it would then be denied the very repair +// that would restore the marker and, worse, denied deletion when its annotation is removed, leaving a +// placement running with no way to reconcile or clean it up. Recognizing either marker keeps the +// policy repairable as long as one survives; a foreign policy that merely collides with the name +// carries neither and is still left untouched. Both markers are derived from the resource's stable +// identity rather than its UID, so the judgment holds across a delete and recreate of the resource. +// +// The one residual gap is a policy that loses both markers at once, which is then indistinguishable +// from a foreign object and abandoned. It is reachable without malice -- a tool that rewrites the +// whole object, such as a plain kubectl apply of a hand-kept manifest or a non-server-side-apply +// Update, drops metadata.labels and metadata.ownerReferences together -- but a policy this controller +// generated is not one a user is expected to manage that way, and requiring both markers to survive +// is the accepted price of never touching a policy that is genuinely someone else's. +func isGeneratedFor(policy client.Object, gvk schema.GroupVersionKind, name string, scheme *runtime.Scheme) bool { + // The owner reference is matched on the source's group, kind, and name -- deliberately not on + // its version or its UID, both of which can change while the resource keeps its name: a source + // deleted and recreated comes back with a new UID, and a served version can be retired. + source := &unstructured.Unstructured{} + source.SetGroupVersionKind(gvk) + source.SetName(name) + if owned, err := controllerutil.HasOwnerReference(policy.GetOwnerReferences(), source, scheme); err == nil && owned { + return true + } + + labels := policy.GetLabels() + for key, want := range parentLabels(gvk, name) { + if labels[key] != want { + return false + } + } + return true +} + +// parentOwnerReference returns the owner reference that ties a generated policy to the resource it +// was generated from, so that deleting the resource collects the policy with it. +// +// The reference deliberately claims neither controller nor blocking ownership: setting either +// requires permission to update the finalizers subresource of the owner, and the owner here can be +// a resource of any kind at all. Cascading deletion, the only property this needs, works the same +// without them. +func parentOwnerReference(source *unstructured.Unstructured) metav1.OwnerReference { + gvk := source.GroupVersionKind() + return metav1.OwnerReference{ + APIVersion: gvk.GroupVersion().String(), + Kind: gvk.Kind, + Name: source.GetName(), + UID: source.GetUID(), + } +} + +// parentResourceSelector returns the selector that places the annotated resource itself. +// +// It is the only resource selector a generated policy has: an annotation asks for the resource it is +// written on to be placed, and nothing else. Should that ever stop being true, note that the +// reconciler replaces the whole spec of a generated policy, so a selector added from elsewhere would +// have to be reconciled here rather than merely appended to the live object. +func parentResourceSelector(source *unstructured.Unstructured) kfplacementv1alpha1.ResourceSelector { + gvk := source.GroupVersionKind() + // The namespace is deliberately left unset. A generated policy for a namespaced resource lives + // in that resource's own namespace, where the API requires the selector's namespace to be + // empty or identical; a generated policy for a cluster-scoped resource has no namespace to name. + return kfplacementv1alpha1.ResourceSelector{ + APIGroup: gvk.Group, + APIVersion: gvk.Version, + Kind: gvk.Kind, + Name: source.GetName(), + } +} + +// withAPIDefaults returns the selectors with every field the API defaults set explicitly, leaving +// the caller's own slice untouched. +func withAPIDefaults(selectors []kfplacementv1alpha1.ClusterSelector) []kfplacementv1alpha1.ClusterSelector { + if selectors == nil { + return nil + } + defaulted := make([]kfplacementv1alpha1.ClusterSelector, len(selectors)) + for i, selector := range selectors { + selector.DeepCopyInto(&defaulted[i]) + if defaulted[i].WhenUnfulfilled == "" { + defaulted[i].WhenUnfulfilled = defaultWhenUnfulfilled + } + } + return defaulted +} + +// desiredPolicy builds the placement policy that should exist for an annotated resource. +// +// The policy's scope follows the resource's own: a namespaced resource yields a PlacementPolicy in +// its namespace, and a cluster-scoped resource yields a ClusterPlacementPolicy. That is what keeps +// the owner reference valid, since a cluster-scoped object owned by a namespaced one is accepted +// on creation and then never collected at all. +// +// Scope is read from the resource's own namespace, which the API server has already reconciled +// with the scope its kind is registered under; the source must therefore be an object read from +// the API server rather than one assembled in memory, which is also what guarantees it carries the +// UID and the kind that the owner reference and the generated name are built from. Taking an +// unstructured object rather than a client.Object is deliberate for the same reason: a typed +// object routinely arrives with an empty kind, which would silently change the generated name. +func desiredPolicy(source *unstructured.Unstructured, selectors []kfplacementv1alpha1.ClusterSelector) kfplacementv1alpha1.PlacementPolicyAccessor { + gvk := source.GroupVersionKind() + namespace := source.GetNamespace() + + policy := emptyPolicyForScope(namespace) + policy.SetName(generatedPolicyName(gvk, namespace, source.GetName())) + policy.SetNamespace(namespace) + policy.SetLabels(parentLabels(gvk, source.GetName())) + policy.SetOwnerReferences([]metav1.OwnerReference{parentOwnerReference(source)}) + *policy.GetSpec() = kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: withAPIDefaults(selectors), + // Only the annotated resource is selected; nothing it depends on. Selecting dependencies + // is tracked in #927, where the questions are which references count, whether following + // them should be opt-in, and how the closure stays fresh as the source changes. + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{parentResourceSelector(source)}, + ResourceRevisionHistoryLimit: ptr.To(defaultResourceRevisionHistoryLimit), + } + return policy +} + +// emptyPolicyForScope returns an empty generated policy of the scope that a resource in the given +// namespace generates: a namespaced resource yields a PlacementPolicy in its own namespace, and a +// cluster-scoped resource, whose namespace is empty, yields a ClusterPlacementPolicy. +// +// This is the single place the scope is decided. The reconciler needs the same answer to read and to +// delete a generated policy as it does to build one, and a disagreement between those would leave a +// policy behind rather than fail. +func emptyPolicyForScope(namespace string) kfplacementv1alpha1.PlacementPolicyAccessor { + if namespace == "" { + return &kfplacementv1alpha1.ClusterPlacementPolicy{} + } + return &kfplacementv1alpha1.PlacementPolicy{} +} + +// generatedPolicyKind returns the kind of the policy generated for a resource in the given +// namespace, spelled as the kind itself is, so that a message carrying it can be pasted straight +// into a kubectl command against the right resource. +func generatedPolicyKind(namespace string) string { + if namespace == "" { + return "ClusterPlacementPolicy" + } + return "PlacementPolicy" +} diff --git a/pkg/v1/controllers/annotationplacement/policy_test.go b/pkg/v1/controllers/annotationplacement/policy_test.go new file mode 100644 index 000000000..1c06bd9a3 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/policy_test.go @@ -0,0 +1,463 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/naming" +) + +var ( + deploymentGVK = schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"} + namespaceGVK = schema.GroupVersionKind{Group: "", Version: "v1", Kind: "Namespace"} +) + +// sourceObject builds the annotated resource a policy is generated from. +func sourceObject(gvk schema.GroupVersionKind, namespace, name string) *unstructured.Unstructured { + source := &unstructured.Unstructured{} + source.SetGroupVersionKind(gvk) + source.SetNamespace(namespace) + source.SetName(name) + source.SetUID(types.UID("uid-" + name)) + return source +} + +func TestGeneratedPolicyName(t *testing.T) { + testCases := []struct { + name string + gvk schema.GroupVersionKind + namespace string + objName string + want string + }{ + { + name: "namespaced resource", + gvk: deploymentGVK, + namespace: "prod", + objName: "app", + want: "deployment-app-" + hashOfIdentity("apps", "Deployment", "prod", "app"), + }, + { + name: "cluster-scoped resource", + gvk: namespaceGVK, + objName: "work", + want: "namespace-work-" + hashOfIdentity("", "Namespace", "", "work"), + }, + { + // The kind is lowercased so that the name is a legal DNS-1123 subdomain; the hash is + // taken over the kind as it is really spelled, so two kinds that differ only in case + // cannot generate the same name. + name: "kind is lowercased in the name but not in the identity", + gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: "MyWidget"}, + namespace: "prod", + objName: "gadget", + want: "mywidget-gadget-" + hashOfIdentity("example.com", "MyWidget", "prod", "gadget"), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := generatedPolicyName(tc.gvk, tc.namespace, tc.objName) + if got != tc.want { + t.Errorf("generatedPolicyName(%v, %q, %q) = %v, want %v", tc.gvk, tc.namespace, tc.objName, got, tc.want) + } + }) + } +} + +func hashOfIdentity(group, kind, namespace, name string) string { + return naming.Hash(fmt.Sprintf("%s/%s/%s/%s", group, kind, namespace, name)) +} + +// TestGeneratedPolicyNameValidity covers the inputs where the name has to be shortened to fit, +// which is where a generated name has twice gone wrong in this project: the result must stay a +// legal object name, and it must still tell two resources apart after the part that distinguishes +// them has been cut away. +func TestGeneratedPolicyNameValidity(t *testing.T) { + longKind := "My" + strings.Repeat("Widget", 40) + longName := strings.Repeat("a", validation.DNS1123SubdomainMaxLength) + + testCases := []struct { + name string + gvk schema.GroupVersionKind + namespace string + objName string + }{ + {name: "short everything", gvk: deploymentGVK, namespace: "prod", objName: "app"}, + { + // An RBAC name carries a colon, legal for the resource but not for a Kubernetes object + // name; without sanitizing it the generated name is rejected and the reconciler hot-loops. + name: "rbac name with a colon", + gvk: schema.GroupVersionKind{Group: "rbac.authorization.k8s.io", Version: "v1", Kind: "ClusterRole"}, + objName: "system:aggregate-to-admin", + }, + { + // A dot beside an illegal character is the case that a naive sanitizer keeping dots gets + // wrong: it would leave a label ending in a dash, which the API server rejects. Resource + // names commonly carry dots (domain-like custom resource names), so this is reachable. + name: "name with a dot beside an illegal character", + gvk: deploymentGVK, + objName: "my.:app", + }, + {name: "name at the object limit", gvk: deploymentGVK, namespace: "prod", objName: longName}, + { + name: "name truncated onto a separator", + gvk: deploymentGVK, + namespace: "prod", + objName: strings.Repeat("a", 194) + "-" + strings.Repeat("b", 40), + }, + { + name: "name truncated onto a dot", + gvk: deploymentGVK, + namespace: "prod", + objName: strings.Repeat("a", 194) + "." + strings.Repeat("b", 40), + }, + { + name: "kind longer than its budget", + gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: longKind}, + namespace: "prod", + objName: longName, + }, + { + // An object read from the API server always carries its kind, but an empty one must + // not put a separator at the front of the generated name, where it would be rejected. + name: "empty kind", + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1"}, + namespace: "prod", + objName: "app", + }, + { + name: "empty kind and empty name", + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1"}, + namespace: "prod", + }, + } + + seen := make(map[string]string, len(testCases)) + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := generatedPolicyName(tc.gvk, tc.namespace, tc.objName) + if errs := validation.IsDNS1123Subdomain(got); len(errs) > 0 { + t.Errorf("generatedPolicyName(%v, %q, %q) = %v, want a valid object name: %s", + tc.gvk, tc.namespace, tc.objName, got, strings.Join(errs, "; ")) + } + if len(got) > validation.DNS1123SubdomainMaxLength { + t.Errorf("len(generatedPolicyName(%v, %q, %q)) = %d, want at most %d", + tc.gvk, tc.namespace, tc.objName, len(got), validation.DNS1123SubdomainMaxLength) + } + if previous, collided := seen[got]; collided { + t.Errorf("generatedPolicyName(%v, %q, %q) = %v, want a name distinct from that of %s", + tc.gvk, tc.namespace, tc.objName, got, previous) + } + seen[got] = tc.name + }) + } +} + +// TestGeneratedPolicyNameDistinguishesIdentities pins the property the hash exists for: resources +// that differ anywhere in their identity, including in parts the readable name never shows, must +// not generate the same policy name. +func TestGeneratedPolicyNameDistinguishesIdentities(t *testing.T) { + longName := strings.Repeat("a", 300) + + identities := []struct { + name string + gvk schema.GroupVersionKind + namespace string + objName string + }{ + {name: "baseline", gvk: deploymentGVK, namespace: "prod", objName: "app"}, + {name: "different namespace", gvk: deploymentGVK, namespace: "staging", objName: "app"}, + {name: "different api group, same kind", gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: "Deployment"}, namespace: "prod", objName: "app"}, + {name: "kind differing only in case", gvk: schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "DeploymenT"}, namespace: "prod", objName: "app"}, + // Two names that sanitize to the same visible segment -- a dot and a dash both render as a + // dash -- must still generate distinct policy names, since the hash is over the raw identity. + {name: "name with a dot", gvk: deploymentGVK, namespace: "prod", objName: "my.app"}, + {name: "name with a dash where the other has a dot", gvk: deploymentGVK, namespace: "prod", objName: "my-app"}, + // Two names identical up to well past the truncation point. + {name: "long name", gvk: deploymentGVK, namespace: "prod", objName: longName + "one"}, + {name: "long name sharing a prefix", gvk: deploymentGVK, namespace: "prod", objName: longName + "two"}, + } + + seen := make(map[string]string, len(identities)) + for _, identity := range identities { + got := generatedPolicyName(identity.gvk, identity.namespace, identity.objName) + if previous, collided := seen[got]; collided { + t.Errorf("generatedPolicyName for %s = %v, want a name distinct from that of %s", identity.name, got, previous) + } + seen[got] = identity.name + } +} + +// TestIsGeneratedFor pins the check that keeps this controller from commandeering or deleting a +// policy someone else authored at a resource's generated name, while still recognizing its own policy +// after either provenance marker has drifted so that the policy stays repairable and removable. +func TestIsGeneratedFor(t *testing.T) { + source := sourceObject(deploymentGVK, "prod", "app") + mine := desiredPolicy(source, nil) + + // The same policy with its provenance labels edited away; the owner reference still identifies it. + labelsStripped := mine.DeepCopyObject().(client.Object) + labelsStripped.SetLabels(nil) + + // The same policy with its owner reference removed; the provenance labels still identify it. + ownerStripped := mine.DeepCopyObject().(client.Object) + ownerStripped.SetOwnerReferences(nil) + + // The same policy stripped of both markers; nothing is left to tell it from a foreign object. + bothStripped := mine.DeepCopyObject().(client.Object) + bothStripped.SetLabels(nil) + bothStripped.SetOwnerReferences(nil) + + // A policy at the same name authored by someone else, carrying neither marker. + foreign := emptyPolicyForScope("prod") + foreign.SetName(mine.GetName()) + foreign.SetNamespace("prod") + + // A policy that carries another resource's provenance -- the collision a bare name match would + // miss -- and no owner reference to this source. + otherSource := sourceObject(deploymentGVK, "prod", "other") + otherLabelled := emptyPolicyForScope("prod") + otherLabelled.SetName(mine.GetName()) + otherLabelled.SetNamespace("prod") + otherLabelled.SetLabels(parentLabels(otherSource.GroupVersionKind(), otherSource.GetName())) + + testCases := []struct { + name string + policy client.Object + want bool + }{ + {name: "the policy this controller generated", policy: mine, want: true}, + {name: "our policy with its labels stripped is still recognized by its owner reference", policy: labelsStripped, want: true}, + {name: "our policy with its owner reference stripped is still recognized by its labels", policy: ownerStripped, want: true}, + {name: "our policy with both markers gone is indistinguishable from foreign", policy: bothStripped, want: false}, + {name: "a foreign policy with neither marker", policy: foreign, want: false}, + {name: "a policy carrying another resource's provenance", policy: otherLabelled, want: false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := isGeneratedFor(tc.policy, source.GroupVersionKind(), source.GetName(), newScheme(t)); got != tc.want { + t.Errorf("isGeneratedFor(%q) = %v, want %v", tc.policy.GetName(), got, tc.want) + } + }) + } +} + +func TestDesiredPolicy(t *testing.T) { + // The parser leaves whenUnfulfilled unset, since the annotation grammar cannot express it; the + // generated policy carries the API's default explicitly so that it does not read as drift. + selectors := []kfplacementv1alpha1.ClusterSelector{ + { + Count: ptr.To(intstr.FromInt32(1)), + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{"env": "prod"}}, + }, + }, + } + wantSelectors := []kfplacementv1alpha1.ClusterSelector{ + { + Count: ptr.To(intstr.FromInt32(1)), + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{"env": "prod"}}, + }, + WhenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + } + + testCases := []struct { + name string + source *unstructured.Unstructured + want client.Object + }{ + { + name: "namespaced resource yields a namespaced policy", + source: sourceObject(deploymentGVK, "prod", "app"), + want: &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: "deployment-app-" + hashOfIdentity("apps", "Deployment", "prod", "app"), + Namespace: "prod", + Labels: map[string]string{ + kfplacementv1alpha1.ParentAPIGroupLabel: "apps", + kfplacementv1alpha1.ParentKindLabel: "Deployment", + kfplacementv1alpha1.ParentNameLabel: "app", + }, + OwnerReferences: []metav1.OwnerReference{{ + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "app", + UID: types.UID("uid-app"), + }}, + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: wantSelectors, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Name: "app", + }}, + ResourceRevisionHistoryLimit: ptr.To(defaultResourceRevisionHistoryLimit), + }, + }, + }, + { + name: "cluster-scoped resource yields a cluster-scoped policy", + source: sourceObject(namespaceGVK, "", "work"), + want: &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: "namespace-work-" + hashOfIdentity("", "Namespace", "", "work"), + Labels: map[string]string{ + // The core API group is the empty string, and the label is present all the + // same, so that core-group resources can be selected like any other. + kfplacementv1alpha1.ParentAPIGroupLabel: "", + kfplacementv1alpha1.ParentKindLabel: "Namespace", + kfplacementv1alpha1.ParentNameLabel: "work", + }, + OwnerReferences: []metav1.OwnerReference{{ + APIVersion: "v1", + Kind: "Namespace", + Name: "work", + UID: types.UID("uid-work"), + }}, + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: wantSelectors, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIVersion: "v1", + Kind: "Namespace", + Name: "work", + }}, + ResourceRevisionHistoryLimit: ptr.To(defaultResourceRevisionHistoryLimit), + }, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := desiredPolicy(tc.source, selectors) + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("desiredPolicy() mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// TestParentOwnerReferenceIsNotControlling pins the choice not to claim controller or blocking +// ownership: either would require permission to update the finalizers subresource of a resource of +// any kind the user cares to annotate. +func TestParentOwnerReferenceIsNotControlling(t *testing.T) { + got := parentOwnerReference(sourceObject(deploymentGVK, "prod", "app")) + if got.Controller != nil { + t.Errorf("parentOwnerReference().Controller = %v, want nil", *got.Controller) + } + if got.BlockOwnerDeletion != nil { + t.Errorf("parentOwnerReference().BlockOwnerDeletion = %v, want nil", *got.BlockOwnerDeletion) + } +} + +// TestParentNameLabelIsShortenedWhenTooLong covers a resource whose name cannot fit in a label +// value, where the label is knowingly lossy and the owner reference is what remains exact. +func TestParentNameLabelIsShortenedWhenTooLong(t *testing.T) { + longName := strings.Repeat("a", 200) + source := sourceObject(deploymentGVK, "prod", longName) + + policy := desiredPolicy(source, nil) + labels := policy.GetLabels() + got := labels[kfplacementv1alpha1.ParentNameLabel] + + if got == longName { + t.Errorf("parent name label = the full name, want it shortened to fit a label value") + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("parent name label = %v, want a valid label value: %s", got, strings.Join(errs, "; ")) + } + if wantOwner := longName; policy.GetOwnerReferences()[0].Name != wantOwner { + t.Errorf("owner reference name = %v, want the unshortened %v", policy.GetOwnerReferences()[0].Name, wantOwner) + } +} + +// TestParentLabelsAreValid covers the label values that have to be shortened to fit. A custom +// resource's API group is validated as a DNS-1123 subdomain and so runs to 253 bytes, four times +// what a label value holds; left unshortened it would make every generated policy for that +// resource fail validation on creation, with nothing but a retrying reconciler to show for it. +func TestParentLabelsAreValid(t *testing.T) { + testCases := []struct { + name string + gvk schema.GroupVersionKind + objName string + }{ + {name: "ordinary resource", gvk: deploymentGVK, objName: "app"}, + {name: "core group resource", gvk: namespaceGVK, objName: "work"}, + { + name: "api group too long for a label value", + gvk: schema.GroupVersionKind{Group: strings.Repeat("g", 90) + ".example.com", Version: "v1", Kind: "Widget"}, + objName: "gadget", + }, + { + name: "kind at the longest a kind can be", + gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: strings.Repeat("W", 63)}, + objName: "gadget", + }, + {name: "name too long for a label value", gvk: deploymentGVK, objName: strings.Repeat("a", 200)}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + for key, value := range parentLabels(tc.gvk, tc.objName) { + if errs := validation.IsValidLabelValue(value); len(errs) > 0 { + t.Errorf("parentLabels()[%s] = %v, want a valid label value: %s", key, value, strings.Join(errs, "; ")) + } + } + }) + } +} + +// TestDesiredPolicySetsDefaultedFields pins that the generated policy carries a value for every +// field the API server would otherwise default. A field left unset is stored with the API's +// default and then reads as a difference on the next pass, which would have the reconciler issue +// an update that changes nothing, forever. +func TestDesiredPolicySetsDefaultedFields(t *testing.T) { + selectors := []kfplacementv1alpha1.ClusterSelector{{Count: ptr.To(intstr.FromInt32(1))}} + spec := desiredPolicy(sourceObject(deploymentGVK, "prod", "app"), selectors).GetSpec() + + if got := spec.ResourceRevisionHistoryLimit; got == nil || *got != defaultResourceRevisionHistoryLimit { + t.Errorf("resourceRevisionHistoryLimit = %v, want %d", got, defaultResourceRevisionHistoryLimit) + } + if got := spec.ClusterSelectors[0].WhenUnfulfilled; got != defaultWhenUnfulfilled { + t.Errorf("whenUnfulfilled = %v, want %v", got, defaultWhenUnfulfilled) + } + // The caller's own selectors must not have been defaulted in place. + if got := selectors[0].WhenUnfulfilled; got != "" { + t.Errorf("the caller's selector whenUnfulfilled = %v, want it left empty", got) + } +} diff --git a/pkg/v1/controllers/annotationplacement/suite_test.go b/pkg/v1/controllers/annotationplacement/suite_test.go new file mode 100644 index 000000000..9acbb1b6b --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/suite_test.go @@ -0,0 +1,96 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/discovery" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/restmapper" + "k8s.io/client-go/tools/events" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var configMapGVK = schema.GroupVersionKind{Group: "", Version: "v1", Kind: "ConfigMap"} + +var ( + testEnv *envtest.Environment + restConfig *rest.Config + hubClient client.Client + reconciler *Reconciler + eventRecorder *events.FakeRecorder + ctx context.Context + cancel context.CancelFunc +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Annotation Based Placement Controller Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("bootstrapping the test environment") + testEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + var err error + restConfig, err = testEnv.Start() + Expect(err).Should(Succeed()) + Expect(restConfig).NotTo(BeNil()) + + Expect(kfplacementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + hubClient, err = client.New(restConfig, client.Options{Scheme: scheme.Scheme}) + Expect(err).Should(Succeed()) + + discoveryClient, err := discovery.NewDiscoveryClientForConfig(restConfig) + Expect(err).Should(Succeed()) + groupResources, err := restmapper.GetAPIGroupResources(discoveryClient) + Expect(err).Should(Succeed()) + + // The recorder is buffered generously: an unread event blocks the reconciler that records it, + // which would surface as a timeout somewhere unrelated rather than as a failed expectation. + eventRecorder = events.NewFakeRecorder(100) + reconciler = &Reconciler{ + // The envtest client reads straight from the API server, so it serves as the API reader + // too; there is no cache to fall behind here. + Client: hubClient, + APIReader: hubClient, + RESTMapper: restmapper.NewDiscoveryRESTMapper(groupResources), + Recorder: eventRecorder, + } +}) + +var _ = AfterSuite(func() { + defer func() { + Expect(testEnv.Stop()).Should(Succeed()) + }() + cancel() +}) diff --git a/pkg/v1/controllers/annotationplacement/watch.go b/pkg/v1/controllers/annotationplacement/watch.go new file mode 100644 index 000000000..973bf6c54 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/watch.go @@ -0,0 +1,192 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "sync" + + "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/predicate" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +const controllerName = "annotation-placement-controller" + +// SourceQueue is how the resource watcher hands annotated resources to this controller. +// +// The annotated resources are of any kind the hub agent watches, which the manager's cache does not +// know ahead of time, so they cannot be watched through the manager and must be fed in instead. +// The feed is a source rather than a channel so that the producer is never blocked and repeated +// reports of one resource collapse: Add writes straight into the controller's own workqueue, which +// never blocks a writer and drops a request already queued for the same resource. A channel would +// push both problems onto the resource watcher, which has a whole fleet's events to deliver and no +// business waiting on this controller. +// +// Resources reported before the manager starts are held until it does, so the watcher does not have +// to know when the controller became ready. That buffer is unbounded and is only ever drained by +// the controller calling Start, so a producer must not outlive the manager it was created for: in +// the hub agent both are started together under the same leader election. +// +// The zero value is ready to use. +type SourceQueue struct { + mu sync.Mutex + queue workqueue.TypedRateLimitingInterface[Request] + pending map[Request]struct{} +} + +// Add reports an annotated resource. It never blocks, and it is safe to call from any goroutine +// and before the manager has started. +func (q *SourceQueue) Add(object client.Object) { + req := RequestFor(object) + if req.Kind == "" { + klog.ErrorS(nil, "Skipped a resource that names no kind", "obj", klog.KObj(object)) + return + } + + q.mu.Lock() + defer q.mu.Unlock() + if q.queue == nil { + if q.pending == nil { + q.pending = make(map[Request]struct{}) + } + // Collapsed the same way the controller's queue collapses, so a resource reported many + // times before the manager started still costs one reconcile once it has. + q.pending[req] = struct{}{} + return + } + q.queue.Add(req) +} + +// Start hands the queue to the controller. It is called by the controller, never directly. +func (q *SourceQueue) Start(_ context.Context, queue workqueue.TypedRateLimitingInterface[Request]) error { + q.mu.Lock() + defer q.mu.Unlock() + q.queue = queue + for req := range q.pending { + queue.Add(req) + } + q.pending = nil + return nil +} + +// SetupWithManager registers the controller with the manager. +// +// Annotated resources arrive through the given SourceQueue -- fed by the resource watcher, in the +// hub agent -- rather than from a watch of their own. The generated policies are watched through +// the manager, and an event on one enqueues the resource it was generated from: without that, an +// edit or a deletion of a generated policy produces no event on its resource, and for a resource +// that never changes again the policy would stay missing or wrong forever. +// +// The caller owns what the controller cannot decide for itself: the placement.kubefleet.dev types +// must be registered in the manager's scheme, and the hub agent must hold RBAC to read, create, +// update, and delete the two policy kinds. +func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, sources *SourceQueue) error { + if sources == nil { + // Reported here rather than left to panic once the manager starts the source: the caller + // already checks this error, and a controller with nothing feeding it places nothing. + return kferrors.NewUnexpectedError(nil, "the annotated resource source is nil", "controller", controllerName) + } + return builder.TypedControllerManagedBy[Request](mgr). + Named(controllerName). + WatchesRawSource(sources). + Watches( + &kfplacementv1alpha1.PlacementPolicy{}, + handler.TypedEnqueueRequestsFromMapFunc(mapGeneratedPolicyToSource), + builder.WithPredicates(generatedPolicyDrift()), + ). + Watches( + &kfplacementv1alpha1.ClusterPlacementPolicy{}, + handler.TypedEnqueueRequestsFromMapFunc(mapGeneratedPolicyToSource), + builder.WithPredicates(generatedPolicyDrift()), + ). + Complete(r) +} + +// generatedPolicyDrift admits the policy events that can mean a generated policy needs repair: a +// creation or deletion, and an update to the spec (which bumps the generation), the labels, or the +// owner references -- exactly what applyDesiredPolicy writes. Status updates are filtered out: the +// controllers that schedule a policy write its status on every change of the fleet, and each event +// let through here costs an uncached read of the annotated resource to learn that nothing drifted. +func generatedPolicyDrift() predicate.Predicate { + ownerReferencesChanged := predicate.Funcs{ + UpdateFunc: func(e event.UpdateEvent) bool { + return !equality.Semantic.DeepEqual(e.ObjectOld.GetOwnerReferences(), e.ObjectNew.GetOwnerReferences()) + }, + } + return predicate.Or(predicate.GenerationChangedPredicate{}, predicate.LabelChangedPredicate{}, ownerReferencesChanged) +} + +// mapGeneratedPolicyToSource enqueues the resource that generated a policy, identified from the +// policy's owner references. +// +// Only the owner whose identity reproduces this policy's own generated name is enqueued. A policy +// may carry more than one owner reference -- a foreign one that applyDesiredPolicy deliberately +// preserves, or any owner on a hand-authored policy that shares these watches -- and following +// those would enqueue a request for a kind the resource watcher does not track. Matching the +// generated name is exact, so only the true source passes. +// +// The owner reference is used rather than the parent labels because the labels are lossy (a long +// name is shortened to a prefix and a hash) and, being labels, can be stripped -- which is itself +// drift this watch exists to repair. On an update both the old and the new object are mapped, so an +// owner reference present only on the old side still enqueues the resource whose policy the update +// just took away from it. +func mapGeneratedPolicyToSource(_ context.Context, policy client.Object) []Request { + // A generated policy always lives in the namespace of the resource it came from, and a + // cluster-scoped one has none, matching a cluster-scoped owner. + policyName, policyNamespace := policy.GetName(), policy.GetNamespace() + for _, owner := range policy.GetOwnerReferences() { + gv, err := schema.ParseGroupVersion(owner.APIVersion) + if err != nil { + klog.ErrorS(err, "Skipped an owner with an unparsable API version", "policy", klog.KObj(policy), "apiVersion", owner.APIVersion) + continue + } + gvk := gv.WithKind(owner.Kind) + if generatedPolicyName(gvk, policyNamespace, owner.Name) != policyName { + // Not the owner this policy was generated from; following it would reach a resource + // the watcher never selected. + continue + } + return []Request{{ + GroupVersionKind: gvk, + NamespacedName: client.ObjectKey{Namespace: policyNamespace, Name: owner.Name}, + }} + } + return nil +} + +// SourceNamespace returns the namespace whose skip-status governs whether a source resource is +// placed. For an ordinary namespaced resource that is its metadata.namespace; a core Namespace +// object carries none and is itself the namespace, so its own name is returned. The skip check +// would otherwise read an empty namespace for a Namespace source and place one KubeFleet excludes. +func SourceNamespace(source *unstructured.Unstructured) string { + if gvk := source.GroupVersionKind(); gvk.Group == "" && gvk.Kind == "Namespace" { + return source.GetName() + } + return source.GetNamespace() +} diff --git a/pkg/v1/controllers/annotationplacement/watch_test.go b/pkg/v1/controllers/annotationplacement/watch_test.go new file mode 100644 index 000000000..166003548 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/watch_test.go @@ -0,0 +1,287 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/util/workqueue" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + + kfplacementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var configMapSourceGVK = schema.GroupVersionKind{Group: "", Version: "v1", Kind: "ConfigMap"} + +// generatedPolicyOwnedBy builds a policy as the cache would deliver it: correctly named for the +// resource that generated it, owned by that resource, plus any extra owner references a caller +// wants to plant. A policy whose name does not derive from an owner is that owner's, which is what +// lets the map function tell the true source from a foreign owner or a hand-authored policy. +func generatedPolicyOwnedBy(source *unstructured.Unstructured, extraOwners ...metav1.OwnerReference) client.Object { + namespace := source.GetNamespace() + policy := emptyPolicyForScope(namespace) + policy.SetNamespace(namespace) + policy.SetName(generatedPolicyName(source.GroupVersionKind(), namespace, source.GetName())) + policy.SetOwnerReferences(append([]metav1.OwnerReference{parentOwnerReference(source)}, extraOwners...)) + return policy +} + +// namedPolicy builds a policy with an explicit name (not derived from any owner), for the cases a +// generating source is not supposed to be found -- a hand-authored policy, or one whose only owners +// are foreign. +func namedPolicy(namespace, name string, owners ...metav1.OwnerReference) client.Object { + policy := &kfplacementv1alpha1.PlacementPolicy{} + policy.SetNamespace(namespace) + policy.SetName(name) + policy.SetOwnerReferences(owners) + return policy +} + +func ownerRef(apiVersion, kind, name string) metav1.OwnerReference { + return metav1.OwnerReference{APIVersion: apiVersion, Kind: kind, Name: name, UID: "00000000-0000-0000-0000-000000000001"} +} + +func TestMapGeneratedPolicyToSource(t *testing.T) { + deploymentSource := sourceObject(deploymentGVK, "prod", "web") + configMapSource := sourceObject(configMapSourceGVK, "prod", "app") + namespaceSource := sourceObject(namespaceGVK, "", "team") + + deploymentRequest := Request{GroupVersionKind: deploymentGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "web"}} + + testCases := []struct { + name string + policy client.Object + want []Request + }{ + { + name: "a generated policy enqueues the resource that generated it", + policy: generatedPolicyOwnedBy(deploymentSource), + want: []Request{deploymentRequest}, + }, + { + name: "a core group generating resource keeps its empty group", + policy: generatedPolicyOwnedBy(configMapSource), + want: []Request{{GroupVersionKind: configMapSourceGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "app"}}}, + }, + { + // A foreign owner reference -- one applyDesiredPolicy preserves, or any owner on a policy + // that merely shares these watches -- must not be followed, or the queue would hold a + // request for a kind the resource watcher never selected. + name: "a foreign owner reference is not followed", + policy: generatedPolicyOwnedBy(deploymentSource, ownerRef("v1", "Pod", "some-pod"), ownerRef("apps/v1", "Deployment", "unrelated")), + want: []Request{deploymentRequest}, + }, + { + // A hand-authored policy is owned by nothing this controller generated: its name does + // not derive from its owner, so no owner is followed. + name: "a policy whose name does not derive from its owner enqueues nothing", + policy: namedPolicy("prod", "hand-authored", ownerRef("apps/v1", "Deployment", "web")), + want: nil, + }, + { + name: "a policy with no owners enqueues nothing", + policy: namedPolicy("prod", "hand-authored"), + want: nil, + }, + { + // ParseGroupVersion rejects a second slash; the owner is skipped rather than turned + // into a bogus kind on the queue. + name: "an owner with an unparsable API version is skipped", + policy: generatedPolicyOwnedBy(deploymentSource, ownerRef("apps/v1/extra", "Deployment", "web")), + want: []Request{deploymentRequest}, + }, + { + name: "a cluster scoped policy enqueues a cluster scoped source", + policy: generatedPolicyOwnedBy(namespaceSource), + want: []Request{{GroupVersionKind: namespaceGVK, NamespacedName: client.ObjectKey{Name: "team"}}}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := mapGeneratedPolicyToSource(context.Background(), tc.policy) + if diff := cmp.Diff(got, tc.want, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("mapGeneratedPolicyToSource() mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// newTestQueue returns the controller workqueue a source is started with, so that a test can see +// exactly what a SourceQueue enqueued. +func newTestQueue() *testQueue { + return &testQueue{TypedRateLimitingInterface: workqueue.NewTypedRateLimitingQueue[Request](workqueue.DefaultTypedControllerRateLimiter[Request]())} +} + +type testQueue struct { + workqueue.TypedRateLimitingInterface[Request] +} + +// drain returns every queued request in order, leaving the queue empty. +func (q *testQueue) drain() []Request { + var got []Request + for q.Len() > 0 { + req, shutdown := q.Get() + if shutdown { + break + } + got = append(got, req) + q.Done(req) + } + return got +} + +func TestSourceQueueAdd(t *testing.T) { + testCases := []struct { + name string + source client.Object + want []Request + }{ + { + name: "an annotated resource is enqueued under its kind and namespaced name", + source: sourceObject(deploymentGVK, "prod", "web"), + want: []Request{{GroupVersionKind: deploymentGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "web"}}}, + }, + { + name: "a cluster scoped resource is enqueued with no namespace", + source: sourceObject(namespaceGVK, "", "team"), + want: []Request{{GroupVersionKind: namespaceGVK, NamespacedName: client.ObjectKey{Name: "team"}}}, + }, + { + // A typed object routinely arrives with an empty kind; a request built from it would + // name no API to read the resource from, so it is dropped rather than enqueued. + name: "an object that names no kind is dropped", + source: &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Namespace: "prod", Name: "web"}}, + want: nil, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + q := &SourceQueue{} + queue := newTestQueue() + if err := q.Start(context.Background(), queue); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + q.Add(tc.source) + + if diff := cmp.Diff(queue.drain(), tc.want, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("SourceQueue.Add() enqueued (-got, +want):\n%s", diff) + } + }) + } +} + +// TestSourceNamespace covers the skip-check namespace: a namespaced resource uses its own +// namespace, but a Namespace object -- which carries none -- is itself the namespace. +func TestSourceNamespace(t *testing.T) { + testCases := []struct { + name string + source *unstructured.Unstructured + want string + }{ + {name: "a namespaced resource uses its namespace", source: sourceObject(deploymentGVK, "prod", "web"), want: "prod"}, + {name: "a Namespace object is itself the namespace", source: sourceObject(namespaceGVK, "", "team"), want: "team"}, + {name: "another cluster scoped resource has no namespace", source: sourceObject(schema.GroupVersionKind{Group: "rbac.authorization.k8s.io", Version: "v1", Kind: "ClusterRole"}, "", "admin"), want: ""}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := SourceNamespace(tc.source); got != tc.want { + t.Errorf("SourceNamespace(%s %s) = %q, want %q", tc.source.GetKind(), tc.source.GetName(), got, tc.want) + } + }) + } +} + +// TestGeneratedPolicyDrift pins which policy updates reach the reconciler: what applyDesiredPolicy +// writes must, a status write must not. +func TestGeneratedPolicyDrift(t *testing.T) { + base := func() *kfplacementv1alpha1.PlacementPolicy { + policy := generatedPolicyOwnedBy(sourceObject(deploymentGVK, "prod", "web")).(*kfplacementv1alpha1.PlacementPolicy) + policy.Generation = 1 + return policy + } + + testCases := []struct { + name string + mutate func(*kfplacementv1alpha1.PlacementPolicy) + want bool + }{ + {name: "status only", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { + p.ResourceVersion = "2" + p.Status.Conditions = []metav1.Condition{{Type: "Scheduled", Status: metav1.ConditionTrue}} + }}, + {name: "spec edit bumps the generation", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { p.Generation = 2 }, want: true}, + {name: "label change", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { p.Labels = map[string]string{"drift": "yes"} }, want: true}, + {name: "owner reference stripped", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { p.OwnerReferences = nil }, want: true}, + } + + pred := generatedPolicyDrift() + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + updated := base() + tc.mutate(updated) + if got := pred.Update(event.UpdateEvent{ObjectOld: base(), ObjectNew: updated}); got != tc.want { + t.Errorf("generatedPolicyDrift().Update() = %v, want %v", got, tc.want) + } + }) + } +} + +// TestSourceQueueHoldsUntilStarted pins that a resource reported before the manager started is not +// lost: the watcher does not have to know when this controller became ready. +func TestSourceQueueHoldsUntilStarted(t *testing.T) { + q := &SourceQueue{} + q.Add(sourceObject(deploymentGVK, "prod", "web")) + + queue := newTestQueue() + if err := q.Start(context.Background(), queue); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + + want := []Request{{GroupVersionKind: deploymentGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "web"}}} + if diff := cmp.Diff(queue.drain(), want); diff != "" { + t.Errorf("SourceQueue.Add() before Start enqueued (-got, +want):\n%s", diff) + } +} + +// TestSourceQueueDeduplicates pins the reason this is a queue rather than a channel: a burst of +// reports for one resource costs one reconcile, not one per report. +func TestSourceQueueDeduplicates(t *testing.T) { + q := &SourceQueue{} + queue := newTestQueue() + if err := q.Start(context.Background(), queue); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + + source := sourceObject(deploymentGVK, "prod", "web") + for i := 0; i < 5; i++ { + q.Add(source) + } + + if got := queue.Len(); got != 1 { + t.Errorf("SourceQueue.Add() x5 queued %d items, want 1", got) + } +} From 000611fecca831225a6c588ca05150fb98b8827b Mon Sep 17 00:00:00 2001 From: Chen Yu Date: Wed, 23 Sep 2026 13:12:17 +0800 Subject: [PATCH 02/20] interface: [FEP-0001] add APIs for stage update run APIs for placement policies (#925) --- Makefile | 4 +- .../v1alpha1/approvalrequests_types.go | 130 ++++ apis/kubefleet.dev/rollout/v1alpha1/doc.go | 20 + .../rollout/v1alpha1/gvk_info.go | 35 + .../rollout/v1alpha1/interface.go | 170 +++++ .../rollout/v1alpha1/stagedupdaterun_types.go | 334 +++++++++ .../v1alpha1/stagedupdaterunstrategy_types.go | 190 +++++ .../rollout/v1alpha1/zz_generated.deepcopy.go | 671 ++++++++++++++++++ ...ollout.kubefleet.dev_approvalrequests.yaml | 137 ++++ ...kubefleet.dev_clusterapprovalrequests.yaml | 137 ++++ ...kubefleet.dev_clusterstagedupdateruns.yaml | 549 ++++++++++++++ ...eet.dev_clusterstagedupdatestrategies.yaml | 223 ++++++ ...ollout.kubefleet.dev_stagedupdateruns.yaml | 549 ++++++++++++++ ....kubefleet.dev_stagedupdatestrategies.yaml | 222 ++++++ 14 files changed, 3369 insertions(+), 2 deletions(-) create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/doc.go create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/interface.go create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go create mode 100644 apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go create mode 100644 config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml create mode 100644 config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml create mode 100644 config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml create mode 100644 config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml create mode 100644 config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml create mode 100644 config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml diff --git a/Makefile b/Makefile index 8065cba00..0d5ee9144 100644 --- a/Makefile +++ b/Makefile @@ -323,9 +323,9 @@ crd-package: ## Package the raw CRDs into a release tarball with a SHA-256 check @echo "Packaged CRDs into $(CRD_PACKAGE_DIR)/$(CRD_PACKAGE_NAME).tgz" .PHONY: crd-verify -crd-verify: ## Verify the chart CRD directories cover every CRD in config/crd/bases; note (chenyu1): kubefleet.dev CRDs are ignored for now until the implementation is completed. +crd-verify: ## Verify the chart CRD directories cover every CRD in config/crd/bases; note (chenyu1): placement.kubefleet.dev and rollout.kubefleet.dev CRDs are ignored for now until the implementation is completed. @bases="$$(mktemp)"; charts="$$(mktemp)"; \ - ls config/crd/bases/ | grep -v '^placement\.kubefleet\.dev' | sort > "$$bases"; \ + ls config/crd/bases/ | grep -Ev '^(placement|rollout)\.kubefleet\.dev' | sort > "$$bases"; \ { ls charts/hub-agent/templates/crds/; ls charts/member-agent/templates/crds/; } | sort > "$$charts"; \ missing="$$(comm -3 "$$bases" "$$charts")"; \ rm -f "$$bases" "$$charts"; \ diff --git a/apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go b/apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go new file mode 100644 index 000000000..5c06ab4a3 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go @@ -0,0 +1,130 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +const ( + ApprovalRequestCondTypeApproved = "Approved" +) + +// ApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters +// for a placement policy, when staged update is in use. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet,kubefleet-rollout} +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ApprovalRequest struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ApprovalRequest API object. + // + // +kubebuilder:validation:Required + Spec ApprovalRequestSpec `json:"spec"` + + // The observed status of the ApprovalRequest API object. + // + // +kubebuilder:validation:Optional + Status ApprovalRequestStatus `json:"status,omitempty"` +} + +// ClusterApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters +// for a cluster placement policy, when staged update is in use. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster,categories={kubefleet,kubefleet-rollout} +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ClusterApprovalRequest struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ClusterApprovalRequest API object. + // + // +kubebuilder:validation:Required + Spec ApprovalRequestSpec `json:"spec"` + + // The observed status of the ClusterApprovalRequest API object. + // + // +kubebuilder:validation:Optional + Status ApprovalRequestStatus `json:"status,omitempty"` +} + +type ApprovalRequestSpec struct { + // The name of the StagedUpdateRun API object that the approval request is associated with. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="stagedUpdateRunName is immutable" + StagedUpdateRunName string `json:"stagedUpdateRunName"` + + // The name of the stage that the approval request is associated with in the staged update run. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern="^[a-z0-9]+$" + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="stageName is immutable" + StageName string `json:"stageName"` +} + +type ApprovalRequestStatus struct { + // A list of observed conditions about the approval request. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +// The list objects for ApprovalRequest and ClusterApprovalRequest APIs. + +// ApprovalRequestList contains a list of ApprovalRequest API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Namespaced" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ApprovalRequestList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ApprovalRequest `json:"items"` +} + +// ClusterApprovalRequestList contains a list of ClusterApprovalRequest API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Cluster" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ClusterApprovalRequestList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ClusterApprovalRequest `json:"items"` +} + +// Set up the API types with the scheme builder. +func init() { + SchemeBuilder.Register(&ApprovalRequest{}, &ApprovalRequestList{}) + SchemeBuilder.Register(&ClusterApprovalRequest{}, &ClusterApprovalRequestList{}) +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/doc.go b/apis/kubefleet.dev/rollout/v1alpha1/doc.go new file mode 100644 index 000000000..db3677d0f --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/doc.go @@ -0,0 +1,20 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// +kubebuilder:object:generate=true +// +k8s:deepcopy-gen=package,register +// +groupName=rollout.kubefleet.dev +package v1alpha1 diff --git a/apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go b/apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go new file mode 100644 index 000000000..e329bffb2 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go @@ -0,0 +1,35 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// +kubebuilder:object:generate=true +// +groupName=rollout.kubefleet.dev +package v1alpha1 + +import ( + "k8s.io/apimachinery/pkg/runtime/schema" + "sigs.k8s.io/controller-runtime/pkg/scheme" +) + +var ( + // GroupVersion is group version used to register these objects + GroupVersion = schema.GroupVersion{Group: "rollout.kubefleet.dev", Version: "v1alpha1"} + + // SchemeBuilder is used to add go types to the GroupVersionKind scheme + SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion} + + // AddToScheme adds the types in this group-version to the given scheme. + AddToScheme = SchemeBuilder.AddToScheme +) diff --git a/apis/kubefleet.dev/rollout/v1alpha1/interface.go b/apis/kubefleet.dev/rollout/v1alpha1/interface.go new file mode 100644 index 000000000..3c3bf1d8a --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/interface.go @@ -0,0 +1,170 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "sigs.k8s.io/controller-runtime/pkg/client" +) + +var _ ApprovalRequestAccessor = &ApprovalRequest{} +var _ ApprovalRequestAccessor = &ClusterApprovalRequest{} + +// ApprovalRequestAccessor is an interface that provides unified access to the Spec and Status of +// ApprovalRequest and ClusterApprovalRequest resources. +// +// +kubebuilder:object:generate=false +type ApprovalRequestAccessor interface { + client.Object + + GetSpec() *ApprovalRequestSpec + GetStatus() *ApprovalRequestStatus + + SetSpec(ApprovalRequestSpec) + SetStatus(ApprovalRequestStatus) +} + +// GetSpec returns the spec of the ApprovalRequest. +func (s *ApprovalRequest) GetSpec() *ApprovalRequestSpec { + return &s.Spec +} + +// GetStatus returns the status of the ApprovalRequest. +func (s *ApprovalRequest) GetStatus() *ApprovalRequestStatus { + return &s.Status +} + +// SetSpec sets the spec of the ApprovalRequest. +func (s *ApprovalRequest) SetSpec(spec ApprovalRequestSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the ApprovalRequest. +func (s *ApprovalRequest) SetStatus(status ApprovalRequestStatus) { + s.Status = status +} + +// GetSpec returns the spec of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) GetSpec() *ApprovalRequestSpec { + return &s.Spec +} + +// GetStatus returns the status of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) GetStatus() *ApprovalRequestStatus { + return &s.Status +} + +// SetSpec sets the spec of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) SetSpec(spec ApprovalRequestSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) SetStatus(status ApprovalRequestStatus) { + s.Status = status +} + +var _ StagedUpdateRunAccessor = &StagedUpdateRun{} +var _ StagedUpdateRunAccessor = &ClusterStagedUpdateRun{} + +// StagedUpdateRunAccessor is an interface that provides unified access to the Spec and Status of +// StagedUpdateRun and ClusterStagedUpdateRun resources. +// +// +kubebuilder:object:generate=false +type StagedUpdateRunAccessor interface { + client.Object + + GetSpec() *StagedUpdateRunSpec + GetStatus() *StagedUpdateRunStatus + + SetSpec(StagedUpdateRunSpec) + SetStatus(StagedUpdateRunStatus) +} + +// GetSpec returns the spec of the StagedUpdateRun. +func (s *StagedUpdateRun) GetSpec() *StagedUpdateRunSpec { + return &s.Spec +} + +// GetStatus returns the status of the StagedUpdateRun. +func (s *StagedUpdateRun) GetStatus() *StagedUpdateRunStatus { + return &s.Status +} + +// SetSpec sets the spec of the StagedUpdateRun. +func (s *StagedUpdateRun) SetSpec(spec StagedUpdateRunSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the StagedUpdateRun. +func (s *StagedUpdateRun) SetStatus(status StagedUpdateRunStatus) { + s.Status = status +} + +// GetSpec returns the spec of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) GetSpec() *StagedUpdateRunSpec { + return &s.Spec +} + +// GetStatus returns the status of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) GetStatus() *StagedUpdateRunStatus { + return &s.Status +} + +// SetSpec sets the spec of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) SetSpec(spec StagedUpdateRunSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) SetStatus(status StagedUpdateRunStatus) { + s.Status = status +} + +var _ StagedUpdateStrategyAccessor = &StagedUpdateStrategy{} +var _ StagedUpdateStrategyAccessor = &ClusterStagedUpdateStrategy{} + +// StagedUpdateStrategyAccessor is an interface that provides unified access to the Spec of +// StagedUpdateStrategy and ClusterStagedUpdateStrategy resources. +// +// +kubebuilder:object:generate=false +type StagedUpdateStrategyAccessor interface { + client.Object + + GetSpec() *StagedUpdateStrategySpec + + SetSpec(StagedUpdateStrategySpec) +} + +// GetSpec returns the spec of the StagedUpdateStrategy. +func (s *StagedUpdateStrategy) GetSpec() *StagedUpdateStrategySpec { + return &s.Spec +} + +// SetSpec sets the spec of the StagedUpdateStrategy. +func (s *StagedUpdateStrategy) SetSpec(spec StagedUpdateStrategySpec) { + s.Spec = spec +} + +// GetSpec returns the spec of the ClusterStagedUpdateStrategy. +func (s *ClusterStagedUpdateStrategy) GetSpec() *StagedUpdateStrategySpec { + return &s.Spec +} + +// SetSpec sets the spec of the ClusterStagedUpdateStrategy. +func (s *ClusterStagedUpdateStrategy) SetSpec(spec StagedUpdateStrategySpec) { + s.Spec = spec +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go new file mode 100644 index 000000000..1da920769 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go @@ -0,0 +1,334 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +const ( + // The condition types of StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunCondTypeInitialized = "Initialized" + StagedUpdateRunCondTypeStarted = "Started" + StagedUpdateRunCondTypeCompleted = "Completed" + + // The condition type of stages to run in StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunPerStageCondTypeStarted = "Started" + StagedUpdateRunPerStageCondTypeCompleted = "Completed" + + // The condition type of clusters to roll out to in StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunPerClusterCondTypeStarted = "Started" + StagedUpdateRunPerClusterCondTypeCompleted = "Completed" + + // The condition type of stage tasks to execute in StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunTaskCondTypeApprovalRequestCreated = "ApprovalRequestCreated" + StagedUpdateRunTaskCondTypeApprovalRequestApproved = "ApprovalRequestApproved" + StagedUpdateRunTaskCondTypeTimedWaitStarted = "TimedWaitStarted" + StagedUpdateRunTaskCondTypeWaitTimeElapsed = "WaitTimeElapsed" +) + +// The reasons for the respective condition types. +const ( + StagedUpdateRunInitializedCondReasonPreppedResourceSnapshotAndAllStages = "PreppedResourceSnapshotAndAllStages" + + StagedUpdateRunTaskApprovalRequestCreatedCondReasonCreated = "RequestCreated" + StagedUpdateRunTaskApprovalRequestApprovedCondReasonApproved = "RequestApproved" + + StagedUpdateRunTaskTimedWaitStartedCondReasonTimerStarted = "TimerStarted" + StagedUpdateRunTaskWaitTimeElapsedCondReasonTimerElapsed = "TimerElapsed" +) + +// StagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a placement policy +// in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage +// sequentially. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet,kubefleet-rollout} +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type StagedUpdateRun struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the StagedUpdateRun API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateRunSpec `json:"spec"` + + // The observed status of the StagedUpdateRun API object. + // + // +kubebuilder:validation:Optional + Status StagedUpdateRunStatus `json:"status,omitempty"` +} + +// ClusterStagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a cluster +// placement policy in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage +// sequentially. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster,categories={kubefleet,kubefleet-rollout} +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ClusterStagedUpdateRun struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ClusterStagedUpdateRun API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateRunSpec `json:"spec"` + + // The observed status of the ClusterStagedUpdateRun API object. + // + // +kubebuilder:validation:Optional + Status StagedUpdateRunStatus `json:"status,omitempty"` +} + +// StagedUpdateRunSpec is the spec of the StagedUpdateRun API object. +// +// +kubebuilder:validation:XValidation:rule="has(self.resourceSnapshotName) == has(oldSelf.resourceSnapshotName)",message="resourceSnapshotName cannot be added or removed after creation" +// +kubebuilder:validation:XValidation:rule="has(self.failurePolicy) == has(oldSelf.failurePolicy)",message="failurePolicy cannot be added or removed after creation" +type StagedUpdateRunSpec struct { + // The name of the placement policy that the StagedUpdateRun API object is associated with, i.e., + // the name of the placement policy where resource changes are rolled out. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="placementPolicyName is immutable" + PlacementPolicyName string `json:"placementPolicyName"` + + // The name of the resource snapshot that the StagedUpdateRun API object is associated with, i.e., + // the name of the resource snapshot where resource changes are introduced. + // + // If left empty, KubeFleet will request a new resource snapshot to be created based on the current state + // of selected resources in the placement policy (if applicable). + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="resourceSnapshotName is immutable" + // +kubebuilder:validation:MaxLength=253 + ResourceSnapshotName string `json:"resourceSnapshotName,omitempty"` + + // The name of the staged update strategy that the StagedUpdateRun API object is associated with, i.e., + // the name of the staged update strategy that organizes clusters into stages and defines how the rollout is executed. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="stagedUpdateStrategyName is immutable" + StagedUpdateStrategyName string `json:"stagedUpdateStrategyName"` + + // Whether the staged update run is suspended. If set to true, KubeFleet will pause the rollout. + // + // If the staged update run is created with this field set to true, KubeFleet will initialize the staged update run, + // i.e., decide on the resource snapshot to roll out and determine the stages and the clusters in each stage, + // but will not start rolling out resource changes until the field is set back to false. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=false + Suspended bool `json:"suspended,omitempty"` + + // The failure policy of the staged update run. It helps KubeFleet determine when to stop the staged update run + // when there are too many failures, so as to keep the impact radius under control. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="failurePolicy is immutable" + FailurePolicy *StagedUpdateRunFailurePolicy `json:"failurePolicy,omitempty"` +} + +type StagedUpdateRunFailurePolicy struct { + // The maximum number of clusters that can fail the rollout before KubeFleet stops the staged + // update run. The count is tracked across all stages and clusters. A cluster is considered to have + // failed the rollout if the new (updated) placement does not become available within the maximum + // wait time. + // + // The default value is 1. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=1 + // +kubebuilder:validation:Minimum=1 + MaxAllowedClusterFailures int32 `json:"maxAllowedClusterFailures,omitempty"` + + // The maximum time to wait for a rollout to a cluster to complete before KubeFleet considers the rollout to have + // failed for the cluster. + // + // The default value is 5 minutes. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=5 + // +kubebuilder:validation:Minimum=1 + MaxWaitTimePerClusterMinutes int32 `json:"maxWaitTimePerClusterMinutes,omitempty"` +} + +type StagedUpdateRunStatus struct { + // A list of observed conditions of the staged update run. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` + + // The name of the resource snapshot that is being rolled out in the staged update run. + // + // If the staged update run is created with the resourceSnapshotName field in the spec set to a specific value, + // this field will be set to the same value; otherwise, KubeFleet will request a new resource snapshot to be created (if applicable) + // and populate this field with the name of the newly created resource snapshot. + // + // +kubebuilder:validation:Optional + ResourceSnapshotNameToRollout string `json:"resourceSnapshotNameToRollout,omitempty"` + + // The status of each stage in the staged update run, including the clusters in each stage and their rollout status. + // + // +kubebuilder:validation:Optional + Stages []PerStageStatus `json:"stages,omitempty"` +} + +type PerStageStatus struct { + // The name of the stage. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern="^[a-z0-9]+$" + StageName string `json:"stageName"` + + // The list of clusters featured in the stage, and their individual rollout status. + // + // +kubebuilder:validation:Optional + Clusters []PerClusterStatus `json:"clusters,omitempty"` + + // The list of tasks to be executed before starting the rollout in the stage, and their execution status. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=1 + BeforeStageTasks []PerStageTaskStatus `json:"beforeStageTasks,omitempty"` + + // The list of tasks to be executed after completing the rollout in the stage, and their execution status. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=2 + AfterStageTasks []PerStageTaskStatus `json:"afterStageTasks,omitempty"` + + // A list of observed conditions about the rollout progress in the stage. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` + + // The timestamp when the rollout in the stage started. If unset, the rollout has not started in the stage yet. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:Format=date-time + StartedTimestamp *metav1.Time `json:"startedTimestamp,omitempty"` + + // The timestamp when the rollout in the stage completed. If unset, the rollout has not completed in the stage yet. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:Format=date-time + CompletedTimestamp *metav1.Time `json:"completedTimestamp,omitempty"` + + // The (resolved) maximum number of clusters that can be updated concurrently within this stage. + // + // +kubebuilder:validation:Optional + MaxConcurrency *int32 `json:"maxConcurrency,omitempty"` +} + +type PerClusterStatus struct { + // The name of the cluster. + // + // +kubebuilder:validation:Required + ClusterName string `json:"clusterName"` + + // The name of the placement binding. + // + // +kubebuilder:validation:Required + PlacementBindingName string `json:"placementBindingName"` + + // A list of observed conditions about the rollout progress of the cluster. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +type PerStageTaskStatus struct { + // The type of the stage task. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:Enum=TimedWait;Approval + Type StageTaskType `json:"type"` + + // The name of the approval request that is created for the stage task. + // + // This field is only set when the task type is Approval. + // + // +kubebuilder:validation:Optional + ApprovalRequestName string `json:"approvalRequestName,omitempty"` + + // The time to wait in a TimedWait task. + // + // +kubebuilder:validation:Optional + WaitTime *metav1.Duration `json:"waitTime,omitempty"` + + // A list of observed conditions about the progress of the stage task. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +// The list objects for StagedUpdateRun and ClusterStagedUpdateRun APIs. + +// StagedUpdateRunList contains a list of StagedUpdateRun API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Namespaced" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type StagedUpdateRunList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []StagedUpdateRun `json:"items"` +} + +// ClusterStagedUpdateRunList contains a list of ClusterStagedUpdateRun API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Cluster" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ClusterStagedUpdateRunList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ClusterStagedUpdateRun `json:"items"` +} + +// Set up the API types with the scheme builder. +func init() { + SchemeBuilder.Register(&StagedUpdateRun{}, &StagedUpdateRunList{}) + SchemeBuilder.Register(&ClusterStagedUpdateRun{}, &ClusterStagedUpdateRunList{}) +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go new file mode 100644 index 000000000..ed6c75def --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go @@ -0,0 +1,190 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" +) + +type StageTaskType string + +const ( + StageTaskTypeTimedWait StageTaskType = "TimedWait" + StageTaskTypeApproval StageTaskType = "Approval" +) + +// StagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes +// for a placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed +// within each stage, and which tasks to execute before and after each stage. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet,kubefleet-rollout} +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type StagedUpdateStrategy struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the StagedUpdateStrategy API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateStrategySpec `json:"spec"` +} + +// ClusterStagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes +// for a cluster placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed +// within each stage, and which tasks to execute before and after each stage. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Cluster,categories={kubefleet,kubefleet-rollout} +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ClusterStagedUpdateStrategy struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ClusterStagedUpdateStrategy API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateStrategySpec `json:"spec"` +} + +type StagedUpdateStrategySpec struct { + // The stages that KubeFleet rolls out resource changes to, in the order they are listed. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MinItems=1 + // +kubebuilder:validation:MaxItems=31 + // +kubebuilder:validation:XValidation:rule="self.all(s, self.exists_one(t, t.name == s.name))",message="stage names must be unique" + Stages []Stage `json:"stages"` +} + +type Stage struct { + // The name of the stage. It must be unique within the staged update strategy and may only consist of + // lowercase alphanumeric characters. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern="^[a-z0-9]+$" + Name string `json:"name"` + + // The label selector that selects clusters to be included in the stage. + // + // Note that if a cluster is selected by multiple stages, it will only be included in the first stage that selects it. + // + // If set to nil, the stage includes no clusters at all. If set to an empty selector, the stage includes all applicable clusters. + // + // +kubebuilder:validation:Optional + LabelSelector *metav1.LabelSelector `json:"labelSelector,omitempty"` + + // The label key used to sort clusters in the stage. KubeFleet rolls out resource changes to clusters in the stage + // based on the sorted order. + // + // Specifically, KubeFleet will read the value of the label key for each cluster in the stage, and sort clusters in ascending + // order based on the label value, interpreted as an integer. If the label is missing on any cluster, an error will be raised. + // + // If this field is not set, KubeFleet will sort clusters in the stage based on the ascending lexical order of their names. + // + // +kubebuilder:validation:Optional + SortingLabelKey *string `json:"sortingLabelKey,omitempty"` + + // The maximum number of clusters that can be rolled out concurrently within this stage. + // + // This field accepts either an integer value (e.g., 5), or a percentage value (e.g., "50%"). For percentage values, + // the concurrency number is calculated based on the total number of clusters in the stage, with fractional results rounded down. + // + // A minimum concurrency of 1 is enforced. + // + // Defaults to 1. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=1 + // +kubebuilder:validation:XIntOrString + // +kubebuilder:validation:Pattern="^(100|[1-9][0-9]?)%$" + // +kubebuilder:validation:XValidation:rule="self == null || type(self) != int || self >= 1",message="maxConcurrency must be at least 1" + MaxConcurrency *intstr.IntOrString `json:"maxConcurrency,omitempty"` + + // A list of tasks to execute after the stage rollout is completed. The tasks run in parallel; a staged update will only + // proceed to the next stage after all tasks are completed. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=2 + // +kubebuilder:validation:XValidation:rule="self.filter(e, e.type == 'Approval').size() <= 1",message="afterStageTasks cannot have more than one Approval task" + // +kubebuilder:validation:XValidation:rule="self.filter(e, e.type == 'TimedWait').size() <= 1",message="afterStageTasks cannot have more than one TimedWait task" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="waitTime does not apply to an Approval task in afterStageTasks" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait' && !has(e.waitTime))",message="waitTime is required for a TimedWait task in afterStageTasks" + AfterStageTasks []StageTask `json:"afterStageTasks,omitempty"` + + // A list of tasks to execute before the stage rollout starts. The tasks run in parallel; the current stage will only start + // after all tasks are completed. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=1 + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="waitTime does not apply to an Approval task in beforeStageTasks" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait')",message="beforeStageTasks cannot include a TimedWait task" + BeforeStageTasks []StageTask `json:"beforeStageTasks,omitempty"` +} + +type StageTask struct { + // The type of the stage task. Currently, the supported task types are TimedWait and Approval. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:Enum=TimedWait;Approval + Type StageTaskType `json:"type"` + + // The time to wait in a TimedWait task. + // + // Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:Pattern="^(?:(?:0|[1-9][0-9]*)(\\.[0-9]+)?(?:s|m|h))+$" + WaitTime *metav1.Duration `json:"waitTime,omitempty"` +} + +// The list objects for StagedUpdateStrategy and ClusterStagedUpdateStrategy APIs. + +// StagedUpdateStrategyList contains a list of StagedUpdateStrategy API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Namespaced" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type StagedUpdateStrategyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []StagedUpdateStrategy `json:"items"` +} + +// ClusterStagedUpdateStrategyList contains a list of ClusterStagedUpdateStrategy API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Cluster" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ClusterStagedUpdateStrategyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ClusterStagedUpdateStrategy `json:"items"` +} + +// Set up the API types with the scheme builder. +func init() { + SchemeBuilder.Register(&StagedUpdateStrategy{}, &StagedUpdateStrategyList{}) + SchemeBuilder.Register(&ClusterStagedUpdateStrategy{}, &ClusterStagedUpdateStrategyList{}) +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go b/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go new file mode 100644 index 000000000..fd435e83e --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go @@ -0,0 +1,671 @@ +//go:build !ignore_autogenerated + +/* +Copyright 2025 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by controller-gen. DO NOT EDIT. + +package v1alpha1 + +import ( + "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/util/intstr" +) + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequest) DeepCopyInto(out *ApprovalRequest) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequest. +func (in *ApprovalRequest) DeepCopy() *ApprovalRequest { + if in == nil { + return nil + } + out := new(ApprovalRequest) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ApprovalRequest) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequestList) DeepCopyInto(out *ApprovalRequestList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ApprovalRequest, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequestList. +func (in *ApprovalRequestList) DeepCopy() *ApprovalRequestList { + if in == nil { + return nil + } + out := new(ApprovalRequestList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ApprovalRequestList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequestSpec) DeepCopyInto(out *ApprovalRequestSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequestSpec. +func (in *ApprovalRequestSpec) DeepCopy() *ApprovalRequestSpec { + if in == nil { + return nil + } + out := new(ApprovalRequestSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequestStatus) DeepCopyInto(out *ApprovalRequestStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequestStatus. +func (in *ApprovalRequestStatus) DeepCopy() *ApprovalRequestStatus { + if in == nil { + return nil + } + out := new(ApprovalRequestStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterApprovalRequest) DeepCopyInto(out *ClusterApprovalRequest) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterApprovalRequest. +func (in *ClusterApprovalRequest) DeepCopy() *ClusterApprovalRequest { + if in == nil { + return nil + } + out := new(ClusterApprovalRequest) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterApprovalRequest) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterApprovalRequestList) DeepCopyInto(out *ClusterApprovalRequestList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ClusterApprovalRequest, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterApprovalRequestList. +func (in *ClusterApprovalRequestList) DeepCopy() *ClusterApprovalRequestList { + if in == nil { + return nil + } + out := new(ClusterApprovalRequestList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterApprovalRequestList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateRun) DeepCopyInto(out *ClusterStagedUpdateRun) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateRun. +func (in *ClusterStagedUpdateRun) DeepCopy() *ClusterStagedUpdateRun { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateRun) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateRun) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateRunList) DeepCopyInto(out *ClusterStagedUpdateRunList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ClusterStagedUpdateRun, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateRunList. +func (in *ClusterStagedUpdateRunList) DeepCopy() *ClusterStagedUpdateRunList { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateRunList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateRunList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateStrategy) DeepCopyInto(out *ClusterStagedUpdateStrategy) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateStrategy. +func (in *ClusterStagedUpdateStrategy) DeepCopy() *ClusterStagedUpdateStrategy { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateStrategy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateStrategy) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateStrategyList) DeepCopyInto(out *ClusterStagedUpdateStrategyList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ClusterStagedUpdateStrategy, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateStrategyList. +func (in *ClusterStagedUpdateStrategyList) DeepCopy() *ClusterStagedUpdateStrategyList { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateStrategyList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateStrategyList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PerClusterStatus) DeepCopyInto(out *PerClusterStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PerClusterStatus. +func (in *PerClusterStatus) DeepCopy() *PerClusterStatus { + if in == nil { + return nil + } + out := new(PerClusterStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PerStageStatus) DeepCopyInto(out *PerStageStatus) { + *out = *in + if in.Clusters != nil { + in, out := &in.Clusters, &out.Clusters + *out = make([]PerClusterStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.BeforeStageTasks != nil { + in, out := &in.BeforeStageTasks, &out.BeforeStageTasks + *out = make([]PerStageTaskStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.AfterStageTasks != nil { + in, out := &in.AfterStageTasks, &out.AfterStageTasks + *out = make([]PerStageTaskStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.StartedTimestamp != nil { + in, out := &in.StartedTimestamp, &out.StartedTimestamp + *out = (*in).DeepCopy() + } + if in.CompletedTimestamp != nil { + in, out := &in.CompletedTimestamp, &out.CompletedTimestamp + *out = (*in).DeepCopy() + } + if in.MaxConcurrency != nil { + in, out := &in.MaxConcurrency, &out.MaxConcurrency + *out = new(int32) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PerStageStatus. +func (in *PerStageStatus) DeepCopy() *PerStageStatus { + if in == nil { + return nil + } + out := new(PerStageStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PerStageTaskStatus) DeepCopyInto(out *PerStageTaskStatus) { + *out = *in + if in.WaitTime != nil { + in, out := &in.WaitTime, &out.WaitTime + *out = new(v1.Duration) + **out = **in + } + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PerStageTaskStatus. +func (in *PerStageTaskStatus) DeepCopy() *PerStageTaskStatus { + if in == nil { + return nil + } + out := new(PerStageTaskStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *Stage) DeepCopyInto(out *Stage) { + *out = *in + if in.LabelSelector != nil { + in, out := &in.LabelSelector, &out.LabelSelector + *out = new(v1.LabelSelector) + (*in).DeepCopyInto(*out) + } + if in.SortingLabelKey != nil { + in, out := &in.SortingLabelKey, &out.SortingLabelKey + *out = new(string) + **out = **in + } + if in.MaxConcurrency != nil { + in, out := &in.MaxConcurrency, &out.MaxConcurrency + *out = new(intstr.IntOrString) + **out = **in + } + if in.AfterStageTasks != nil { + in, out := &in.AfterStageTasks, &out.AfterStageTasks + *out = make([]StageTask, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.BeforeStageTasks != nil { + in, out := &in.BeforeStageTasks, &out.BeforeStageTasks + *out = make([]StageTask, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Stage. +func (in *Stage) DeepCopy() *Stage { + if in == nil { + return nil + } + out := new(Stage) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StageTask) DeepCopyInto(out *StageTask) { + *out = *in + if in.WaitTime != nil { + in, out := &in.WaitTime, &out.WaitTime + *out = new(v1.Duration) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StageTask. +func (in *StageTask) DeepCopy() *StageTask { + if in == nil { + return nil + } + out := new(StageTask) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRun) DeepCopyInto(out *StagedUpdateRun) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRun. +func (in *StagedUpdateRun) DeepCopy() *StagedUpdateRun { + if in == nil { + return nil + } + out := new(StagedUpdateRun) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateRun) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunFailurePolicy) DeepCopyInto(out *StagedUpdateRunFailurePolicy) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunFailurePolicy. +func (in *StagedUpdateRunFailurePolicy) DeepCopy() *StagedUpdateRunFailurePolicy { + if in == nil { + return nil + } + out := new(StagedUpdateRunFailurePolicy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunList) DeepCopyInto(out *StagedUpdateRunList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]StagedUpdateRun, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunList. +func (in *StagedUpdateRunList) DeepCopy() *StagedUpdateRunList { + if in == nil { + return nil + } + out := new(StagedUpdateRunList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateRunList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunSpec) DeepCopyInto(out *StagedUpdateRunSpec) { + *out = *in + if in.FailurePolicy != nil { + in, out := &in.FailurePolicy, &out.FailurePolicy + *out = new(StagedUpdateRunFailurePolicy) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunSpec. +func (in *StagedUpdateRunSpec) DeepCopy() *StagedUpdateRunSpec { + if in == nil { + return nil + } + out := new(StagedUpdateRunSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunStatus) DeepCopyInto(out *StagedUpdateRunStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.Stages != nil { + in, out := &in.Stages, &out.Stages + *out = make([]PerStageStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunStatus. +func (in *StagedUpdateRunStatus) DeepCopy() *StagedUpdateRunStatus { + if in == nil { + return nil + } + out := new(StagedUpdateRunStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateStrategy) DeepCopyInto(out *StagedUpdateStrategy) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateStrategy. +func (in *StagedUpdateStrategy) DeepCopy() *StagedUpdateStrategy { + if in == nil { + return nil + } + out := new(StagedUpdateStrategy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateStrategy) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateStrategyList) DeepCopyInto(out *StagedUpdateStrategyList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]StagedUpdateStrategy, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateStrategyList. +func (in *StagedUpdateStrategyList) DeepCopy() *StagedUpdateStrategyList { + if in == nil { + return nil + } + out := new(StagedUpdateStrategyList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateStrategyList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateStrategySpec) DeepCopyInto(out *StagedUpdateStrategySpec) { + *out = *in + if in.Stages != nil { + in, out := &in.Stages, &out.Stages + *out = make([]Stage, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateStrategySpec. +func (in *StagedUpdateStrategySpec) DeepCopy() *StagedUpdateStrategySpec { + if in == nil { + return nil + } + out := new(StagedUpdateStrategySpec) + in.DeepCopyInto(out) + return out +} diff --git a/config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml b/config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml new file mode 100644 index 000000000..b7d05ebfe --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml @@ -0,0 +1,137 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: approvalrequests.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ApprovalRequest + listKind: ApprovalRequestList + plural: approvalrequests + singular: approvalrequest + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters + for a placement policy, when staged update is in use. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ApprovalRequest API object. + properties: + stageName: + description: The name of the stage that the approval request is associated + with in the staged update run. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + x-kubernetes-validations: + - message: stageName is immutable + rule: self == oldSelf + stagedUpdateRunName: + description: The name of the StagedUpdateRun API object that the approval + request is associated with. + type: string + x-kubernetes-validations: + - message: stagedUpdateRunName is immutable + rule: self == oldSelf + required: + - stageName + - stagedUpdateRunName + type: object + status: + description: The observed status of the ApprovalRequest API object. + properties: + conditions: + description: A list of observed conditions about the approval request. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml b/config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml new file mode 100644 index 000000000..343fae26a --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml @@ -0,0 +1,137 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: clusterapprovalrequests.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ClusterApprovalRequest + listKind: ClusterApprovalRequestList + plural: clusterapprovalrequests + singular: clusterapprovalrequest + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ClusterApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters + for a cluster placement policy, when staged update is in use. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ClusterApprovalRequest API object. + properties: + stageName: + description: The name of the stage that the approval request is associated + with in the staged update run. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + x-kubernetes-validations: + - message: stageName is immutable + rule: self == oldSelf + stagedUpdateRunName: + description: The name of the StagedUpdateRun API object that the approval + request is associated with. + type: string + x-kubernetes-validations: + - message: stagedUpdateRunName is immutable + rule: self == oldSelf + required: + - stageName + - stagedUpdateRunName + type: object + status: + description: The observed status of the ClusterApprovalRequest API object. + properties: + conditions: + description: A list of observed conditions about the approval request. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml new file mode 100644 index 000000000..825a18cf8 --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml @@ -0,0 +1,549 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: clusterstagedupdateruns.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ClusterStagedUpdateRun + listKind: ClusterStagedUpdateRunList + plural: clusterstagedupdateruns + singular: clusterstagedupdaterun + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ClusterStagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a cluster + placement policy in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage + sequentially. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ClusterStagedUpdateRun API object. + properties: + failurePolicy: + description: |- + The failure policy of the staged update run. It helps KubeFleet determine when to stop the staged update run + when there are too many failures, so as to keep the impact radius under control. + properties: + maxAllowedClusterFailures: + default: 1 + description: |- + The maximum number of clusters that can fail the rollout before KubeFleet stops the staged + update run. The count is tracked across all stages and clusters. A cluster is considered to have + failed the rollout if the new (updated) placement does not become available within the maximum + wait time. + + The default value is 1. + format: int32 + minimum: 1 + type: integer + maxWaitTimePerClusterMinutes: + default: 5 + description: |- + The maximum time to wait for a rollout to a cluster to complete before KubeFleet considers the rollout to have + failed for the cluster. + + The default value is 5 minutes. + format: int32 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: failurePolicy is immutable + rule: self == oldSelf + placementPolicyName: + description: |- + The name of the placement policy that the StagedUpdateRun API object is associated with, i.e., + the name of the placement policy where resource changes are rolled out. + type: string + x-kubernetes-validations: + - message: placementPolicyName is immutable + rule: self == oldSelf + resourceSnapshotName: + description: |- + The name of the resource snapshot that the StagedUpdateRun API object is associated with, i.e., + the name of the resource snapshot where resource changes are introduced. + + If left empty, KubeFleet will request a new resource snapshot to be created based on the current state + of selected resources in the placement policy (if applicable). + maxLength: 253 + type: string + x-kubernetes-validations: + - message: resourceSnapshotName is immutable + rule: self == oldSelf + stagedUpdateStrategyName: + description: |- + The name of the staged update strategy that the StagedUpdateRun API object is associated with, i.e., + the name of the staged update strategy that organizes clusters into stages and defines how the rollout is executed. + type: string + x-kubernetes-validations: + - message: stagedUpdateStrategyName is immutable + rule: self == oldSelf + suspended: + default: false + description: |- + Whether the staged update run is suspended. If set to true, KubeFleet will pause the rollout. + + If the staged update run is created with this field set to true, KubeFleet will initialize the staged update run, + i.e., decide on the resource snapshot to roll out and determine the stages and the clusters in each stage, + but will not start rolling out resource changes until the field is set back to false. + type: boolean + required: + - placementPolicyName + - stagedUpdateStrategyName + type: object + x-kubernetes-validations: + - message: resourceSnapshotName cannot be added or removed after creation + rule: has(self.resourceSnapshotName) == has(oldSelf.resourceSnapshotName) + - message: failurePolicy cannot be added or removed after creation + rule: has(self.failurePolicy) == has(oldSelf.failurePolicy) + status: + description: The observed status of the ClusterStagedUpdateRun API object. + properties: + conditions: + description: A list of observed conditions of the staged update run. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + resourceSnapshotNameToRollout: + description: |- + The name of the resource snapshot that is being rolled out in the staged update run. + + If the staged update run is created with the resourceSnapshotName field in the spec set to a specific value, + this field will be set to the same value; otherwise, KubeFleet will request a new resource snapshot to be created (if applicable) + and populate this field with the name of the newly created resource snapshot. + type: string + stages: + description: The status of each stage in the staged update run, including + the clusters in each stage and their rollout status. + items: + properties: + afterStageTasks: + description: The list of tasks to be executed after completing + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 2 + type: array + beforeStageTasks: + description: The list of tasks to be executed before starting + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 1 + type: array + clusters: + description: The list of clusters featured in the stage, and + their individual rollout status. + items: + properties: + clusterName: + description: The name of the cluster. + type: string + conditions: + description: A list of observed conditions about the rollout + progress of the cluster. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + placementBindingName: + description: The name of the placement binding. + type: string + required: + - clusterName + - placementBindingName + type: object + type: array + completedTimestamp: + description: The timestamp when the rollout in the stage completed. + If unset, the rollout has not completed in the stage yet. + format: date-time + type: string + conditions: + description: A list of observed conditions about the rollout + progress in the stage. + items: + description: Condition contains details for one aspect of + the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + maxConcurrency: + description: The (resolved) maximum number of clusters that + can be updated concurrently within this stage. + format: int32 + type: integer + stageName: + description: The name of the stage. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + startedTimestamp: + description: The timestamp when the rollout in the stage started. + If unset, the rollout has not started in the stage yet. + format: date-time + type: string + required: + - stageName + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml new file mode 100644 index 000000000..62d9b1b44 --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml @@ -0,0 +1,223 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: clusterstagedupdatestrategies.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ClusterStagedUpdateStrategy + listKind: ClusterStagedUpdateStrategyList + plural: clusterstagedupdatestrategies + singular: clusterstagedupdatestrategy + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ClusterStagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes + for a cluster placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed + within each stage, and which tasks to execute before and after each stage. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ClusterStagedUpdateStrategy API + object. + properties: + stages: + description: The stages that KubeFleet rolls out resource changes + to, in the order they are listed. + items: + properties: + afterStageTasks: + description: |- + A list of tasks to execute after the stage rollout is completed. The tasks run in parallel; a staged update will only + proceed to the next stage after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 2 + type: array + x-kubernetes-validations: + - message: afterStageTasks cannot have more than one Approval + task + rule: self.filter(e, e.type == 'Approval').size() <= 1 + - message: afterStageTasks cannot have more than one TimedWait + task + rule: self.filter(e, e.type == 'TimedWait').size() <= 1 + - message: waitTime does not apply to an Approval task in afterStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: waitTime is required for a TimedWait task in afterStageTasks + rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' + beforeStageTasks: + description: |- + A list of tasks to execute before the stage rollout starts. The tasks run in parallel; the current stage will only start + after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: waitTime does not apply to an Approval task in beforeStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: beforeStageTasks cannot include a TimedWait task + rule: '!self.exists(e, e.type == ''TimedWait'')' + labelSelector: + description: |- + The label selector that selects clusters to be included in the stage. + + Note that if a cluster is selected by multiple stages, it will only be included in the first stage that selects it. + + If set to nil, the stage includes no clusters at all. If set to an empty selector, the stage includes all applicable clusters. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + maxConcurrency: + anyOf: + - type: integer + - type: string + default: 1 + description: |- + The maximum number of clusters that can be rolled out concurrently within this stage. + + This field accepts either an integer value (e.g., 5), or a percentage value (e.g., "50%"). For percentage values, + the concurrency number is calculated based on the total number of clusters in the stage, with fractional results rounded down. + + A minimum concurrency of 1 is enforced. + + Defaults to 1. + pattern: ^(100|[1-9][0-9]?)%$ + x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: maxConcurrency must be at least 1 + rule: self == null || type(self) != int || self >= 1 + name: + description: |- + The name of the stage. It must be unique within the staged update strategy and may only consist of + lowercase alphanumeric characters. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + sortingLabelKey: + description: |- + The label key used to sort clusters in the stage. KubeFleet rolls out resource changes to clusters in the stage + based on the sorted order. + + Specifically, KubeFleet will read the value of the label key for each cluster in the stage, and sort clusters in ascending + order based on the label value, interpreted as an integer. If the label is missing on any cluster, an error will be raised. + + If this field is not set, KubeFleet will sort clusters in the stage based on the ascending lexical order of their names. + type: string + required: + - name + type: object + maxItems: 31 + minItems: 1 + type: array + x-kubernetes-validations: + - message: stage names must be unique + rule: self.all(s, self.exists_one(t, t.name == s.name)) + required: + - stages + type: object + required: + - spec + type: object + served: true + storage: true diff --git a/config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml b/config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml new file mode 100644 index 000000000..03658496b --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml @@ -0,0 +1,549 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: stagedupdateruns.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: StagedUpdateRun + listKind: StagedUpdateRunList + plural: stagedupdateruns + singular: stagedupdaterun + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + StagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a placement policy + in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage + sequentially. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the StagedUpdateRun API object. + properties: + failurePolicy: + description: |- + The failure policy of the staged update run. It helps KubeFleet determine when to stop the staged update run + when there are too many failures, so as to keep the impact radius under control. + properties: + maxAllowedClusterFailures: + default: 1 + description: |- + The maximum number of clusters that can fail the rollout before KubeFleet stops the staged + update run. The count is tracked across all stages and clusters. A cluster is considered to have + failed the rollout if the new (updated) placement does not become available within the maximum + wait time. + + The default value is 1. + format: int32 + minimum: 1 + type: integer + maxWaitTimePerClusterMinutes: + default: 5 + description: |- + The maximum time to wait for a rollout to a cluster to complete before KubeFleet considers the rollout to have + failed for the cluster. + + The default value is 5 minutes. + format: int32 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: failurePolicy is immutable + rule: self == oldSelf + placementPolicyName: + description: |- + The name of the placement policy that the StagedUpdateRun API object is associated with, i.e., + the name of the placement policy where resource changes are rolled out. + type: string + x-kubernetes-validations: + - message: placementPolicyName is immutable + rule: self == oldSelf + resourceSnapshotName: + description: |- + The name of the resource snapshot that the StagedUpdateRun API object is associated with, i.e., + the name of the resource snapshot where resource changes are introduced. + + If left empty, KubeFleet will request a new resource snapshot to be created based on the current state + of selected resources in the placement policy (if applicable). + maxLength: 253 + type: string + x-kubernetes-validations: + - message: resourceSnapshotName is immutable + rule: self == oldSelf + stagedUpdateStrategyName: + description: |- + The name of the staged update strategy that the StagedUpdateRun API object is associated with, i.e., + the name of the staged update strategy that organizes clusters into stages and defines how the rollout is executed. + type: string + x-kubernetes-validations: + - message: stagedUpdateStrategyName is immutable + rule: self == oldSelf + suspended: + default: false + description: |- + Whether the staged update run is suspended. If set to true, KubeFleet will pause the rollout. + + If the staged update run is created with this field set to true, KubeFleet will initialize the staged update run, + i.e., decide on the resource snapshot to roll out and determine the stages and the clusters in each stage, + but will not start rolling out resource changes until the field is set back to false. + type: boolean + required: + - placementPolicyName + - stagedUpdateStrategyName + type: object + x-kubernetes-validations: + - message: resourceSnapshotName cannot be added or removed after creation + rule: has(self.resourceSnapshotName) == has(oldSelf.resourceSnapshotName) + - message: failurePolicy cannot be added or removed after creation + rule: has(self.failurePolicy) == has(oldSelf.failurePolicy) + status: + description: The observed status of the StagedUpdateRun API object. + properties: + conditions: + description: A list of observed conditions of the staged update run. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + resourceSnapshotNameToRollout: + description: |- + The name of the resource snapshot that is being rolled out in the staged update run. + + If the staged update run is created with the resourceSnapshotName field in the spec set to a specific value, + this field will be set to the same value; otherwise, KubeFleet will request a new resource snapshot to be created (if applicable) + and populate this field with the name of the newly created resource snapshot. + type: string + stages: + description: The status of each stage in the staged update run, including + the clusters in each stage and their rollout status. + items: + properties: + afterStageTasks: + description: The list of tasks to be executed after completing + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 2 + type: array + beforeStageTasks: + description: The list of tasks to be executed before starting + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 1 + type: array + clusters: + description: The list of clusters featured in the stage, and + their individual rollout status. + items: + properties: + clusterName: + description: The name of the cluster. + type: string + conditions: + description: A list of observed conditions about the rollout + progress of the cluster. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + placementBindingName: + description: The name of the placement binding. + type: string + required: + - clusterName + - placementBindingName + type: object + type: array + completedTimestamp: + description: The timestamp when the rollout in the stage completed. + If unset, the rollout has not completed in the stage yet. + format: date-time + type: string + conditions: + description: A list of observed conditions about the rollout + progress in the stage. + items: + description: Condition contains details for one aspect of + the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + maxConcurrency: + description: The (resolved) maximum number of clusters that + can be updated concurrently within this stage. + format: int32 + type: integer + stageName: + description: The name of the stage. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + startedTimestamp: + description: The timestamp when the rollout in the stage started. + If unset, the rollout has not started in the stage yet. + format: date-time + type: string + required: + - stageName + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml b/config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml new file mode 100644 index 000000000..9bcea3419 --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml @@ -0,0 +1,222 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: stagedupdatestrategies.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: StagedUpdateStrategy + listKind: StagedUpdateStrategyList + plural: stagedupdatestrategies + singular: stagedupdatestrategy + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + StagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes + for a placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed + within each stage, and which tasks to execute before and after each stage. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the StagedUpdateStrategy API object. + properties: + stages: + description: The stages that KubeFleet rolls out resource changes + to, in the order they are listed. + items: + properties: + afterStageTasks: + description: |- + A list of tasks to execute after the stage rollout is completed. The tasks run in parallel; a staged update will only + proceed to the next stage after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 2 + type: array + x-kubernetes-validations: + - message: afterStageTasks cannot have more than one Approval + task + rule: self.filter(e, e.type == 'Approval').size() <= 1 + - message: afterStageTasks cannot have more than one TimedWait + task + rule: self.filter(e, e.type == 'TimedWait').size() <= 1 + - message: waitTime does not apply to an Approval task in afterStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: waitTime is required for a TimedWait task in afterStageTasks + rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' + beforeStageTasks: + description: |- + A list of tasks to execute before the stage rollout starts. The tasks run in parallel; the current stage will only start + after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: waitTime does not apply to an Approval task in beforeStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: beforeStageTasks cannot include a TimedWait task + rule: '!self.exists(e, e.type == ''TimedWait'')' + labelSelector: + description: |- + The label selector that selects clusters to be included in the stage. + + Note that if a cluster is selected by multiple stages, it will only be included in the first stage that selects it. + + If set to nil, the stage includes no clusters at all. If set to an empty selector, the stage includes all applicable clusters. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + maxConcurrency: + anyOf: + - type: integer + - type: string + default: 1 + description: |- + The maximum number of clusters that can be rolled out concurrently within this stage. + + This field accepts either an integer value (e.g., 5), or a percentage value (e.g., "50%"). For percentage values, + the concurrency number is calculated based on the total number of clusters in the stage, with fractional results rounded down. + + A minimum concurrency of 1 is enforced. + + Defaults to 1. + pattern: ^(100|[1-9][0-9]?)%$ + x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: maxConcurrency must be at least 1 + rule: self == null || type(self) != int || self >= 1 + name: + description: |- + The name of the stage. It must be unique within the staged update strategy and may only consist of + lowercase alphanumeric characters. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + sortingLabelKey: + description: |- + The label key used to sort clusters in the stage. KubeFleet rolls out resource changes to clusters in the stage + based on the sorted order. + + Specifically, KubeFleet will read the value of the label key for each cluster in the stage, and sort clusters in ascending + order based on the label value, interpreted as an integer. If the label is missing on any cluster, an error will be raised. + + If this field is not set, KubeFleet will sort clusters in the stage based on the ascending lexical order of their names. + type: string + required: + - name + type: object + maxItems: 31 + minItems: 1 + type: array + x-kubernetes-validations: + - message: stage names must be unique + rule: self.all(s, self.exists_one(t, t.name == s.name)) + required: + - stages + type: object + required: + - spec + type: object + served: true + storage: true From 51a058fbbcf072688512989677699c2beb5f6244 Mon Sep 17 00:00:00 2001 From: Sujay Babu Sekar Date: Thu, 24 Sep 2026 02:07:40 +0200 Subject: [PATCH 03/20] feat: support vendor-neutral hub kubeconfig authentication for member-agent (#913) --- charts/member-agent/README.md | 356 ++++++++++++++++++ charts/member-agent/templates/deployment.yaml | 70 +++- .../templates/serviceaccount.yaml | 4 + charts/member-agent/values.yaml | 35 ++ cmd/memberagent/main.go | 64 +++- cmd/memberagent/main_test.go | 192 +++++++++- cmd/memberagent/options/hub.go | 32 +- cmd/memberagent/options/options_test.go | 12 + cmd/memberagent/options/validation.go | 5 + cmd/memberagent/options/validation_test.go | 9 + cmd/memberagent/testdata/kubeconfig | 17 + cmd/memberagent/testdata/kubeconfig-exec | 27 ++ cmd/memberagent/testdata/kubeconfig-malformed | 2 + 13 files changed, 787 insertions(+), 38 deletions(-) create mode 100644 cmd/memberagent/testdata/kubeconfig create mode 100644 cmd/memberagent/testdata/kubeconfig-exec create mode 100644 cmd/memberagent/testdata/kubeconfig-malformed diff --git a/charts/member-agent/README.md b/charts/member-agent/README.md index b5429039b..b0b801915 100644 --- a/charts/member-agent/README.md +++ b/charts/member-agent/README.md @@ -103,6 +103,11 @@ helm upgrade member-agent kubefleet/member-agent --namespace fleet-system | logVerbosity | Log level. Uses V logs (klog) | `3` | | tlsClientInsecure | Skip TLS server certificate verification when the member agent connects to the hub cluster. Leave this `false` unless you explicitly trust the endpoint and understand the risk. | `false` | | useCAAuth | Use certificate-based authentication for the hub connection instead of the token-based path. | `false` | +| useKubeconfig | Authenticate to the hub cluster via a full kubeconfig instead of the refresh-token sidecar or `useCAAuth`. The vendor-neutral way to authenticate via a federated identity. Mutually exclusive with `useCAAuth`. See [Hub connection via kubeconfig](#hub-connection-via-kubeconfig-federated-identity-any-vendor). | `false` | +| config.hubKubeconfigSecretName | The name of a Secret (key `kubeconfig`) containing a full kubeconfig for the hub connection, mounted and passed to the process via the `KUBE_CONFIG_PATH` environment variable. Required (the chart fails to render otherwise) when `useKubeconfig` is `true`. | `` | +| podLabels | Extra labels applied to the member-agent pod template — e.g. workload-identity labels a `hubKubeconfigSecretName` exec plugin needs. | `{}` | +| serviceAccountAnnotations | Extra annotations applied to the member-agent ServiceAccount — e.g. workload-identity annotations a `hubKubeconfigSecretName` exec plugin needs. | `{}` | +| extraVolumes / extraVolumeMounts / extraInitContainers | Standard passthroughs (rendered via `toYaml`) for getting an exec plugin binary, or a workload-API socket, into the pod without a custom image. `extraVolumeMounts` applies to the member-agent container. | `[]` | | propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` | | region | The region where the member cluster resides | `` | | enableNamespaceCollectionInPropertyProvider | Enable namespace collection in the property provider; when enabled, the member agent will collect and report the list of namespaces present in the member cluster to the hub cluster for use in scheduling decisions | `false` | @@ -123,6 +128,357 @@ By default, the chart keeps TLS server certificate verification enabled for the Set `tlsClientInsecure=true` only for explicitly trusted test environments where certificate verification cannot be configured. +## Hub connection via kubeconfig (federated identity, any vendor) + +As an alternative to the refresh-token sidecar, the member agent can load a full kubeconfig for the +hub connection (`useKubeconfig` + `config.hubKubeconfigSecretName`, i.e. `--use-kubeconfig` plus the +`KUBE_CONFIG_PATH` environment variable), the same way `kubectl` does. +This mode requires no sidecar container and no shared token volume — `client-go` handles token +caching and refresh itself, whatever credential mechanism the kubeconfig's `users[].user.exec` +stanza invokes. + +This chart, and KubeFleet's own code, have **no vendor-specific knowledge** of any cloud's identity +system — the kubeconfig you supply can invoke whatever exec-credential plugin your platform +provides (`client.authentication.k8s.io`, the same mechanism `kubectl` itself uses). Everything +vendor-specific — the plugin binary, its arguments, and any workload-identity metadata — lives +entirely in the kubeconfig you write and the `podLabels`/`serviceAccountAnnotations` values you set; +none of it is baked into this chart. + +**Kubeconfig mode is fully authoritative for the hub connection.** Once `useKubeconfig` is set, the +loaded kubeconfig's own server URL, TLS configuration, and authentication are used as-is; none of +the following are consulted, even if also set: `config.hubURL`/`HUB_SERVER_URL`, `useCAAuth`, +`tlsClientInsecure`/`--tls-insecure`, `config.CABundle`/`CA_BUNDLE`, `config.hubCA`/ +`HUB_CERTIFICATE_AUTHORITY`, and the refresh-token sidecar's `CONFIG_PATH`/`IDENTITY_KEY`/ +`IDENTITY_CERT`. If you need TLS verification disabled for testing, set +`insecure-skip-tls-verify: true` on the kubeconfig's own `cluster` entry rather than +`tlsClientInsecure` — the latter has no effect in this mode. + +**Security considerations**: the hub kubeconfig can carry — directly or via its exec plugin's +own credentials — everything needed to authenticate to the hub cluster, so treat the Secret +holding it like any other high-value credential (restrict who can `get`/`list` it via RBAC, as +you would `hub-kubeconfig-secret` or `hub-identity` for the other modes). Prefer mounting the +kubeconfig and any exec-plugin binary via Secrets/`extraVolumes` rather than embedding secret +material in Helm values or `--set` flags — values and `--set` arguments can end up in shell +history, `helm get values` output, or process listings, none of which apply to a Secret mounted +as a file. + +Three worked examples: + +### Azure / AKS (kubelogin) + +`kubelogin` covers three Azure identity mechanisms — federated (workload) identity, managed +identity, and a service principal's client secret or certificate — selected by the `--login` mode +in its exec stanza. All three use the same `--use-kubeconfig`/`KUBE_CONFIG_PATH` mechanism and +differ only in the kubeconfig content and prerequisites shown below. + +However you build the kubeconfig for any of the three, package it into a Secret the same way +(`kubectl create secret generic hub-kubeconfig --from-file=kubeconfig=./kubeconfig.yaml`), shown in +each example below. + +`kubelogin` also needs to be present in the member-agent image for all three — either build a small +custom image on top of the official one: +```dockerfile +FROM ghcr.io/kubefleet-dev/kubefleet/member-agent: +COPY --from=ghcr.io/azure/kubelogin: /usr/local/bin/kubelogin /usr/local/bin/kubelogin +``` +or use `extraInitContainers`/`extraVolumes`/`extraVolumeMounts` to place it on a shared volume at +runtime instead — the official `ghcr.io/azure/kubelogin` image is `scratch`-based (no shell), so the +init container has to be a different, shell-capable image that fetches the release binary itself: + +```yaml +# values-kubelogin-init.yaml +extraVolumes: +- name: kubelogin-bin + emptyDir: {} +extraVolumeMounts: +- name: kubelogin-bin + mountPath: /usr/local/bin +extraInitContainers: +- name: install-kubelogin + image: busybox:1.36 + # Match the release asset to the node architecture (kubelogin-linux-arm64.zip on arm64 nodes). + command: + - sh + - -c + - | + wget -qO /tmp/kubelogin.zip https://github.com/Azure/kubelogin/releases/download/v0.2.19/kubelogin-linux-amd64.zip + unzip -oq /tmp/kubelogin.zip -d /tmp + cp /tmp/bin/linux_amd64/kubelogin /kubelogin-bin/kubelogin + chmod +x /kubelogin-bin/kubelogin + volumeMounts: + - name: kubelogin-bin + mountPath: /kubelogin-bin +``` +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set-string podLabels."azure\.workload\.identity/use"=true \ + --set serviceAccountAnnotations."azure\.workload\.identity/client-id"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +`kubelogin-bin` is mounted at `/usr/local/bin` on the member-agent container too, so it works with +the bare `command: kubelogin` shown in every example below — no image build required, at the cost +of a network fetch on every pod start. + +#### Federated identity (`--login workloadidentity`) + +Set up [AKS workload identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) on the member cluster, and grant the federated managed identity the appropriate RBAC permissions on the hub cluster (an AKS cluster with Azure AD authentication enabled). Then: + +```yaml +# kubeconfig.yaml +apiVersion: v1 +kind: Config +clusters: +- cluster: + certificate-authority-data: + server: https:// + name: hub +contexts: +- context: + cluster: hub + user: fed + name: hub +current-context: hub +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --environment + - AzurePublicCloud + - --server-id + - + - --client-id + - + - --tenant-id + - + - --login + - workloadidentity + provideClusterInfo: false + interactiveMode: Never +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set-string podLabels."azure\.workload\.identity/use"=true \ + --set serviceAccountAnnotations."azure\.workload\.identity/client-id"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +#### Managed identity (`--login msi`) + +An alternative to federated identity: uses the IMDS-based managed identity available on any Azure +VM/node — so it needs no AKS workload identity setup, no `podLabels`, and no +`serviceAccountAnnotations`. Grant the managed identity the appropriate RBAC permissions on the hub +cluster, then: + +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - + - --login + - msi + - --client-id # omit this flag entirely to use the system-assigned identity instead + - + provideClusterInfo: false + interactiveMode: Never +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +#### Client secret / client certificate (`--login spn`) + +Authenticates as an Azure AD app registration (service principal) using a client secret or a client +certificate, via `--client-id`/`--client-secret`/`--client-certificate`/`--tenant-id` — the same +flag-based style as the federated-identity and managed-identity variants above. + +**Client secret flow:** +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - + - --login + - spn + - --client-id + - + - --client-secret + - + - --tenant-id + - + provideClusterInfo: false + interactiveMode: Never +``` +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +**Client certificate flow:** `--client-certificate` must point to a file on disk, so the +certificate needs its own mount via `extraVolumes`/`extraVolumeMounts` rather than living inline in +the kubeconfig: +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - + - --login + - spn + - --client-id + - + - --client-certificate + - /etc/kubefleet/spn-cert/tls.crt + - --tenant-id + - + provideClusterInfo: false + interactiveMode: Never +``` +This flow also needs its own Secret for the certificate: +```console +kubectl create secret tls spn-cert --namespace fleet-system --cert=/path/to/spn.crt --key=/path/to/spn.key +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set 'extraVolumes[1].name=spn-cert' \ + --set 'extraVolumes[1].secret.secretName=spn-cert' \ + --set 'extraVolumeMounts[1].name=spn-cert' \ + --set 'extraVolumeMounts[1].mountPath=/etc/kubefleet/spn-cert' \ + --set 'extraVolumeMounts[1].readOnly=true' +``` +`values-kubelogin-init.yaml` already occupies index `0` of `extraVolumes`/`extraVolumeMounts` (the +`kubelogin-bin` `emptyDir`), so the certificate Secret's volume/mount use index `1` here to avoid +overwriting it. +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +### AWS / EKS (IRSA) + +Set up [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) on the member cluster, and grant the IAM role access to the hub EKS cluster (an EKS Access Entry or the `aws-auth` ConfigMap). The kubeconfig's exec stanza runs the small, purpose-built `aws-iam-authenticator` binary (lighter to bundle than the full `aws` CLI): + +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: aws-iam-authenticator + args: + - token + - -i + - + provideClusterInfo: false + interactiveMode: Never +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set serviceAccountAnnotations."eks\.amazonaws\.com/role-arn"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +No `podLabels` needed — IRSA only requires the ServiceAccount annotation. + +### GCP / GKE (Workload Identity) + +Set up [GKE Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) on the member cluster (binds the pod's ServiceAccount to a Google service account), and grant that service account the appropriate IAM role on the hub GKE cluster. The kubeconfig's exec stanza runs `gke-gcloud-auth-plugin`: + +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1beta1 + command: gke-gcloud-auth-plugin + args: + - --use_application_default_credentials + provideClusterInfo: true +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set serviceAccountAnnotations."iam\.gke\.io/gcp-service-account"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +No `podLabels` needed here either — Workload Identity is resolved automatically for any pod running +under the annotated ServiceAccount. + ## Override Azure cloud config **If PropertyProvider feature is set to azure, then a cloud configuration is required.** diff --git a/charts/member-agent/templates/deployment.yaml b/charts/member-agent/templates/deployment.yaml index 30b564c4d..2d0f918fe 100644 --- a/charts/member-agent/templates/deployment.yaml +++ b/charts/member-agent/templates/deployment.yaml @@ -1,3 +1,12 @@ +{{- $useSidecar := and (not .Values.useCAAuth) (not .Values.useKubeconfig) }} +{{- $mountIdentitySecret := and .Values.useCAAuth .Values.config.identitySecretName }} +{{- $mountHubKubeconfig := and (not .Values.useCAAuth) .Values.useKubeconfig }} +{{- if and .Values.useCAAuth .Values.useKubeconfig }} +{{- fail "useCAAuth and useKubeconfig must not both be true: with both set, useCAAuth silently wins (only --use-ca-auth is passed to the member agent) and the requested kubeconfig mode is never applied. Set exactly one." }} +{{- end }} +{{- if and .Values.useKubeconfig (not .Values.config.hubKubeconfigSecretName) }} +{{- fail "config.hubKubeconfigSecretName must be set when useKubeconfig is true (it names the Secret, key \"kubeconfig\", to mount as the hub kubeconfig)" }} +{{- end }} apiVersion: apps/v1 kind: Deployment metadata: @@ -13,6 +22,9 @@ spec: metadata: labels: {{- include "member-agent.selectorLabels" . | nindent 8 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} spec: restartPolicy: Always serviceAccountName: {{ include "member-agent.fullname" . }}-sa @@ -40,6 +52,8 @@ spec: - --leader-elect=true {{- if .Values.useCAAuth }} - --use-ca-auth={{ .Values.useCAAuth }} + {{- else if $mountHubKubeconfig }} + - --use-kubeconfig=true {{- else }} - --tls-insecure={{ .Values.tlsClientInsecure }} {{- end }} @@ -90,19 +104,29 @@ spec: - --enable-namespace-collection-in-property-provider={{ .Values.enableNamespaceCollectionInPropertyProvider }} {{- end }} env: + {{- if not $mountHubKubeconfig }} - name: HUB_SERVER_URL value: "{{ .Values.config.hubURL }}" + {{- end }} + {{- if $mountHubKubeconfig }} + - name: KUBE_CONFIG_PATH + value: "/etc/kubefleet/hub-kubeconfig/kubeconfig" + {{- end }} + {{- if $useSidecar }} - name: CONFIG_PATH value: "/config/token" + {{- end }} - name: MEMBER_CLUSTER_NAME value: "{{ .Values.config.memberClusterName }}" + {{- if not $mountHubKubeconfig }} - name: HUB_CERTIFICATE_AUTHORITY value: "{{ .Values.config.hubCA }}" + {{- end }} {{- if .Values.useCAAuth }} - name: IDENTITY_KEY - value: "{{ .Values.config.identityKey }}" + value: "{{ if .Values.config.identitySecretName }}/etc/kubefleet/hub-identity/tls.key{{ else }}{{ .Values.config.identityKey }}{{ end }}" - name: IDENTITY_CERT - value: "{{ .Values.config.identityCert }}" + value: "{{ if .Values.config.identitySecretName }}/etc/kubefleet/hub-identity/tls.crt{{ else }}{{ .Values.config.identityCert }}{{ end }}" - name: CA_BUNDLE value: "{{ .Values.config.CABundle }}" {{- end }} @@ -137,10 +161,10 @@ spec: httpGet: path: /readyz port: hubhealthz - {{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }} + {{- if or ($useSidecar) (eq .Values.propertyProvider "azure") ($mountIdentitySecret) ($mountHubKubeconfig) (.Values.extraVolumeMounts) }} volumeMounts: - {{- if not .Values.useCAAuth }} - - name: provider-token + {{- if $useSidecar }} + - name: provider-token mountPath: /config {{- end }} {{- if eq .Values.propertyProvider "azure" }} @@ -148,8 +172,21 @@ spec: mountPath: /etc/kubernetes/provider readOnly: true {{- end }} + {{- if $mountIdentitySecret }} + - name: hub-identity + mountPath: /etc/kubefleet/hub-identity + readOnly: true + {{- end }} + {{- if $mountHubKubeconfig }} + - name: hub-kubeconfig + mountPath: /etc/kubefleet/hub-kubeconfig + readOnly: true + {{- end }} + {{- with .Values.extraVolumeMounts }} + {{- toYaml . | nindent 10 }} + {{- end }} {{- end }} - {{- if not .Values.useCAAuth }} + {{- if $useSidecar }} - name: refresh-token image: "{{ .Values.refreshtoken.repository }}:{{ .Values.refreshtoken.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.refreshtoken.pullPolicy }} @@ -175,9 +212,13 @@ spec: - name: provider-token mountPath: /config {{- end }} - {{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }} + {{- if .Values.extraInitContainers }} + initContainers: + {{- toYaml .Values.extraInitContainers | nindent 8 }} + {{- end }} + {{- if or ($useSidecar) (eq .Values.propertyProvider "azure") ($mountIdentitySecret) ($mountHubKubeconfig) (.Values.extraVolumes) }} volumes: - {{- if not .Values.useCAAuth }} + {{- if $useSidecar }} - name: provider-token emptyDir: {} {{- end }} @@ -186,6 +227,19 @@ spec: secret: secretName: cloud-config {{- end }} + {{- if $mountIdentitySecret }} + - name: hub-identity + secret: + secretName: {{ .Values.config.identitySecretName }} + {{- end }} + {{- if $mountHubKubeconfig }} + - name: hub-kubeconfig + secret: + secretName: {{ .Values.config.hubKubeconfigSecretName }} + {{- end }} + {{- with .Values.extraVolumes }} + {{- toYaml . | nindent 6 }} + {{- end }} {{- end }} {{- with .Values.nodeSelector }} nodeSelector: diff --git a/charts/member-agent/templates/serviceaccount.yaml b/charts/member-agent/templates/serviceaccount.yaml index e312541d3..213f0cd00 100644 --- a/charts/member-agent/templates/serviceaccount.yaml +++ b/charts/member-agent/templates/serviceaccount.yaml @@ -5,3 +5,7 @@ metadata: namespace: {{ .Values.namespace }} labels: {{- include "member-agent.labels" . | nindent 4 }} + {{- with .Values.serviceAccountAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} diff --git a/charts/member-agent/values.yaml b/charts/member-agent/values.yaml index 5e3c304c6..48d783e54 100644 --- a/charts/member-agent/values.yaml +++ b/charts/member-agent/values.yaml @@ -35,6 +35,19 @@ config: identityKey: "identity-key-path" identityCert: "identity-cert-path" CABundle: "ca-bundle-path" + # identitySecretName, when set, mounts a kubernetes.io/tls Secret (keys tls.crt/tls.key) of + # this name and uses it for IDENTITY_KEY/IDENTITY_CERT instead of the identityKey/identityCert + # paths above. Only used when useCAAuth is true. Leave empty ("") to keep bringing your own + # cert/key files by another means (identityKey/identityCert must then point at that mount). + identitySecretName: "" + # hubKubeconfigSecretName names a Secret (key "kubeconfig") containing a full kubeconfig for + # the hub connection, mounted and passed to the process via the KUBE_CONFIG_PATH environment + # variable. Only used when useKubeconfig is true. This is the vendor-neutral way to + # authenticate via a federated identity (AWS IRSA, GCP Workload Identity, Azure AD workload + # identity, SPIFFE, etc.): the kubeconfig's exec stanza invokes whatever credential plugin + # your platform provides — this chart has no vendor-specific knowledge of any of them. See + # the README for worked examples. + hubKubeconfigSecretName: "" azureCloudConfig: cloud: "" tenantId: "" @@ -59,6 +72,28 @@ azure: tlsClientInsecure: false useCAAuth: false +# useKubeconfig authenticates to the hub cluster via a full kubeconfig instead of the +# refresh-token sidecar or useCAAuth — see config.hubKubeconfigSecretName above for the +# companion value naming the Secret that holds it. Mutually exclusive with useCAAuth. +useKubeconfig: false + +# podLabels and serviceAccountAnnotations are passed straight through to the pod template and the +# member agent's ServiceAccount, respectively. Use them to attach whatever platform-specific +# workload-identity metadata your hub-kubeconfig's exec plugin needs — e.g. for AKS workload +# identity: podLabels: {azure.workload.identity/use: "true"}, serviceAccountAnnotations: +# {azure.workload.identity/client-id: }; for AWS IRSA or GCP Workload Identity, +# only a serviceAccountAnnotations entry is needed. See the README for worked examples. +podLabels: {} +serviceAccountAnnotations: {} + +# extraVolumes, extraVolumeMounts (on the member-agent container), and extraInitContainers are +# escape hatches for getting a hub-kubeconfig exec plugin's binary (or, for SPIFFE, a workload-API +# socket) into the pod, without this chart needing any vendor-specific knowledge of how. The +# alternative is building a custom image layered on top of the official one. +extraVolumes: [] +extraVolumeMounts: [] +extraInitContainers: [] + enablePprof: true pprofPort: 6065 hubPprofPort: 6066 diff --git a/cmd/memberagent/main.go b/cmd/memberagent/main.go index 7300acba3..2b1da861f 100644 --- a/cmd/memberagent/main.go +++ b/cmd/memberagent/main.go @@ -38,6 +38,7 @@ import ( "k8s.io/client-go/dynamic" clientgoscheme "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" "k8s.io/client-go/util/retry" "k8s.io/klog/v2" ctrl "sigs.k8s.io/controller-runtime" @@ -99,13 +100,15 @@ func main() { // Set up controller-runtime logger ctrl.SetLogger(zap.New(zap.UseDevMode(true))) - hubURL := os.Getenv("HUB_SERVER_URL") - - if hubURL == "" { - klog.ErrorS(errors.New("hub server api cannot be empty"), "Failed to read URL for the hub cluster") - klog.FlushAndExit(klog.ExitFlushTimeout, 1) + var hubURL string + if !opts.HubConnectivityOpts.UseKubeConfig { + hubURL = os.Getenv("HUB_SERVER_URL") + if hubURL == "" { + klog.ErrorS(errors.New("hub server api cannot be empty"), "Failed to read URL for the hub cluster") + klog.FlushAndExit(klog.ExitFlushTimeout, 1) + } } - hubConfig, err := buildHubConfig(hubURL, opts.HubConnectivityOpts.UseCertificateAuth, opts.HubConnectivityOpts.UseInsecureTLSClient) + hubConfig, err := buildHubConfig(hubURL, opts.HubConnectivityOpts) if err != nil { klog.ErrorS(err, "Failed to build Kubernetes client configuration for the hub cluster") klog.FlushAndExit(klog.ExitFlushTimeout, 1) @@ -182,11 +185,32 @@ func main() { } } -func buildHubConfig(hubURL string, useCertificateAuth bool, tlsClientInsecure bool) (*rest.Config, error) { +func buildHubConfig(hubURL string, opts options.HubConnectivityOptions) (*rest.Config, error) { var hubConfig = &rest.Config{ Host: hubURL, } - if useCertificateAuth { + + if opts.UseKubeConfig { + // A full kubeconfig is authoritative for the hub connection: it already carries the + // server URL, TLS configuration, and authentication (a bearer token, a client + // certificate, or a standard Kubernetes exec credential plugin for a federated + // identity), so none of the env-var/flag-based configuration below applies. + kubeConfigPath := os.Getenv("KUBE_CONFIG_PATH") + if kubeConfigPath == "" { + err := errors.New("hub kubeconfig file path cannot be empty if use-kubeconfig is set") + klog.ErrorS(err, "Failed to retrieve hub kubeconfig path") + return nil, err + } + clientConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( + &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeConfigPath}, + &clientcmd.ConfigOverrides{}) + loadedConfig, err := clientConfig.ClientConfig() + if err != nil { + klog.ErrorS(err, "Failed to load hub kubeconfig", "path", kubeConfigPath) + return nil, err + } + return applyHubKubeHeader(loadedConfig) + } else if opts.UseCertificateAuth { keyFilePath := os.Getenv("IDENTITY_KEY") certFilePath := os.Getenv("IDENTITY_CERT") if keyFilePath == "" { @@ -224,11 +248,19 @@ func buildHubConfig(hubURL string, useCertificateAuth bool, tlsClientInsecure bo hubConfig.BearerTokenFile = tokenFilePath } - hubConfig.TLSClientConfig.Insecure = tlsClientInsecure - if tlsClientInsecure { + return applyHubTLSConfig(hubConfig, opts) +} + +// applyHubTLSConfig applies the TLS insecure/CA settings shared by the certificate-auth and +// token-auth hub connection modes, then wraps the config with the HUB_KUBE_HEADER transport if +// one is configured. The kubeconfig mode does not go through this path: a loaded kubeconfig is +// already authoritative for TLS and skips straight to the header wrapping. +func applyHubTLSConfig(hubConfig *rest.Config, opts options.HubConnectivityOptions) (*rest.Config, error) { + hubConfig.TLSClientConfig.Insecure = opts.UseInsecureTLSClient + if opts.UseInsecureTLSClient { klog.Warning("TLS verification is disabled for hub cluster connection. This is insecure and should not be used in production.") } - if !tlsClientInsecure { + if !opts.UseInsecureTLSClient { caBundle, ok := os.LookupEnv("CA_BUNDLE") if ok && caBundle == "" { err := errors.New("environment variable CA_BUNDLE should not be empty") @@ -259,8 +291,14 @@ func buildHubConfig(hubURL string, useCertificateAuth bool, tlsClientInsecure bo } } - // Sometime the hub cluster need additional http header for authentication or authorization. - // the "HUB_KUBE_HEADER" to allow sending custom header to hub's API Server for authentication and authorization. + return applyHubKubeHeader(hubConfig) +} + +// applyHubKubeHeader wires in the "HUB_KUBE_HEADER" custom header, if set, to the given hub +// rest.Config. Sometimes the hub cluster needs an additional HTTP header for authentication or +// authorization; this applies regardless of how the rest of hubConfig was built, since it is an +// independent, orthogonal transport-level concern (e.g. a front-door header requirement). +func applyHubKubeHeader(hubConfig *rest.Config) (*rest.Config, error) { if header, ok := os.LookupEnv("HUB_KUBE_HEADER"); ok { r := textproto.NewReader(bufio.NewReader(strings.NewReader(header))) h, err := r.ReadMIMEHeader() diff --git a/cmd/memberagent/main_test.go b/cmd/memberagent/main_test.go index d729a6fba..041373ca5 100644 --- a/cmd/memberagent/main_test.go +++ b/cmd/memberagent/main_test.go @@ -5,27 +5,29 @@ import ( "github.com/stretchr/testify/assert" "k8s.io/client-go/rest" + + "github.com/kubefleet-dev/kubefleet/cmd/memberagent/options" ) func Test_buildHubConfig(t *testing.T) { t.Run("use CA auth, no key file - error", func(t *testing.T) { t.Setenv("IDENTITY_KEY", "") t.Setenv("IDENTITY_CERT", "/path/to/cert") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use CA auth, no cert file - error", func(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use CA auth - success", func(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "/path/to/cert") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -40,7 +42,7 @@ func Test_buildHubConfig(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "/path/to/cert") t.Setenv("CA_BUNDLE", "") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) @@ -48,7 +50,7 @@ func Test_buildHubConfig(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "/path/to/cert") t.Setenv("CA_BUNDLE", "/path/to/ca/bundle") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -63,7 +65,7 @@ func Test_buildHubConfig(t *testing.T) { t.Run("use CA data - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -77,7 +79,7 @@ func Test_buildHubConfig(t *testing.T) { t.Run("empty CA data - error", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_CERTIFICATE_AUTHORITY", "") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) @@ -85,25 +87,25 @@ func Test_buildHubConfig(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") t.Setenv("CA_BUNDLE", "/path/to/ca/bundle") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use token auth, no token path - error", func(t *testing.T) { t.Setenv("CONFIG_PATH", "") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use token auth, not exists token path - error", func(t *testing.T) { t.Setenv("CONFIG_PATH", "/hot/exists/token/path") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use token auth - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -113,7 +115,7 @@ func Test_buildHubConfig(t *testing.T) { }) t.Run("No CA bundle, no Hub CA, not insecure - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -123,7 +125,7 @@ func Test_buildHubConfig(t *testing.T) { }) t.Run("use insecure client - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") - config, err := buildHubConfig("https://hub.domain.com", false, true) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: true}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -137,9 +139,171 @@ func Test_buildHubConfig(t *testing.T) { t.Run("use insecure client and custom header - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_KUBE_HEADER", "Member-Resource-ID: some-id") - config, err := buildHubConfig("https://hub.domain.com", false, true) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: true}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.NotNil(t, config.WrapTransport) + }) + t.Run("use CA auth and insecure client - success", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: true}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, rest.Config{ + Host: "https://hub.domain.com", + TLSClientConfig: rest.TLSClientConfig{ + KeyFile: "/path/to/key", + CertFile: "/path/to/cert", + Insecure: true, + }, + }, *config) + }) + t.Run("use CA auth and CA data - success", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, rest.Config{ + Host: "https://hub.domain.com", + TLSClientConfig: rest.TLSClientConfig{ + KeyFile: "/path/to/key", + CertFile: "/path/to/cert", + CAData: []byte("this is a fake ca"), + }, + }, *config) + }) + t.Run("use CA auth, empty CA data - error", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_CERTIFICATE_AUTHORITY", "") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use CA auth, both of CA bundle and CA data present - error", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") + t.Setenv("CA_BUNDLE", "/path/to/ca/bundle") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use CA auth with custom header - success", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_KUBE_HEADER", "Member-Resource-ID: some-id") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.NotNil(t, config.WrapTransport) + }) + t.Run("use hub kubeconfig - success", func(t *testing.T) { + // hubURL is deliberately left empty here, mirroring main()'s behavior of skipping the + // HUB_SERVER_URL read entirely when UseKubeConfig is set: the kubeconfig's own + // cluster server URL is authoritative. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + assert.Equal(t, "fixture-bearer-token", config.BearerToken) + }) + t.Run("use hub kubeconfig with an exec credential plugin - success", func(t *testing.T) { + // Exercises the users[].user.exec flow this feature exists for (a federated-identity + // exec plugin, e.g. kubelogin/aws-iam-authenticator/gke-gcloud-auth-plugin), as opposed + // to the plain-bearer-token fixture above. clientcmd populates rest.Config.ExecProvider + // from the exec stanza but does not invoke the plugin binary at ClientConfig() time - + // that only happens lazily on the first real HTTP request - so this can assert on the + // parsed exec configuration without needing "kubelogin" to actually be on PATH. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig-exec") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + if assert.NotNil(t, config.ExecProvider) { + assert.Equal(t, "kubelogin", config.ExecProvider.Command) + assert.Equal(t, []string{"get-token", "--server-id", "fixture-server-id", "--login", "workloadidentity"}, config.ExecProvider.Args) + assert.Equal(t, "client.authentication.k8s.io/v1", config.ExecProvider.APIVersion) + } + }) + t.Run("use hub kubeconfig, no path - error", func(t *testing.T) { + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use hub kubeconfig, not exists - error", func(t *testing.T) { + t.Setenv("KUBE_CONFIG_PATH", "./testdata/does-not-exist") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use hub kubeconfig with custom header - success", func(t *testing.T) { + t.Setenv("HUB_KUBE_HEADER", "Member-Resource-ID: some-id") + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) assert.NotNil(t, config) assert.Nil(t, err) assert.NotNil(t, config.WrapTransport) }) + t.Run("use hub kubeconfig, malformed content - error", func(t *testing.T) { + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig-malformed") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use hub kubeconfig, hubURL argument is ignored - success", func(t *testing.T) { + // The kubeconfig's own cluster server URL is authoritative; a non-empty hubURL passed + // in (which main() only does when UseKubeConfig is unset) must not leak through. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("https://should-be-ignored.example.com", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + }) + t.Run("use hub kubeconfig, TLSClientConfig.Insecure from kubeconfig is not overridden - success", func(t *testing.T) { + // The fixture kubeconfig sets insecure-skip-tls-verify: true; UseInsecureTLSClient is + // left at its false zero-value here to confirm the kubeconfig's own setting wins, + // rather than being reset by the (ignored, per hub.go's doc comment) opts field. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + UseInsecureTLSClient: false, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.True(t, config.TLSClientConfig.Insecure) + }) + t.Run("use hub kubeconfig takes precedence over CA auth - success", func(t *testing.T) { + // options.Validate() rejects UseCertificateAuth+UseKubeConfig together in practice, but + // buildHubConfig itself does not re-validate; this locks in that UseKubeConfig, checked + // first in the if/else chain, wins if it is ever called with both set. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + UseCertificateAuth: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + assert.Empty(t, config.TLSClientConfig.CertFile) + assert.Empty(t, config.TLSClientConfig.KeyFile) + }) } diff --git a/cmd/memberagent/options/hub.go b/cmd/memberagent/options/hub.go index aacd89091..648bef8bb 100644 --- a/cmd/memberagent/options/hub.go +++ b/cmd/memberagent/options/hub.go @@ -29,9 +29,9 @@ type HubConnectivityOptions struct { // and certificate to use for authentication via the `IDENTITY_KEY` and `IDENTITY_CERT` // environment variables respectively. // - // Otherwise, the member agent will use token-based authentication when connecting - // to the hub cluster. The agent will read the token from the file path specified - // by the `CONFIG_PATH` environment variable. + // Otherwise, unless UseKubeConfig is set, the member agent will use token-based + // authentication when connecting to the hub cluster. The agent will read the token + // from the file path specified by the `CONFIG_PATH` environment variable. UseCertificateAuth bool // Use an insecure client or not when connecting to the hub cluster. @@ -41,6 +41,26 @@ type HubConnectivityOptions struct { // variable; you can also give the member agent a file path to the CA data instead // via the `HUB_CERTIFICATE_AUTHORITY` environment variable. UseInsecureTLSClient bool + + // Use a kubeconfig file for the hub cluster connection instead of the refresh-token + // sidecar or UseCertificateAuth. When set, provide the member agent the path to a + // kubeconfig file via the `KUBE_CONFIG_PATH` environment variable; it's loaded via + // client-go's clientcmd (the same mechanism kubectl uses). This takes over the hub + // connection entirely: HUB_SERVER_URL, UseCertificateAuth, UseInsecureTLSClient, + // CA_BUNDLE/HUB_CERTIFICATE_AUTHORITY, and CONFIG_PATH/IDENTITY_KEY/IDENTITY_CERT are all + // ignored, since the kubeconfig already carries the server URL, TLS configuration, and + // authentication (a bearer token, a client certificate, or a standard Kubernetes exec + // credential plugin — https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins). + // + // This is the vendor-neutral way to authenticate to the hub via a federated identity + // (AWS IRSA, GCP Workload Identity, Azure AD workload identity, SPIFFE, etc.): point this + // at a kubeconfig whose exec stanza invokes whatever credential plugin your platform + // provides. KubeFleet itself has no vendor-specific knowledge of any of these; the exec + // plugin binary must be present in the member agent image (e.g. via a custom image + // layered on top of the official one, or via the Helm chart's extraInitContainers). + // + // Mutually exclusive with UseCertificateAuth. + UseKubeConfig bool } // AddFlags adds flags for HubConnectivityOptions to the specified FlagSet. @@ -56,4 +76,10 @@ func (o *HubConnectivityOptions) AddFlags(flags *flag.FlagSet) { "tls-insecure", false, "Use an insecure client or not when connecting to the hub cluster.") + + flags.BoolVar( + &o.UseKubeConfig, + "use-kubeconfig", + false, + "Use a kubeconfig file for the hub cluster connection instead of the refresh-token sidecar or --use-ca-auth. Requires the KUBE_CONFIG_PATH environment variable to also be set. Mutually exclusive with --use-ca-auth.") } diff --git a/cmd/memberagent/options/options_test.go b/cmd/memberagent/options/options_test.go index dbc1f4c91..42b436b87 100644 --- a/cmd/memberagent/options/options_test.go +++ b/cmd/memberagent/options/options_test.go @@ -58,6 +58,18 @@ func TestHubConnectivityOptions(t *testing.T) { UseInsecureTLSClient: true, }, }, + { + name: "use kubeconfig specified", + flagSetName: "useKubeconfigSpecified", + args: []string{ + "--use-kubeconfig=true", + }, + wantHubConnectOpts: HubConnectivityOptions{ + UseCertificateAuth: false, + UseInsecureTLSClient: false, + UseKubeConfig: true, + }, + }, } for _, tc := range testCases { diff --git a/cmd/memberagent/options/validation.go b/cmd/memberagent/options/validation.go index 5016205d8..947e41171 100644 --- a/cmd/memberagent/options/validation.go +++ b/cmd/memberagent/options/validation.go @@ -28,6 +28,11 @@ func (o *Options) Validate() field.ErrorList { errs := field.ErrorList{} newPath := field.NewPath("Options") + // Cross-field validation for hub connectivity options. + if o.HubConnectivityOpts.UseCertificateAuth && o.HubConnectivityOpts.UseKubeConfig { + errs = append(errs, field.Invalid(newPath.Child("HubConnectivityOpts").Child("UseKubeConfig"), o.HubConnectivityOpts.UseKubeConfig, "UseCertificateAuth and UseKubeConfig must not both be set")) + } + // Cross-field validation for controller manager options. if float64(o.CtrlManagerOptions.HubManagerOpts.Burst) < o.CtrlManagerOptions.HubManagerOpts.QPS { errs = append(errs, field.Invalid(newPath.Child("HubManagerOpts").Child("Burst"), o.CtrlManagerOptions.HubManagerOpts.Burst, "The burst limit for hub cluster client-side throttling must be greater than or equal to its QPS limit")) diff --git a/cmd/memberagent/options/validation_test.go b/cmd/memberagent/options/validation_test.go index a77ef9ad9..5986401af 100644 --- a/cmd/memberagent/options/validation_test.go +++ b/cmd/memberagent/options/validation_test.go @@ -64,6 +64,15 @@ func TestValidate(t *testing.T) { opt: newTestOptions(nil), want: field.ErrorList{}, }, + "use certificate auth and use kubeconfig together": { + opt: newTestOptions(func(option *Options) { + option.HubConnectivityOpts.UseCertificateAuth = true + option.HubConnectivityOpts.UseKubeConfig = true + }), + want: field.ErrorList{ + field.Invalid(newPath.Child("HubConnectivityOpts").Child("UseKubeConfig"), true, "UseCertificateAuth and UseKubeConfig must not both be set"), + }, + }, "hub burst less than hub QPS": { opt: newTestOptions(func(option *Options) { option.CtrlManagerOptions.HubManagerOpts.QPS = 200 diff --git a/cmd/memberagent/testdata/kubeconfig b/cmd/memberagent/testdata/kubeconfig new file mode 100644 index 000000000..43859c4dd --- /dev/null +++ b/cmd/memberagent/testdata/kubeconfig @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Config +clusters: +- cluster: + server: https://hub.fixture.example.com + insecure-skip-tls-verify: true + name: hub +contexts: +- context: + cluster: hub + user: fixture-user + name: hub +current-context: hub +users: +- name: fixture-user + user: + token: fixture-bearer-token diff --git a/cmd/memberagent/testdata/kubeconfig-exec b/cmd/memberagent/testdata/kubeconfig-exec new file mode 100644 index 000000000..68f688c31 --- /dev/null +++ b/cmd/memberagent/testdata/kubeconfig-exec @@ -0,0 +1,27 @@ +apiVersion: v1 +kind: Config +clusters: +- cluster: + server: https://hub.fixture.example.com + insecure-skip-tls-verify: true + name: hub +contexts: +- context: + cluster: hub + user: fixture-exec-user + name: hub +current-context: hub +users: +- name: fixture-exec-user + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - fixture-server-id + - --login + - workloadidentity + provideClusterInfo: false + interactiveMode: Never diff --git a/cmd/memberagent/testdata/kubeconfig-malformed b/cmd/memberagent/testdata/kubeconfig-malformed new file mode 100644 index 000000000..d91a64f78 --- /dev/null +++ b/cmd/memberagent/testdata/kubeconfig-malformed @@ -0,0 +1,2 @@ +this is not: [valid kubeconfig yaml + - broken structure From a0a38bda930b2449ea559999a11281db5fa4f01a Mon Sep 17 00:00:00 2001 From: Akshita kumari <110122283+akshita317@users.noreply.github.com> Date: Thu, 24 Sep 2026 21:51:58 +0530 Subject: [PATCH 04/20] test: add scheduled-to-bound coverage for TestPickBindingsToRoll (#847) --- pkg/controllers/rollout/controller_test.go | 199 ++++++++++++++++++++- 1 file changed, 198 insertions(+), 1 deletion(-) diff --git a/pkg/controllers/rollout/controller_test.go b/pkg/controllers/rollout/controller_test.go index 8ac5b7884..e4395079a 100644 --- a/pkg/controllers/rollout/controller_test.go +++ b/pkg/controllers/rollout/controller_test.go @@ -1235,7 +1235,204 @@ func TestPickBindingsToRoll(t *testing.T) { wantWaitTime time.Duration wantErr error }{ - // TODO: add more tests + "test scheduled binding to bound, latest resources and nil overrides - rollout allowed": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster1), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickAllPlacementType, 0), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, generateDefaultRollingUpdateConfig(), nil)), + wantTobeUpdatedBindings: []int{0}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test scheduled binding to bound with PickFixed placement type - rollout allowed": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster1), + } + }, + latestResourceSnapshotName: "snapshot-2", + crp: clusterResourcePlacementForTest("test", + &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{cluster1}, + }, + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, generateDefaultRollingUpdateConfig(), nil)), + wantTobeUpdatedBindings: []int{0}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-2", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test multiple scheduled bindings with maxSurge set to zero and no ready bindings - rollout allowed": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster1), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster2), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickNPlacementType, 2), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, &placementv1beta1.RollingUpdateConfig{ + MaxUnavailable: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "20%", + }, + MaxSurge: &intstr.IntOrString{ + Type: intstr.Int, + IntVal: 0, + }, + UnavailablePeriodSeconds: ptr.To(1), + }, nil)), + // maxSurge bounds how far the can-be-ready set may exceed the target. Scheduled bindings + // are not counted as can-be-ready, so with nothing ready yet the budget is the full + // target number and both bindings are allowed to roll. + wantTobeUpdatedBindings: []int{0, 1}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster2, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test scheduled binding with maxSurge set to zero and target met by ready bound bindings - rollout blocked": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster1), + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster2), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster3), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickNPlacementType, 2), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, &placementv1beta1.RollingUpdateConfig{ + MaxUnavailable: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "20%", + }, + MaxSurge: &intstr.IntOrString{ + Type: intstr.Int, + IntVal: 0, + }, + UnavailablePeriodSeconds: ptr.To(1), + }, nil)), + // The two ready bound bindings already fill the target, so with maxSurge zero there is + // no room to bring the scheduled binding up and it stays unselected this round. + wantTobeUpdatedBindings: []int{}, + wantStaleUnselectedBindings: []int{2}, + wantUpToDateBoundBindings: []int{0, 1}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster2, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster3, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test multiple scheduled bindings with percentage maxSurge - surge budget rounds up": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster1), + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster2), + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster3), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster4), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster5), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster6), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickNPlacementType, 3), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, &placementv1beta1.RollingUpdateConfig{ + MaxUnavailable: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "20%", + }, + MaxSurge: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "50%", + }, + UnavailablePeriodSeconds: ptr.To(1), + }, nil)), + // 50% of the target of 3 rounds up to 2, so the can-be-ready ceiling is 5. The three + // ready bound bindings already count towards it, leaving room for exactly two of the + // three scheduled bindings; the third stays unselected this round. + wantTobeUpdatedBindings: []int{3, 4}, + wantStaleUnselectedBindings: []int{5}, + wantUpToDateBoundBindings: []int{0, 1, 2}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster2, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster3, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster4, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster5, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster6, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, "test scheduled binding to bound, outdated resources and nil overrides - rollout allowed": { allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { return []*placementv1beta1.ClusterResourceBinding{ From 6baf58f1c1c75f31b0099909630261c1faeece0b Mon Sep 17 00:00:00 2001 From: Akshita kumari <110122283+akshita317@users.noreply.github.com> Date: Mon, 28 Sep 2026 23:39:15 +0530 Subject: [PATCH 05/20] fix: report negative label indexes without a nil wrapped error (#939) ExtractIndex and ParsePolicyIndexFromLabel rejected a non-integer and a negative index in one branch, formatting both with ": %w" and the error from strconv.Atoi. For a negative index that error is nil, so the message ended in a formatting artifact: invalid policy index "-1", error: %!w() Handle the two cases separately. A non-integer still wraps the strconv error; a negative index now says: invalid policy index "-1": must be a non-negative integer The existing tests covered a negative index but only checked that an error came back. The new ones assert the message, and fail without the change on the negative-index cases. ParsePolicyIndexFromLabel had no tests; they cover valid indexes on cluster-scoped and namespaced snapshots, missing labels, a missing index label, a non-integer and a negative index. pkg/utils/labels goes from 46.7% to 100% coverage. Signed-off-by: Akshita kumari <110122283+akshita317@users.noreply.github.com> --- pkg/utils/labels/labels.go | 10 ++- pkg/utils/labels/labels_test.go | 133 ++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+), 2 deletions(-) diff --git a/pkg/utils/labels/labels.go b/pkg/utils/labels/labels.go index ee00c6acb..b832894d3 100644 --- a/pkg/utils/labels/labels.go +++ b/pkg/utils/labels/labels.go @@ -40,9 +40,12 @@ func ExtractResourceSnapshotIndexFromWork(work client.Object) (int, error) { func ExtractIndex(object client.Object, labelKey string) (int, error) { indexStr := object.GetLabels()[labelKey] v, err := strconv.Atoi(indexStr) - if err != nil || v < 0 { + if err != nil { return -1, fmt.Errorf("invalid resource index %q, error: %w", indexStr, err) } + if v < 0 { + return -1, fmt.Errorf("invalid resource index %q: must be a non-negative integer", indexStr) + } return v, nil } @@ -56,8 +59,11 @@ func ParsePolicyIndexFromLabel(policySnapshot client.Object) (int, error) { indexLabel := labels[fleetv1beta1.PolicyIndexLabel] v, err := strconv.Atoi(indexLabel) - if err != nil || v < 0 { + if err != nil { return -1, fmt.Errorf("invalid policy index %q, error: %w", indexLabel, err) } + if v < 0 { + return -1, fmt.Errorf("invalid policy index %q: must be a non-negative integer", indexLabel) + } return v, nil } diff --git a/pkg/utils/labels/labels_test.go b/pkg/utils/labels/labels_test.go index 97ac45392..b5adacb14 100644 --- a/pkg/utils/labels/labels_test.go +++ b/pkg/utils/labels/labels_test.go @@ -20,12 +20,15 @@ import ( "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" fleetv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" ) const ( snapshotName = "test-snapshot" + + nonIntegerIndex = "abc" ) func TestExtractResourceIndexFromClusterResourceSnapshot(t *testing.T) { @@ -169,3 +172,133 @@ func TestExtractResourceSnapshotIndexFromWork(t *testing.T) { }) } } + +func TestExtractIndex_ErrorMessage(t *testing.T) { + testCases := []struct { + name string + labelValue string + wantErrMsg string + }{ + { + name: "not an integer", + labelValue: nonIntegerIndex, + wantErrMsg: `invalid resource index "abc", error: strconv.Atoi: parsing "abc": invalid syntax`, + }, + { + name: "negative integer", + labelValue: "-1", + wantErrMsg: `invalid resource index "-1": must be a non-negative integer`, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + snapshot := &fleetv1beta1.ClusterResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.ResourceIndexLabel: tc.labelValue}, + }, + } + _, err := ExtractIndex(snapshot, fleetv1beta1.ResourceIndexLabel) + if err == nil { + t.Fatalf("ExtractIndex() error = nil, want %q", tc.wantErrMsg) + } + if err.Error() != tc.wantErrMsg { + t.Errorf("ExtractIndex() error = %q, want %q", err.Error(), tc.wantErrMsg) + } + }) + } +} + +func TestParsePolicyIndexFromLabel(t *testing.T) { + testCases := []struct { + name string + policySnapshot client.Object + wantIndex int + wantErrMsg string + }{ + { + name: "valid index on a cluster-scoped snapshot", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: "3"}, + }, + }, + wantIndex: 3, + }, + { + name: "valid index on a namespaced snapshot", + policySnapshot: &fleetv1beta1.SchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Namespace: "test-namespace", + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: "0"}, + }, + }, + wantIndex: 0, + }, + { + name: "no labels", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{Name: snapshotName}, + }, + wantIndex: -1, + wantErrMsg: "no labels found on policy snapshot", + }, + { + name: "index label missing", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{"other": "label"}, + }, + }, + wantIndex: -1, + wantErrMsg: `invalid policy index "", error: strconv.Atoi: parsing "": invalid syntax`, + }, + { + name: "not an integer", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: nonIntegerIndex}, + }, + }, + wantIndex: -1, + wantErrMsg: `invalid policy index "abc", error: strconv.Atoi: parsing "abc": invalid syntax`, + }, + { + name: "negative integer", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: "-1"}, + }, + }, + wantIndex: -1, + wantErrMsg: `invalid policy index "-1": must be a non-negative integer`, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotIndex, err := ParsePolicyIndexFromLabel(tc.policySnapshot) + if gotIndex != tc.wantIndex { + t.Errorf("ParsePolicyIndexFromLabel() index = %d, want %d", gotIndex, tc.wantIndex) + } + if tc.wantErrMsg == "" { + if err != nil { + t.Fatalf("ParsePolicyIndexFromLabel() error = %v, want nil", err) + } + return + } + if err == nil { + t.Fatalf("ParsePolicyIndexFromLabel() error = nil, want %q", tc.wantErrMsg) + } + if err.Error() != tc.wantErrMsg { + t.Errorf("ParsePolicyIndexFromLabel() error = %q, want %q", err.Error(), tc.wantErrMsg) + } + }) + } +} From 6221437a5e698eb5771f4ea440d32c014d503ba0 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 1 Oct 2026 21:46:26 +1000 Subject: [PATCH 06/20] chore: bump the all-others group across 1 directory with 4 updates (#940) --- go.mod | 8 ++++---- go.sum | 24 ++++++++++++------------ 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/go.mod b/go.mod index a8c1ce1d0..af0c9ae0d 100644 --- a/go.mod +++ b/go.mod @@ -6,12 +6,12 @@ require ( github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 github.com/Azure/karpenter-provider-azure v1.14.2 - github.com/crossplane/crossplane-runtime/v2 v2.4.0 + github.com/crossplane/crossplane-runtime/v2 v2.4.2 github.com/evanphx/json-patch/v5 v5.9.11 github.com/go-logr/logr v1.4.4 github.com/google/go-cmp v0.7.0 - github.com/onsi/ginkgo/v2 v2.32.2 - github.com/onsi/gomega v1.43.0 + github.com/onsi/ginkgo/v2 v2.33.0 + github.com/onsi/gomega v1.44.0 github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_model v0.6.3 github.com/qri-io/jsonpointer v0.1.1 @@ -20,7 +20,7 @@ require ( github.com/stretchr/testify v1.12.1 github.com/wI2L/jsondiff v0.7.1 go.goms.io/fleet-networking v0.3.44 - go.uber.org/atomic v1.11.0 + go.uber.org/atomic v1.12.0 go.uber.org/zap v1.28.0 golang.org/x/sync v0.23.0 golang.org/x/time v0.16.0 diff --git a/go.sum b/go.sum index 140652798..e0d809464 100644 --- a/go.sum +++ b/go.sum @@ -110,8 +110,8 @@ github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2y github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= -github.com/crossplane/crossplane-runtime/v2 v2.4.0 h1:m5iGVHbtEh9nZYz7w/wUWDsXVlpG2c8v5i0SZScwHh8= -github.com/crossplane/crossplane-runtime/v2 v2.4.0/go.mod h1:Pf+hg5A/46QaGjKi7Pk+HdsrFOA/THvST9qM4El8Rzs= +github.com/crossplane/crossplane-runtime/v2 v2.4.2 h1:MXc37IyL7iyrQIxs0qrAtdOsy0mdQkxOMie4UVWSFQ4= +github.com/crossplane/crossplane-runtime/v2 v2.4.2/go.mod h1:jDwOCwt+LMD6KXsnvvm1mNNiVqKZ2ZWZxuUnkxdkBPE= github.com/crossplane/crossplane/apis/v2 v2.3.4 h1:qB1YqF300msWEkDQTFUhqfeOq3HtVfSZ1+3wNz1JStA= github.com/crossplane/crossplane/apis/v2 v2.3.4/go.mod h1:s0cfMZA+Y/dwQ577PLQn5SBWyUg9k6vKjGAkHEwBShg= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -271,10 +271,10 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/oklog/ulid/v2 v2.1.1 h1:suPZ4ARWLOJLegGFiZZ1dFAkqzhMjL3J1TzI+5wHz8s= github.com/oklog/ulid/v2 v2.1.1/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= -github.com/onsi/ginkgo/v2 v2.32.2 h1:2o6vyFvR6snrJWgRVztC+OwuqqPEMI1UzYl2s2iU7Cg= -github.com/onsi/ginkgo/v2 v2.32.2/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= -github.com/onsi/gomega v1.43.0 h1:VlG/1FxqNxhSO+lq/OHBNaaqwiBK/mO8JbVkX9Y+FeU= -github.com/onsi/gomega v1.43.0/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg= +github.com/onsi/ginkgo/v2 v2.33.0 h1:C8gBA6Uc2ZEubiV+SXiu5tZnMTwEmXHgkJwGozKtZf8= +github.com/onsi/ginkgo/v2 v2.33.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= +github.com/onsi/gomega v1.44.0 h1:eAiGl3Pw5jz5GQdDff0BcxYpAX1JxW8xD7mFUuwNfZQ= +github.com/onsi/gomega v1.44.0/go.mod h1:e/C2HwaZ1DhvjzXXuFhcR7hY7Sh9pl7MmoWKEjzwcdA= github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc= github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ= github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ= @@ -351,8 +351,8 @@ go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6 go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o= go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c= go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI= -go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= -go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= +go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE= +go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y= @@ -432,10 +432,10 @@ golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gomodules.xyz/jsonpatch/v2 v2.5.0 h1:JELs8RLM12qJGXU4u/TO3V25KW8GreMKl9pdkk14RM0= gomodules.xyz/jsonpatch/v2 v2.5.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY= -google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1:Kjn0N0tCrDgiAFW+lGO4JZ3ck44CehvJQMAwj9QF0G8= -google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d h1:FarXi840EJWSHYTN3ERkADbPWjl307+FGrA22KAVjjc= +google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d/go.mod h1:K/+WGbmBY7aNW1HDw1fJnKYo10i0DkAX6pows00dLig= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d h1:IL4hdHzcUv2l/gcg98/Rj3FbtE6axwqslOW8SW0C+S0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU= google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8= google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= From 54c554f3efb7e01e8ea21b7fd6f92e9e25be9774 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 1 Oct 2026 21:46:53 +1000 Subject: [PATCH 07/20] chore: bump codecov/codecov-action from 7.1.0 to 7.1.1 (#934) --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bc9a86b3c..e8aa678c6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -77,7 +77,7 @@ jobs: KUBEFLEET_CI_TEST_RUNNER_NAME: 'ginkgo' - name: Upload Codecov report - uses: codecov/codecov-action@0b35c9ecc4f0529d0eb674914510c22f85b196b4 # v7.1.0 + uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1 with: ## Repository upload token - get it from codecov.io. Required only for private repositories token: ${{ secrets.CODECOV_TOKEN }} From 2e6ad573073a06a0645761363aed1ffd6de67d12 Mon Sep 17 00:00:00 2001 From: Chen Yu Date: Tue, 6 Oct 2026 07:00:58 +0800 Subject: [PATCH 08/20] feat: [FEP-0001] use field indexes in the new work generator (#942) --- ...0001-add-work-owner-binding-field-index.md | 39 +++++++++++++++++++ .../placementresourcesnapshot_types.go | 2 +- pkg/v1/controllers/workgenerator/retrieval.go | 23 +++++++---- pkg/v1/utils/fieldindexers/hub.go | 27 +++++++++++++ 4 files changed, 82 insertions(+), 9 deletions(-) create mode 100644 .github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md diff --git a/.github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md b/.github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md new file mode 100644 index 000000000..d0431a8e1 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md @@ -0,0 +1,39 @@ +# Add Work Owner Binding Field Index + +**Date**: October 2, 2026 00:01 AEST +**Task**: Add a composite field index for Work objects keyed by owner binding namespace and name. + +## Requirements + +1. Add the `workOwnedByBinding` custom field. +2. Format indexed values as `%s/%s`. +3. Extract the owner namespace from `WorkOwnerNamespaceLabelKey`. +4. Extract the owner binding name from `WorkOwnedByPlacementBindingLabelKey`. +5. Register the Work field extractor in `SetupWithHubControllerManager`. +6. Use the custom field index when listing Work objects by owner binding. +7. Use the existing owner-and-index custom field when listing secondary placement resource snapshots. + +## Plan + +1. Add exported field-name and value-format constants to `pkg/v1/utils/fieldindexers/hub.go`. +2. Add a Work owner-binding field extractor using the owner namespace and binding labels, while distinguishing a missing owner-namespace label from a valid empty cluster-scoped namespace. +3. Register the extractor for `placementv1alpha1.Work` in `SetupWithHubControllerManager`. +4. Update `listWorksByOwnerBinding` to query the composite custom field while retaining the Work namespace filter. +5. Update secondary placement resource snapshot retrieval to query the existing composite owner-and-index field. +6. Run the targeted field-indexer and workgenerator package tests and formatting. + +## Decisions + +1. Use the owner-binding label as the indexed binding-name source. +2. Permit an empty owner namespace only when the owner-namespace label is present, preserving support for cluster-scoped bindings. +3. Unit tests are omitted for now at the user's request. + +## Status + +Implementation complete. + +## Validation + +- `gofmt -w pkg/v1/utils/fieldindexers/hub.go pkg/v1/controllers/workgenerator/retrieval.go` +- `go test ./pkg/v1/utils/fieldindexers ./pkg/v1/controllers/workgenerator` (passed; packages currently have no test files) +- `git diff --check` (passed) diff --git a/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go b/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go index f98db8ed8..ece2237d4 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go @@ -41,7 +41,7 @@ const ( // might truncate the name and add a hash suffix as needed. // // This label is set on all placement resource snapshots. - PlacementResourceSnapshotOwnedByLabelKey = "placement.kubefleet.dev/placement-resource-snapshot-owned-by" + PlacementResourceSnapshotOwnedByLabelKey = "placement.kubefleet.dev/owned-by-placement-policy" // PlacementResourceSnapshotIndexLabelKey is a label key that denotes the index of a placement resource snapshot. // Its value is the index integer formatted as a string. // diff --git a/pkg/v1/controllers/workgenerator/retrieval.go b/pkg/v1/controllers/workgenerator/retrieval.go index 040fd8740..119800b1d 100644 --- a/pkg/v1/controllers/workgenerator/retrieval.go +++ b/pkg/v1/controllers/workgenerator/retrieval.go @@ -29,6 +29,7 @@ import ( placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" "github.com/kubefleet-dev/kubefleet/pkg/utils" "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/fieldindexers" ) func (r *Reconciler) retrievePlacementBinding(ctx context.Context, namespacedName types.NamespacedName) (placementv1alpha1.PlacementBindingAccessor, error) { @@ -55,9 +56,12 @@ func (r *Reconciler) listWorksByOwnerBinding(ctx context.Context, clusterName, o workList := &placementv1alpha1.WorkList{} listOptions := []client.ListOption{ client.InNamespace(memberClusterNamespace), - client.MatchingLabels{ - placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: workOwnerLabelValue(ownerBindingName), - placementv1alpha1.WorkOwnerNamespaceLabelKey: ownerBindingNSName, + client.MatchingFields{ + fieldindexers.WorkOwnedByBindingCustomFieldName: fmt.Sprintf( + fieldindexers.WorkOwnedByBindingCustomFieldValFmt, + ownerBindingNSName, + workOwnerLabelValue(ownerBindingName), + ), }, } if err := r.hubClient.List(ctx, workList, listOptions...); err != nil { @@ -109,15 +113,18 @@ func (r *Reconciler) retrievePrimaryAndSecondaryPlacementResourceSnapshots( return nil, errors.NewUnexpectedError(nil, "the primary placement resource snapshot is missing required labels", "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot)) } - labelMatchers := client.MatchingLabels{ - placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: ownedBy, - placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: index, + fieldMatchers := client.MatchingFields{ + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, + ownedBy, + index, + ), } var snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor if namespace == "" { snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} - if err := r.hubClient.List(ctx, snapshotList, labelMatchers); err != nil { + if err := r.hubClient.List(ctx, snapshotList, fieldMatchers); err != nil { return nil, errors.NewAPIServerError(err, "failed to list cluster placement resource snapshots", true) } snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotList.Items)) @@ -126,7 +133,7 @@ func (r *Reconciler) retrievePrimaryAndSecondaryPlacementResourceSnapshots( } } else { snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} - if err := r.hubClient.List(ctx, snapshotList, client.InNamespace(namespace), labelMatchers); err != nil { + if err := r.hubClient.List(ctx, snapshotList, client.InNamespace(namespace), fieldMatchers); err != nil { return nil, errors.NewAPIServerError(err, "failed to list placement resource snapshots", true) } snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotList.Items)) diff --git a/pkg/v1/utils/fieldindexers/hub.go b/pkg/v1/utils/fieldindexers/hub.go index d6d253d34..a4838ee94 100644 --- a/pkg/v1/utils/fieldindexers/hub.go +++ b/pkg/v1/utils/fieldindexers/hub.go @@ -50,6 +50,10 @@ const ( // This is added to help the placement resource snapshot manager retrieve all placement resource snapshots of a // specific index associated with a placement policy. PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName = "ownedByWithIndex" + + // WorkOwnedByBindingCustomFieldName is the name of the custom field that indexes Work objects by their owner + // placement bindings. + WorkOwnedByBindingCustomFieldName = "workOwnedByBinding" ) const ( @@ -66,6 +70,12 @@ const ( // // Note that slashes are used to avoid unexpected collisions. PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt = "%s/%s" + + // WorkOwnedByBindingCustomFieldValFmt is used to format the value for the custom field, + // `WorkOwnedByBindingCustomFieldName`. + // + // The first value is the owner binding namespace, and the second value is the owner binding name. + WorkOwnedByBindingCustomFieldValFmt = "%s/%s" ) type fieldValueExtractor func(obj client.Object) ([]string, error) @@ -110,6 +120,16 @@ var ( } return []string{fmt.Sprintf(PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, ownedBy, index)}, nil } + + workOwnedByBindingFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + ownerNS, ownerNSFound := obj.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerBinding := obj.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if !ownerNSFound || ownerBinding == "" { + wrappedErr := errors.NewUnexpectedError(nil, "work is missing required owner binding metadata") + return nil, wrappedErr + } + return []string{fmt.Sprintf(WorkOwnedByBindingCustomFieldValFmt, ownerNS, ownerBinding)}, nil + } ) // SetupWithHubControllerManager sets up the indices that controllers from the KubeFleet hub agent need to run properly. @@ -145,5 +165,12 @@ func SetupWithHubControllerManager(ctx context.Context, mgr ctrl.Manager) error return errors.Wraps(err, "failed to set up cluster placement resource snapshot owner and index field index") } + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.Work{}, + WorkOwnedByBindingCustomFieldName, workOwnedByBindingFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up work owner binding field index") + } + return nil } From a0255703675043589a94d59dbe1e4f1a194b456b Mon Sep 17 00:00:00 2001 From: Chen Yu Date: Wed, 7 Oct 2026 05:15:24 +0800 Subject: [PATCH 09/20] feat: [FEP-0001] add work applier (1/) (#911) --- .../2026-09-10-0000-cleanup-leftover-works.md | 32 + apis/cluster/v1beta1/zz_generated.deepcopy.go | 2 +- .../placement/v1alpha1/gvk_info.go | 5 + .../placement/v1alpha1/work_types.go | 80 +- .../v1alpha1/zz_generated.deepcopy.go | 110 ++ .../rollout/v1alpha1/zz_generated.deepcopy.go | 2 +- .../v1alpha1/zz_generated.deepcopy.go | 2 +- .../v1beta1/zz_generated.deepcopy.go | 2 +- .../placement.kubefleet.dev_appliedworks.yaml | 103 ++ pkg/controllers/workapplier/waves.go | 4 +- pkg/utils/errors/errors.go | 15 + pkg/utils/errors/errors_test.go | 102 + pkg/v1/controllers/workapplier/apply.go | 674 +++++++ .../workapplier/availability_tracker.go | 328 ++++ pkg/v1/controllers/workapplier/cleanup.go | 316 ++++ pkg/v1/controllers/workapplier/controller.go | 374 ++++ .../controller_integration_test.go | 1636 +++++++++++++++++ pkg/v1/controllers/workapplier/defaults.go | 47 + pkg/v1/controllers/workapplier/join_leave.go | 103 ++ pkg/v1/controllers/workapplier/preprocess.go | 582 ++++++ pkg/v1/controllers/workapplier/process.go | 491 +++++ pkg/v1/controllers/workapplier/retrieval.go | 201 ++ pkg/v1/controllers/workapplier/status.go | 528 ++++++ pkg/v1/controllers/workapplier/suite_test.go | 216 +++ .../workapplier/takeover_drift_detection.go | 573 ++++++ pkg/v1/controllers/workapplier/utils_test.go | 701 +++++++ pkg/v1/controllers/workapplier/waves.go | 180 ++ pkg/v1/utils/fieldindexers/hub.go | 39 +- pkg/v1/utils/fieldindexers/member.go | 115 ++ pkg/v1/utils/fieldindexers/utils.go | 48 + pkg/v1/utils/ownerreferences/refs.go | 63 + test/apis/v1alpha1/zz_generated.deepcopy.go | 2 +- 32 files changed, 7638 insertions(+), 38 deletions(-) create mode 100644 .github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md create mode 100644 config/crd/bases/placement.kubefleet.dev_appliedworks.yaml create mode 100644 pkg/v1/controllers/workapplier/apply.go create mode 100644 pkg/v1/controllers/workapplier/availability_tracker.go create mode 100644 pkg/v1/controllers/workapplier/cleanup.go create mode 100644 pkg/v1/controllers/workapplier/controller.go create mode 100644 pkg/v1/controllers/workapplier/controller_integration_test.go create mode 100644 pkg/v1/controllers/workapplier/defaults.go create mode 100644 pkg/v1/controllers/workapplier/join_leave.go create mode 100644 pkg/v1/controllers/workapplier/preprocess.go create mode 100644 pkg/v1/controllers/workapplier/process.go create mode 100644 pkg/v1/controllers/workapplier/retrieval.go create mode 100644 pkg/v1/controllers/workapplier/status.go create mode 100644 pkg/v1/controllers/workapplier/suite_test.go create mode 100644 pkg/v1/controllers/workapplier/takeover_drift_detection.go create mode 100644 pkg/v1/controllers/workapplier/utils_test.go create mode 100644 pkg/v1/controllers/workapplier/waves.go create mode 100644 pkg/v1/utils/fieldindexers/member.go create mode 100644 pkg/v1/utils/fieldindexers/utils.go create mode 100644 pkg/v1/utils/ownerreferences/refs.go diff --git a/.github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md b/.github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md new file mode 100644 index 000000000..27a2ab336 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md @@ -0,0 +1,32 @@ +# Cleanup Leftover Works + +## Goal + +Implement `cleanupLeftOverWorks` so manifests previously applied or prepared by leftover Work objects are removed unless still referenced by the current linked Work set. + +## Plan + +- [x] Collect manifest identifiers whose Applied condition is true or has the preparing-to-process reason. +- [x] Exclude identifiers present in `seenManifIDs`. +- [x] Remove the collected manifests in parallel through `removeOneLeftOverManifest`. +- [x] Aggregate per-manifest failures and run focused validation. + +## Success Criteria + +- Current manifests are preserved. +- Only previously applied or pre-processed leftover manifests are selected. +- Selected manifests are removed through the existing ownership-aware helper in one parallel batch. + +## Validation + +- `gofmt` completed successfully. +- Editor diagnostics report no errors in `cleanup.go`. +- `go test ./pkg/v1/controllers/workapplier` is currently blocked by pre-existing unused `leftOverWorks` and `seenManifestIDs` variables in `controller.go`, where the surrounding cleanup call site is still unfinished. + +## Placement Deletion Follow-Up + +- [ ] Preserve explicit sync strategy values while applying defaults. +- [ ] Delete linked AppliedWork objects with foreground or orphan propagation according to `WhenPlacementDeleted`. +- [ ] After `foregroundDeletionWaitTime`, explicitly clean tracked manifests blocking foreground deletion. +- [ ] Remove Work cleanup finalizers only after the corresponding AppliedWork is gone. +- [ ] Wire placement and leftover cleanup into reconciliation and validate the package. diff --git a/apis/cluster/v1beta1/zz_generated.deepcopy.go b/apis/cluster/v1beta1/zz_generated.deepcopy.go index a51641c3f..cec52aa39 100644 --- a/apis/cluster/v1beta1/zz_generated.deepcopy.go +++ b/apis/cluster/v1beta1/zz_generated.deepcopy.go @@ -21,7 +21,7 @@ limitations under the License. package v1beta1 import ( - "k8s.io/api/core/v1" + v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" ) diff --git a/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go b/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go index 81efe4327..40d7c469f 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go +++ b/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go @@ -33,3 +33,8 @@ var ( // AddToScheme adds the types in this group-version to the given scheme. AddToScheme = SchemeBuilder.AddToScheme ) + +const ( + // The kind names for all the custom resources in this GV. + AppliedWorkKind = "AppliedWork" +) diff --git a/apis/kubefleet.dev/placement/v1alpha1/work_types.go b/apis/kubefleet.dev/placement/v1alpha1/work_types.go index eb78ba88c..dec02200b 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/work_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/work_types.go @@ -19,6 +19,7 @@ package v1alpha1 import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" ) const ( @@ -54,6 +55,11 @@ const ( WorkDerivedFromSourceAnnotationKey = "placement.kubefleet.dev/derived-from" ) +const ( + LastAppliedConfigAnnotationKey = "placement.kubefleet.dev/last-applied-configuration" + LastAppliedManifestHashAnnotationKey = "placement.kubefleet.dev/manifest-hash" +) + const ( // The condition types for the Work API. WorkCondTypeApplied = "Applied" @@ -64,6 +70,15 @@ const ( ManifestCondTypeAvailable = "Available" ) +const ( + WorkAppliedCondPreparingToProcessReason = "PreparingToProcess" + WorkAppliedCondAllManifestsAppliedReason = "AllManifestsApplied" + WorkAppliedCondNotAllManifestsAppliedReason = "SomeManifestsAreNotYetApplied" + + WorkAvailableCondAllManifestsAvailableReason = "AllManifestsAvailable" + WorkAvailableCondNotAllManifestsAvailableReason = "SomeManifestsAreNotYetAvailable" +) + // Work is the KubeFleet API used for synchronizing resources to place between // the hub cluster and a member cluster in the member cluster reserved namespace. // @@ -231,7 +246,70 @@ type WorkList struct { Items []Work `json:"items"` } +// AppliedWork is the KubeFleet API that serves as owners for manifests applied to a member cluster via +// a work object. +// +// +genclient +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster,categories={kubefleet, kubefleet-placement} +// +kubebuilder:storageversion +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type AppliedWork struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // Spec represents the desired configuration of AppliedWork. + // +kubebuilder:validation:Required + Spec AppliedWorkSpec `json:"spec"` + + // Status represents the current status of AppliedWork. + // +kubebuilder:validation:Optional + Status AppliedWorkStatus `json:"status,omitempty"` +} + +type AppliedWorkSpec struct { + // The name of the corresponding work object. Note that the object is on the KubeFleet hub cluster. + // + // +kubebuilder:validation:Required + WorkName string `json:"workName"` + + // The namespace of the corresponding work object. Note that the object is on the KubeFleet hub cluster. + // + // +kubebuilder:validation:Required + WorkNamespace string `json:"workNamespace"` +} + +type AppliedWorkStatus struct { + // A list of applied manifests. This list is populated for informational purposes only. + // + // +kubebuilder:validation:Optional + AppliedResources []AppliedResource `json:"appliedResources,omitempty"` +} + +type AppliedResource struct { + // The ID of the applied manifest. + ManifestIdentifier `json:",inline"` + + // The UID of the applied manifest. + // + // +kubebuilder:validation:Required + UID types.UID `json:"uid,omitempty"` +} + +// AppliedWorkList contains a list of AppliedWork. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Cluster +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type AppliedWorkList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []AppliedWork `json:"items"` +} + // Set up the API types with the scheme builder. func init() { - SchemeBuilder.Register(&Work{}, &WorkList{}) + SchemeBuilder.Register(&Work{}, &WorkList{}, &AppliedWork{}, &AppliedWorkList{}) } diff --git a/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go b/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go index d7f6bfc40..5d97c6694 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go +++ b/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go @@ -26,6 +26,116 @@ import ( "k8s.io/apimachinery/pkg/util/intstr" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedResource) DeepCopyInto(out *AppliedResource) { + *out = *in + out.ManifestIdentifier = in.ManifestIdentifier +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedResource. +func (in *AppliedResource) DeepCopy() *AppliedResource { + if in == nil { + return nil + } + out := new(AppliedResource) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWork) DeepCopyInto(out *AppliedWork) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWork. +func (in *AppliedWork) DeepCopy() *AppliedWork { + if in == nil { + return nil + } + out := new(AppliedWork) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AppliedWork) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWorkList) DeepCopyInto(out *AppliedWorkList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]AppliedWork, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWorkList. +func (in *AppliedWorkList) DeepCopy() *AppliedWorkList { + if in == nil { + return nil + } + out := new(AppliedWorkList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AppliedWorkList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWorkSpec) DeepCopyInto(out *AppliedWorkSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWorkSpec. +func (in *AppliedWorkSpec) DeepCopy() *AppliedWorkSpec { + if in == nil { + return nil + } + out := new(AppliedWorkSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWorkStatus) DeepCopyInto(out *AppliedWorkStatus) { + *out = *in + if in.AppliedResources != nil { + in, out := &in.AppliedResources, &out.AppliedResources + *out = make([]AppliedResource, len(*in)) + copy(*out, *in) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWorkStatus. +func (in *AppliedWorkStatus) DeepCopy() *AppliedWorkStatus { + if in == nil { + return nil + } + out := new(AppliedWorkStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *BindingManager) DeepCopyInto(out *BindingManager) { *out = *in diff --git a/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go b/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go index fd435e83e..a20199581 100644 --- a/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go +++ b/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go @@ -21,7 +21,7 @@ limitations under the License. package v1alpha1 import ( - "k8s.io/apimachinery/pkg/apis/meta/v1" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" ) diff --git a/apis/placement/v1alpha1/zz_generated.deepcopy.go b/apis/placement/v1alpha1/zz_generated.deepcopy.go index 6d1656d18..df9f5e6d7 100644 --- a/apis/placement/v1alpha1/zz_generated.deepcopy.go +++ b/apis/placement/v1alpha1/zz_generated.deepcopy.go @@ -22,7 +22,7 @@ package v1alpha1 import ( "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" - "k8s.io/apimachinery/pkg/apis/meta/v1" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" ) diff --git a/apis/placement/v1beta1/zz_generated.deepcopy.go b/apis/placement/v1beta1/zz_generated.deepcopy.go index 73d66c8fa..b9ff2e710 100644 --- a/apis/placement/v1beta1/zz_generated.deepcopy.go +++ b/apis/placement/v1beta1/zz_generated.deepcopy.go @@ -21,7 +21,7 @@ limitations under the License. package v1beta1 import ( - "k8s.io/apimachinery/pkg/apis/meta/v1" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" ) diff --git a/config/crd/bases/placement.kubefleet.dev_appliedworks.yaml b/config/crd/bases/placement.kubefleet.dev_appliedworks.yaml new file mode 100644 index 000000000..571549fb6 --- /dev/null +++ b/config/crd/bases/placement.kubefleet.dev_appliedworks.yaml @@ -0,0 +1,103 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: appliedworks.placement.kubefleet.dev +spec: + group: placement.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-placement + kind: AppliedWork + listKind: AppliedWorkList + plural: appliedworks + singular: appliedwork + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + AppliedWork is the KubeFleet API that serves as owners for manifests applied to a member cluster via + a work object. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: Spec represents the desired configuration of AppliedWork. + properties: + workName: + description: The name of the corresponding work object. Note that + the object is on the KubeFleet hub cluster. + type: string + workNamespace: + description: The namespace of the corresponding work object. Note + that the object is on the KubeFleet hub cluster. + type: string + required: + - workName + - workNamespace + type: object + status: + description: Status represents the current status of AppliedWork. + properties: + appliedResources: + description: A list of applied manifests. This list is populated for + informational purposes only. + items: + properties: + apiGroup: + type: string + apiVersion: + type: string + kind: + type: string + name: + description: The name of the manifest. + type: string + namespace: + description: The namespace of the manifest. + type: string + ordinal: + description: The ordinal of the manifest. + type: integer + resource: + type: string + uid: + description: The UID of the applied manifest. + type: string + required: + - apiVersion + - kind + - name + - ordinal + - resource + - uid + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/pkg/controllers/workapplier/waves.go b/pkg/controllers/workapplier/waves.go index 6433584be..9a810b9dc 100644 --- a/pkg/controllers/workapplier/waves.go +++ b/pkg/controllers/workapplier/waves.go @@ -112,8 +112,8 @@ type bundleProcessingWave struct { bundles []*manifestProcessingBundle } -// organizeBundlesIntoProcessingWaves organizes the list of bundles into different -// waves of bundles for parallel processing based on their GVR information. +// organizeBundlesIntoProcessingWaves organizes the list of manifests (their processing states) into different +// waves for parallel processing based on their GVR information. func organizeBundlesIntoProcessingWaves(bundles []*manifestProcessingBundle, workRef klog.ObjectRef) []*bundleProcessingWave { // Pre-allocate the map; 7 is the total count of default wave numbers, though // not all wave numbers might be used. diff --git a/pkg/utils/errors/errors.go b/pkg/utils/errors/errors.go index 3f04059e5..ba84bccab 100644 --- a/pkg/utils/errors/errors.go +++ b/pkg/utils/errors/errors.go @@ -152,6 +152,21 @@ func Args(err error, kvs ...interface{}) []interface{} { return compositeKVs } +// Category returns the error category of the given error. It will return ErrCategoryUncategorized +// if the error is nil, is not an *Error with a category, or does not wrap an *Error with a category. +func Category(err error) ErrCategory { + var kferr *Error + // Note: errors.As returns false for nil errors, and might yield a typed nil *Error. + if !errors.As(err, &kferr) || kferr == nil { + return ErrCategoryUncategorized + } + if kferr.category != "" { + return kferr.category + } + // Each recursive call descends further down the error chain, so it always terminates. + return Category(kferr.wrapped) +} + // CallerFrameOverview is a simple struct that summarizes a single call frame. type CallerFrameOverview struct { Function string diff --git a/pkg/utils/errors/errors_test.go b/pkg/utils/errors/errors_test.go index d026b56c8..b372c346f 100644 --- a/pkg/utils/errors/errors_test.go +++ b/pkg/utils/errors/errors_test.go @@ -286,6 +286,108 @@ func TestArgs(t *testing.T) { } } +func TestCategory(t *testing.T) { + var typedNilErr *Error + testCases := []struct { + name string + err error + wantCategory ErrCategory + }{ + { + name: "nil error", + err: nil, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "plain error (not an *Error)", + err: fmt.Errorf("plain error"), + wantCategory: ErrCategoryUncategorized, + }, + { + name: "typed nil *Error", + err: typedNilErr, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with no category set and no wrapped error", + err: &Error{}, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with explicit category", + err: &Error{category: ErrCategoryUser}, + wantCategory: ErrCategoryUser, + }, + { + name: "*Error with explicit category, wrapping an *Error of a different category", + err: &Error{category: ErrCategoryAPIServer, wrapped: &Error{category: ErrCategoryUser}}, + wantCategory: ErrCategoryAPIServer, + }, + { + name: "*Error with no category set, wrapping a plain error", + err: &Error{wrapped: fmt.Errorf("plain error")}, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with no category set, wrapping a typed nil *Error", + err: &Error{wrapped: typedNilErr}, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with no category set, wrapping a categorized *Error", + err: &Error{wrapped: &Error{category: ErrCategoryTransient}}, + wantCategory: ErrCategoryTransient, + }, + { + name: "*Error with no category set, wrapping a categorized *Error through a plain error", + err: &Error{wrapped: fmt.Errorf("plain error: %w", &Error{category: ErrCategoryUnexpected})}, + wantCategory: ErrCategoryUnexpected, + }, + { + name: "*Error with no category set, wrapping multiple *Errors with no category set", + err: &Error{wrapped: &Error{wrapped: &Error{wrapped: &Error{category: ErrCategoryAPIServer}}}}, + wantCategory: ErrCategoryAPIServer, + }, + { + name: "plain error wrapping a typed nil *Error", + err: fmt.Errorf("plain error: %w", typedNilErr), + wantCategory: ErrCategoryUncategorized, + }, + { + name: "plain error wrapping a categorized *Error", + err: fmt.Errorf("plain error: %w", &Error{category: ErrCategoryTransient}), + wantCategory: ErrCategoryTransient, + }, + { + name: "plain error wrapping an *Error with no category set, which wraps a categorized *Error", + err: fmt.Errorf("plain error: %w", &Error{wrapped: &Error{category: ErrCategoryUser}}), + wantCategory: ErrCategoryUser, + }, + { + name: "error created by NewTransientError", + err: NewTransientError(nil, "transient error"), + wantCategory: ErrCategoryTransient, + }, + { + name: "error created by Wraps over a categorized *Error", + err: Wraps(NewAPIServerError(nil, "API server error", false), "high-level description"), + wantCategory: ErrCategoryAPIServer, + }, + { + name: "error created by Wraps over a plain error", + err: Wraps(fmt.Errorf("plain error"), "high-level description"), + wantCategory: ErrCategoryUncategorized, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if gotCategory := Category(tc.err); gotCategory != tc.wantCategory { + t.Errorf("Category(%v) = %v, want %v", tc.err, gotCategory, tc.wantCategory) + } + }) + } +} + func TestUnwrap(t *testing.T) { innerErr := fmt.Errorf("inner error") testCases := []struct { diff --git a/pkg/v1/controllers/workapplier/apply.go b/pkg/v1/controllers/workapplier/apply.go new file mode 100644 index 000000000..026ec6514 --- /dev/null +++ b/pkg/v1/controllers/workapplier/apply.go @@ -0,0 +1,674 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/validation" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/jsonmergepatch" + "k8s.io/apimachinery/pkg/util/mergepatch" + "k8s.io/apimachinery/pkg/util/strategicpatch" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "k8s.io/klog/v2" + "k8s.io/kubectl/pkg/util/deployment" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/utils/resource" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/ownerreferences" +) + +var builtInScheme = runtime.NewScheme() + +func init() { + // This is a trick that allows KubeFleet to check if a resource is a K8s built-in one. + _ = clientgoscheme.AddToScheme(builtInScheme) +} + +// applyInDryRunMode dry-runs an apply op. +func (r *Reconciler) applyInDryRunMode( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, +) (*unstructured.Unstructured, error) { + // In this method, KubeFleet will always use forced server-side apply + // w/o optimistic lock for diff calculation. + // + // This is OK as partial comparison concerns only fields that are currently present + // in the manifest object, and KubeFleet will clear out system managed and read-only fields + // before the comparison. + // + // Note that full comparison can be carried out directly without involving the apply op. + return r.serverSideApply(ctx, gvr, manifestObj, inMemberClusterObj, true, false, true) +} + +func (r *Reconciler) apply( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (*unstructured.Unstructured, error) { + gvr := manifestProcessingState.gvr + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + expectedAppliedWorkOwnerRef := manifestProcessingState.ownedBy + + // Create a sanitized copy of the manifest object. + // + // TO-DO (chenyu1): this processing step can be dropped if the KubeFleet hub agent can guarantee that all manifest + // objects have already been sanitized. + manifestObjCopy := sanitizeManifestObject(manifestProcessingState.manifestObj) + + // Compute the hash of the manifest object. + // + // Originally the manifest hash is kept only if three-way merge patch (client side apply) + // is used; with the new drift detection and takeover capabilities, the manifest hash + // will always be kept regardless of the apply method in use, as it is needed for + // drift detection purposes. + // + // Note that certain fields have been removed from the manifest object in the hash computation + // process. + if err := setLastAppliedManifestHashAnnotation(manifestObjCopy); err != nil { + return nil, errors.Wraps(err, "failed to set manifest hash annotation") + } + + // Validate owner references. + // + // As previously mentioned, with the new capabilities, at this point of the workflow, + // KubeFleet has been added as an owner for the object. Still, to guard against cases where + // co-ownership is turned on then off or the addition of new owner references + // in the manifest, KubeFleet will still perform a validation round. + if err := validateOwnerRefs(manifestProcessingState); err != nil { + return nil, errors.Wraps(err, "failed to validate owner references") + } + + // Add the owner reference information. + ownerreferences.Own(manifestObjCopy, expectedAppliedWorkOwnerRef) + + // If three-way merge patch is used, set the KubeFleet-specific last applied annotation. + // Note that this op might not complete due to the last applied annotation being too large; + // this is not recognized as an error and KubeFleet will switch to server-side apply instead. + isLastAppliedAnnotationSet := false + if syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodClientSideApply { + var err error + isLastAppliedAnnotationSet, err = setLastAppliedConfigAnnotation(manifestObjCopy) + if err != nil { + return nil, errors.Wraps(err, "failed to set last applied configuration annotation") + } + + // Note that KubeFleet might choose to skip the last applied annotation due to size limits + // even if no error has occurred. + klog.V(2).InfoS("Completed the last applied annotation setting process", "isSet", isLastAppliedAnnotationSet, + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + } + + // Create the object if it does not exist in the member cluster. + if inMemberClusterObj == nil { + shouldCreateNS, err := r.shouldCreateNamespaceFor(ctx, manifestObjCopy, syncStrategy) + if err != nil { + return nil, errors.Wraps(err, "failed to determine if the namespace should be created") + } + if shouldCreateNS { + if err := r.createNamespaceFor(ctx, manifestObjCopy); err != nil { + return nil, errors.Wraps(err, "failed to create namespace for the manifest object") + } + } + return r.createManifestObject(ctx, gvr, manifestObjCopy) + } + + // Run the apply op. Note that KubeFleet will always attempt to apply the manifest, even if + // the manifest object hash does not change. + + // Optimistic lock is enabled when the sync strategy dictates that a drift should be reported + // rather than overwritten (i.e., the WhenDrifted field is set to ReportError); this helps + // KubeFleet guard against cases where inadvertent changes are being made in an untimely manner + // (i.e., changes are made when the KubeFleet agent is preparing an apply op). + // + // Note that if the sync strategy dictates that drifts should be overwritten (i.e., + // the WhenDrifted field is set to ApplyAnyway), KubeFleet will not enable optimistic lock. This + // is consistent with the behavior before the drift detection and takeover experience + // is added. + isOptimisticLockEnabled := shouldEnableOptimisticLock(syncStrategy) + + switch { + case syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodClientSideApply && isLastAppliedAnnotationSet: + // The sync strategy dictates that three-way merge patch + // (client-side apply) should be used, and the last applied annotation + // has been set. + klog.V(2).InfoS("Using three-way merge patch to apply the manifest object", + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + return r.threeWayMergePatch(ctx, gvr, manifestObjCopy, inMemberClusterObj, isOptimisticLockEnabled, false) + case syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodClientSideApply: + // The sync strategy dictates that three-way merge patch + // (client-side apply) should be used, but the last applied annotation + // cannot be set. KubeFleet will fall back to server-side apply. + klog.V(2).InfoS("Falling back to server-side apply as the last applied annotation cannot be set", + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + return r.serverSideApply( + ctx, + gvr, manifestObjCopy, inMemberClusterObj, + // When falling back to SSA, always disable force apply ops (this is also the default + // behavior). + // + // Note that the work applier might still enable force apply ops if it finds that + // self-conflicts might occur. + false, isOptimisticLockEnabled, false, + ) + case syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodServerSideApply: + // The sync strategy dictates that server-side apply should be used. + // + // The server-side apply options are not defaulted by the API server when absent. + forceConflicts := syncStrategy.ServerSideApplyOptions != nil && syncStrategy.ServerSideApplyOptions.ForceConflicts + klog.V(2).InfoS("Using server-side apply to apply the manifest object", + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + return r.serverSideApply( + ctx, + gvr, manifestObjCopy, inMemberClusterObj, + forceConflicts, isOptimisticLockEnabled, false, + ) + default: + // An unexpected apply method has been set. Normally this will never run as the built-in + // validation would block invalid values. + return nil, errors.NewUnexpectedError(nil, "an unexpected apply method is found", "applyMethod", syncStrategy.ApplyMethod) + } +} + +// shouldEnableOptimisticLock checks if optimistic lock should be enabled given a sync strategy. +func shouldEnableOptimisticLock(syncStrategy *placementv1alpha1.SyncStrategy) bool { + // Optimistic lock is enabled if drifts are to be reported rather than overwritten. + return syncStrategy.WhenDrifted == placementv1alpha1.WhenDriftedOptionReportError +} + +// sanitizeManifestObject sanitizes the manifest object before applying it. +// +// The sanitization logic here is consistent with that of the CRP controller, sans the API server +// specific parts; see also the generateRawContent function in the respective controller. +// +// Note that this function returns a copy of the manifest object; the original object will be left +// untouched. +func sanitizeManifestObject(manifestObj *unstructured.Unstructured) *unstructured.Unstructured { + // Create a deep copy of the object. + manifestObjCopy := manifestObj.DeepCopy() + + // Remove certain labels and annotations. + if annotations := manifestObjCopy.GetAnnotations(); annotations != nil { + // Remove the two KubeFleet reserved annotations. This is normally not set by users. + delete(annotations, placementv1alpha1.LastAppliedManifestHashAnnotationKey) + delete(annotations, placementv1alpha1.LastAppliedConfigAnnotationKey) + + // Remove the last applied configuration set by kubectl. + delete(annotations, corev1.LastAppliedConfigAnnotation) + + // Remove the revision annotation set by deployment controller. + delete(annotations, deployment.RevisionAnnotation) + + if len(annotations) == 0 { + manifestObjCopy.SetAnnotations(nil) + } else { + manifestObjCopy.SetAnnotations(annotations) + } + } + + // Remove certain system-managed fields. + manifestObjCopy.SetOwnerReferences(nil) + manifestObjCopy.SetManagedFields(nil) + + // Remove the read-only fields. + manifestObjCopy.SetCreationTimestamp(metav1.Time{}) + manifestObjCopy.SetDeletionTimestamp(nil) + manifestObjCopy.SetDeletionGracePeriodSeconds(nil) + manifestObjCopy.SetGeneration(0) + manifestObjCopy.SetResourceVersion("") + manifestObjCopy.SetSelfLink("") + manifestObjCopy.SetUID("") + + // Remove the status field. + unstructured.RemoveNestedField(manifestObjCopy.Object, "status") + + // Note: in the KubeFleet hub agent logic, the system also handles the Service and Job objects + // in a special way, so as to remove certain fields that are set by the hub cluster API + // server automatically; for the KubeFleet member agent logic here, however, KubeFleet assumes + // that if these fields are set, users must have set them on purpose, and they should not + // be removed. The difference comes to the fact that the KubeFleet member agent sanitization + // logic concerns only the enveloped objects, which are free from any hub cluster API + // server manipulation anyway. + + return manifestObjCopy +} + +// setLastAppliedManifestHashAnnotation computes the hash of the provided manifest and sets an annotation of the +// hash on the provided unstructured object. +func setLastAppliedManifestHashAnnotation(manifestObj *unstructured.Unstructured) error { + cleanedManifestObj := discardFieldsIrrelevantInComparisonFrom(manifestObj) + manifestObjHash, err := resource.HashOf(cleanedManifestObj.Object) + if err != nil { + return errors.NewUnexpectedError(err, "failed to compute the hash of the manifest object") + } + + annotations := manifestObj.GetAnnotations() + if annotations == nil { + annotations = map[string]string{} + } + annotations[placementv1alpha1.LastAppliedManifestHashAnnotationKey] = manifestObjHash + manifestObj.SetAnnotations(annotations) + return nil +} + +// validateOwnerRefs validates the owner references of an applied manifest, checking +// if an apply op can be performed on the object. +func validateOwnerRefs( + manifestProcessingState *manifestProcessingState, +) error { + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + expectedAppliedWorkOwnerRef := manifestProcessingState.ownedBy + isCoOwnershipAllowed := syncStrategy.WhenOwnedByOthers == placementv1alpha1.WhenOwnedByOthersOptionShareOwnership + + manifestObjOwnerRefs := manifestProcessingState.manifestObj.GetOwnerReferences() + + // If the manifest object already features some owner reference(s), but co-ownership is + // disallowed, the validation fails. + // + // This is just a sanity check; normally the branch will never get triggered as KubeFleet would + // perform sanitization on the manifest object before applying it, which removes all owner + // references. + if len(manifestObjOwnerRefs) > 0 && !isCoOwnershipAllowed { + return errors.NewUnexpectedError(nil, "manifest is set to have owner references but co-ownership is disallowed") + } + + // Do a sanity check to verify that no AppliedWork object is directly added as an owner + // in the manifest object. Normally the branch will never get triggered as KubeFleet would + // perform sanitization on the manifest object before applying it, which removes all owner + // references. + for idx := range manifestObjOwnerRefs { + ownerRef := &manifestObjOwnerRefs[idx] + if ownerRef.APIVersion == placementv1alpha1.GroupVersion.String() && ownerRef.Kind == placementv1alpha1.AppliedWorkKind { + return errors.NewUnexpectedError(nil, "an appliedWork object is unexpectedly added as an owner in the manifest object") + } + } + + if inMemberClusterObj == nil { + // The manifest object has never been applied yet; no need to do further validation. + return nil + } + inMemberClusterObjOwnerRefs := inMemberClusterObj.GetOwnerReferences() + + // If the live object is co-owned but co-ownership is no longer allowed, the validation fails. + if len(inMemberClusterObjOwnerRefs) > 1 && !isCoOwnershipAllowed { + return errors.NewUserError(nil, "object is co-owned by multiple objects but co-ownership has been disallowed", + "observedOwners", inMemberClusterObjOwnerRefs) + } + + // Note that at this point of execution, one of the owner references is guaranteed to be the + // expected AppliedWork object. For safety reasons, KubeFleet will still do a sanity check. + found := false + for idx := range inMemberClusterObjOwnerRefs { + if ownerreferences.AreEqual(&inMemberClusterObjOwnerRefs[idx], expectedAppliedWorkOwnerRef) { + found = true + break + } + } + if !found { + return errors.NewUnexpectedError(nil, "object is not owned by the expected appliedWork object") + } + + // If the object is already owned by another appliedWork object, the validation fails. + // + // Normally this branch will never get executed as KubeFleet would refuse to take over an object + // that has been owned by another appliedWork object. + if isPlacedByKubeFleetInDuplicate(inMemberClusterObjOwnerRefs, expectedAppliedWorkOwnerRef) { + return errors.NewUnexpectedError(nil, "object is already owned by another appliedWork object") + } + + return nil +} + +// setLastAppliedConfigAnnotation sets the last applied annotation on the provided manifest object. +func setLastAppliedConfigAnnotation(manifestObj *unstructured.Unstructured) (bool, error) { + annotations := manifestObj.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + + // Remove the last applied annotation just in case. + delete(annotations, placementv1alpha1.LastAppliedConfigAnnotationKey) + manifestObj.SetAnnotations(annotations) + + lastAppliedManifestJSONBytes, err := manifestObj.MarshalJSON() + if err != nil { + return false, errors.NewUnexpectedError(err, "failed to marshal the manifest object into JSON") + } + annotations[placementv1alpha1.LastAppliedConfigAnnotationKey] = string(lastAppliedManifestJSONBytes) + isLastAppliedAnnotationSet := true + + if err := validation.ValidateAnnotationsSize(annotations); err != nil { + // If the annotation size exceeds the limit, KubeFleet will set the annotation to an empty string. + annotations[placementv1alpha1.LastAppliedConfigAnnotationKey] = "" + isLastAppliedAnnotationSet = false + } + + manifestObj.SetAnnotations(annotations) + return isLastAppliedAnnotationSet, nil +} + +// shouldCreateNamespaceFor checks if a namespace must be created in the member cluster before the manifest +// object itself can be applied. +func (r *Reconciler) shouldCreateNamespaceFor(ctx context.Context, manifestObj *unstructured.Unstructured, syncStrategy *placementv1alpha1.SyncStrategy) (bool, error) { + if syncStrategy.WhenNamespaceDoesNotExist != placementv1alpha1.WhenNamespaceDoesNotExistOptionCreateNamespace { + // The sync strategy forbids KubeFleet from creating the namespace. + return false, nil + } + + ns := manifestObj.GetNamespace() + if len(ns) == 0 { + // The manifest object is cluster-scoped; no namespace is needed. + return false, nil + } + + err := r.spokeClient.Get(ctx, types.NamespacedName{Name: ns}, &corev1.Namespace{}) + switch { + case err == nil: + return false, nil + case apierrors.IsNotFound(err): + klog.V(2).InfoS("The namespace of the manifest object does not exist in the member cluster", "namespace", ns) + return true, nil + default: + // An unexpected error has occurred. + return false, errors.NewAPIServerError(err, + "failed to check if the namespace of the manifest object exists in the member cluster", true, "namespace", ns) + } +} + +// createNamespaceFor creates the namespace that a manifest object resides in. +// +// The namespace itself is not managed by KubeFleet; it is left behind when the placement is removed. +func (r *Reconciler) createNamespaceFor(ctx context.Context, manifestObj *unstructured.Unstructured) error { + ns := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: manifestObj.GetNamespace(), + }, + } + + err := r.spokeClient.Create(ctx, ns, client.FieldOwner(fieldManagerName)) + switch { + case err == nil: + klog.V(2).InfoS("Created the namespace for the manifest object", + "namespace", ns.Name, "manifestObj", klog.KObj(manifestObj)) + return nil + case apierrors.IsAlreadyExists(err): + // The namespace has been created; this is not considered an error. + klog.V(2).InfoS("The namespace for the manifest object already exists in the member cluster", + "namespace", ns.Name, "manifestObj", klog.KObj(manifestObj)) + return nil + default: + // false as this is a write op. + return errors.NewAPIServerError(err, "failed to create the namespace for the manifest object", false, + "namespace", ns.Name) + } +} + +// createManifestObject creates the manifest object in the member cluster. +func (r *Reconciler) createManifestObject( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObject *unstructured.Unstructured, +) (*unstructured.Unstructured, error) { + createOpts := metav1.CreateOptions{ + FieldManager: fieldManagerName, + } + createdObj, err := r.spokeDynamicClient.Resource(*gvr).Namespace(manifestObject.GetNamespace()).Create(ctx, manifestObject, createOpts) + if err != nil { + // false as the dynamic client is non-caching. + return nil, errors.NewAPIServerError(err, "failed to create the manifest object", false) + } + klog.V(2).InfoS("Created the manifest object", "GVR", *gvr, "manifestObj", klog.KObj(createdObj)) + return createdObj, nil +} + +// threeWayMergePatch uses three-way merge patch to apply the manifest object. +func (r *Reconciler) threeWayMergePatch( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, + optimisticLock, dryRun bool, +) (*unstructured.Unstructured, error) { + // Enable optimistic lock by forcing the resource version field to be added to the + // JSON merge patch. Optimistic lock is always enabled in the dry run mode. + if optimisticLock || dryRun { + curResourceVer := inMemberClusterObj.GetResourceVersion() + if len(curResourceVer) == 0 { + return nil, errors.NewUnexpectedError(nil, "failed to enable optimistic lock: resource version is empty on the object from the member cluster", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + } + + // Add the resource version to the manifest object. + manifestObj.SetResourceVersion(curResourceVer) + + // Remove the resource version from the object in the member cluster. + inMemberClusterObj.SetResourceVersion("") + } + + // Create a three-way merge patch. + patch, err := buildThreeWayMergePatch(manifestObj, inMemberClusterObj) + if err != nil { + return nil, errors.Wraps(err, "failed to create three-way merge patch") + } + data, err := patch.Data(manifestObj) + if err != nil { + // KubeFleet uses raw patch; this branch should never run. + return nil, errors.NewUnexpectedError(err, "failed to get patch data") + } + + // Use three-way merge (similar to kubectl client side apply) to patch the object in the + // member cluster. + // + // This will: + // * Remove fields that are present in the last applied configuration but not in the + // manifest object. + // * Create fields that are present in the manifest object but not in the object from the member cluster. + // * Update fields that are present in both the manifest object and the object from the member cluster. + patchOpts := metav1.PatchOptions{ + FieldManager: fieldManagerName, + } + if dryRun { + patchOpts.DryRun = []string{metav1.DryRunAll} + } + patchedObj, err := r.spokeDynamicClient. + Resource(*gvr).Namespace(manifestObj.GetNamespace()). + Patch(ctx, manifestObj.GetName(), patch.Type(), data, patchOpts) + if err != nil { + // false as the dynamic client is non-caching. + return nil, errors.NewAPIServerError(err, "failed to patch the manifest object", false) + } + return patchedObj, nil +} + +// threeWayMergePatch creates a patch by computing a three-way diff based on +// the manifest object, the live object, and the last applied configuration as kept in +// the annotations. +func buildThreeWayMergePatch(manifestObj, liveObj *unstructured.Unstructured) (client.Patch, error) { + // Marshal the manifest object into JSON bytes. + manifestObjJSONBytes, err := manifestObj.MarshalJSON() + if err != nil { + return nil, err + } + // Marshal the live object into JSON bytes. + liveObjJSONBytes, err := liveObj.MarshalJSON() + if err != nil { + return nil, err + } + // Retrieve the last applied configuration from the annotations. This can be an empty string. + lastAppliedObjJSONBytes := getLastAppliedConfigAnnotation(liveObj) + + var patchType types.PatchType + var patchData []byte + var lookupPatchMeta strategicpatch.LookupPatchMeta + + versionedObject, err := builtInScheme.New(liveObj.GetObjectKind().GroupVersionKind()) + switch { + case runtime.IsNotRegisteredError(err): + // use JSONMergePatch for custom resources + // because StrategicMergePatch doesn't support custom resources + patchType = types.MergePatchType + preconditions := []mergepatch.PreconditionFunc{ + mergepatch.RequireKeyUnchanged("apiVersion"), + mergepatch.RequireKeyUnchanged("kind"), + mergepatch.RequireMetadataKeyUnchanged("name"), + } + patchData, err = jsonmergepatch.CreateThreeWayJSONMergePatch( + lastAppliedObjJSONBytes, manifestObjJSONBytes, liveObjJSONBytes, preconditions...) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to create three-way JSON merge patch") + } + case err != nil: + return nil, err + default: + // use StrategicMergePatch for K8s built-in resources + patchType = types.StrategicMergePatchType + lookupPatchMeta, err = strategicpatch.NewPatchMetaFromStruct(versionedObject) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to create patch meta from struct (strategic merge patch)") + } + patchData, err = strategicpatch.CreateThreeWayMergePatch(lastAppliedObjJSONBytes, manifestObjJSONBytes, liveObjJSONBytes, lookupPatchMeta, true) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to create three-way strategic merge patch") + } + } + return client.RawPatch(patchType, patchData), nil +} + +// getLastAppliedConfigAnnotation returns the last applied annotation of a manifest object. +func getLastAppliedConfigAnnotation(inMemberClusterObj *unstructured.Unstructured) []byte { + annotations := inMemberClusterObj.GetAnnotations() + if annotations == nil { + // The last applied annotation is not found in the live object; normally this should not + // happen, but KubeFleet can still handle this situation. + klog.V(2).InfoS("No annotations in the live object", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return nil + } + + lastAppliedManifestJSONStr, found := annotations[placementv1alpha1.LastAppliedConfigAnnotationKey] + if !found { + // The last applied annotation is not found in the live object; normally this should not + // happen, but KubeFleet can still handle this situation. + klog.V(2).InfoS("The last applied annotation is not found in the live object", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return nil + } + + return []byte(lastAppliedManifestJSONStr) +} + +// serverSideApply uses server-side apply to apply the manifest object. +func (r *Reconciler) serverSideApply( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, + force, optimisticLock, dryRun bool, +) (*unstructured.Unstructured, error) { + // Enable optimistic lock by forcing the resource version field to be added to the + // JSON merge patch. Optimistic lock is always disabled in the dry run mode. + if optimisticLock && !dryRun { + curResourceVer := inMemberClusterObj.GetResourceVersion() + if len(curResourceVer) == 0 { + return nil, errors.NewUnexpectedError(nil, "failed to enable optimistic lock: resource version is empty on the object from the member cluster", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + } + + // Add the resource version to the manifest object. + manifestObj.SetResourceVersion(curResourceVer) + + // Remove the resource version from the object in the member cluster. + inMemberClusterObj.SetResourceVersion("") + } + + // Check if forced server-side apply is needed even if it is not turned on by the user. + // + // Note (chenyu1): This is added to addresses cases where Kubernetes might register + // KubeFleet (the member agent) as an Update typed field manager for the object, which blocks + // the same agent itself from performing a server-side apply due to conflicts, + // as Kubernetes considers Update typed and Apply typed field managers to be different + // entities, despite having the same identifier. In these cases, users will see their + // first apply attempt being successful, yet any subsequent update would fail due to + // conflicts. There are also a few other similar cases that are solved by this check; + // see the inner comments for specifics. + if shouldUseForcedServerSideApply(inMemberClusterObj) { + force = true + } + + // Use server-side apply to apply the manifest object. + // + // See the Kubernetes documentation on structured merged diff for the exact behaviors. + applyOpts := metav1.ApplyOptions{ + FieldManager: fieldManagerName, + Force: force, + } + if dryRun { + applyOpts.DryRun = []string{metav1.DryRunAll} + } + appliedObj, err := r.spokeDynamicClient. + Resource(*gvr).Namespace(manifestObj.GetNamespace()). + Apply(ctx, manifestObj.GetName(), manifestObj, applyOpts) + if err != nil { + return nil, errors.NewAPIServerError(err, "failed to apply the manifest object: an error is returned by the API server", false) + } + return appliedObj, nil +} + +// shouldUseForcedServerSideApply checks if forced server-side apply should be used even if +// the force option is not turned on. +func shouldUseForcedServerSideApply(inMemberClusterObj *unstructured.Unstructured) bool { + managedFields := inMemberClusterObj.GetManagedFields() + for idx := range managedFields { + mf := &managedFields[idx] + // fieldManagerName is the field manager name used by KubeFleet; its presence + // suggests that some (not necessarily all) fields are managed by KubeFleet. + // + // `before-first-apply` is a field manager name used by Kubernetes to "properly" + // track field managers between non-apply and apply ops. Specifically, this + // manager is added when an object is being applied, but Kubernetes finds + // that the object does not have any managed field specified. + // + // Note (chenyu1): unfortunately this name is not exposed as a public variable. See + // the Kubernetes source code for more information. + if mf.Manager != fieldManagerName && mf.Manager != "before-first-apply" { + // There exists a field manager this is neither KubeFleet nor the `before-first-apply` + // field manager, which suggests that the object (or at least some of its fields) + // is managed by another entity. KubeFleet will not enable forced server-side apply in + // this case and let user decide if forced apply is needed. + klog.V(2).InfoS("Found a field manager that is neither KubeFleet nor the `before-first-apply` field manager; KubeFleet will not enable forced server-side apply unless explicitly requested", + "fieldManager", mf.Manager, + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return false + } + } + + // All field managers are either KubeFleet or the `before-first-apply` field manager; + // use forced server-side apply to avoid confusing self-conflicts. This would + // allow KubeFleet to (correctly) assume ownership of managed fields. + klog.V(2).InfoS("All field managers are either KubeFleet or the `before-first-apply` field manager; KubeFleet will enable forced server-side apply", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return true +} diff --git a/pkg/v1/controllers/workapplier/availability_tracker.go b/pkg/v1/controllers/workapplier/availability_tracker.go new file mode 100644 index 000000000..659876292 --- /dev/null +++ b/pkg/v1/controllers/workapplier/availability_tracker.go @@ -0,0 +1,328 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + appv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + policyv1 "k8s.io/api/policy/v1" + apiextensionshelpers "k8s.io/apiextensions-apiserver/pkg/apihelpers" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/component-helpers/apps/poddisruptionbudget" + "k8s.io/klog/v2" + + "github.com/kubefleet-dev/kubefleet/pkg/utils" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +// trackInMemberClusterObjAvailability tracks the availability of applied objects in the member cluster. +func (r *Reconciler) trackInMemberClusterObjAvailability(ctx context.Context, manifestProcessingStates []*manifestProcessingState) error { + // Track the availability of all the applied objects in the member cluster in parallel. + // + // This is concurrency-safe as the processing state slice has been pre-allocated. + + // Prepare a child context. + // Cancel the child context anyway to avoid leaks. + childCtx, cancel := context.WithCancel(ctx) + defer cancel() + + doWork := func(pieces int) { + state := manifestProcessingStates[pieces] + ownerWorkObj := state.fromWorkObj + primaryWorkObj := state.fromPrimaryWorkObject + + if !isManifestObjectApplied(state.applyRes) { + // The manifest object has not been applied yet. No availability check is needed. + state.availabilityCheckRes = AvailabilityResultTypeSkipped + + // Note that some of the objects might have failed the pre-processing stage and do not + // even have a GVR or a manifest object. + if state.gvr != nil && state.manifestObj != nil { + klog.V(2).InfoS("The manifest object is not applied yet, skipping the availability check", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "inMemberClusterObj", klog.KObj(state.inMemberClusterObj), + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj)) + } else { + klog.V(2).InfoS("The manifest object is not applied yet, skipping the availability check", + "ordinal", pieces, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj)) + } + return + } + + availabilityCheckRes, err := trackInMemberClusterObjAvailabilityByGVR(state.gvr, state.inMemberClusterObj) + if err != nil { + // An unexpected error has occurred during the availability check. + state.availabilityCheckErr = err + state.availabilityCheckRes = AvailabilityResultTypeFailed + klog.ErrorS(err, "Failed to track the availability of the applied object in the member cluster", + errors.Args(err, + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj))...) + return + } + state.availabilityCheckRes = availabilityCheckRes + klog.V(2).InfoS("Tracked availability of a resource", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj), + "availabilityCheckRes", availabilityCheckRes) + } + + // Run the availability check in parallel. + r.parallelizer.ParallelizeUntil(childCtx, len(manifestProcessingStates), doWork, "trackInMemberClusterObjAvailability") + + // Unlike some other steps in the reconciliation loop, the availability checking step does not end + // with a contextual API call; consequently, if the context has been cancelled during this step, + // some checks might not run at all, and passing such states to the next step may trigger + // unexpected behaviors. To address this, at the end of this step the work applier checks for context + // cancellation directly. + if err := ctx.Err(); err != nil { + klog.V(2).InfoS("Availability checking has been interrupted as the main context has been cancelled") + return errors.NewTransientError(err, "availability checking has been interrupted") + } + return nil +} + +// isManifestObjectApplied returns whether an apply result type indicates that the manifest object +// has been applied. +func isManifestObjectApplied(applyRes ApplyResultType) bool { + return applyRes == ApplyResTypeApplied || applyRes == ApplyResTypeAppliedWithFailedDriftDetection +} + +// trackInMemberClusterObjAvailabilityByGVR tracks the availability of an object in the member cluster based +// on its GVR. +func trackInMemberClusterObjAvailabilityByGVR( + gvr *schema.GroupVersionResource, + inMemberClusterObj *unstructured.Unstructured, +) (AvailabilityCheckResultType, error) { + switch *gvr { + case utils.DeploymentGVR: + return trackDeploymentAvailability(inMemberClusterObj) + case utils.StatefulSetGVR: + return trackStatefulSetAvailability(inMemberClusterObj) + case utils.DaemonSetGVR: + return trackDaemonSetAvailability(inMemberClusterObj) + case utils.ServiceGVR: + return trackServiceAvailability(inMemberClusterObj) + case utils.CustomResourceDefinitionGVR: + return trackCRDAvailability(inMemberClusterObj) + case utils.PodDisruptionBudgetGVR: + return trackPDBAvailability(inMemberClusterObj) + default: + if isDataResource(*gvr) { + klog.V(2).InfoS("The object from the member cluster is a data object, consider it to be immediately available", + "GVR", *gvr, "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Cannot determine the availability of the object from the member cluster; untrack its availability", + "GVR", *gvr, "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotTrackable, nil + } +} + +// trackDeploymentAvailability tracks the availability of a deployment in the member cluster. +func trackDeploymentAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var deploy appv1.Deployment + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &deploy); err != nil { + // Normally this branch should never run. + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a deployment") + } + + // Check if the deployment is available. + requiredReplicas := int32(1) + if deploy.Spec.Replicas != nil { + requiredReplicas = *deploy.Spec.Replicas + } + if deploy.Status.ObservedGeneration == deploy.Generation && + requiredReplicas == deploy.Status.AvailableReplicas && + requiredReplicas == deploy.Status.UpdatedReplicas && + deploy.Status.UnavailableReplicas == 0 { + klog.V(2).InfoS("Deployment is available", "deployment", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Deployment is not ready yet, will check later to see if it becomes available", "deployment", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackStatefulSetAvailability tracks the availability of a stateful set in the member cluster. +func trackStatefulSetAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var statefulSet appv1.StatefulSet + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &statefulSet); err != nil { + // Normally this branch should never run. + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a stateful set") + } + + // Check if the stateful set is available. + // + // A statefulSet is available if all if its replicas are available and the current replica count + // is equal to the updated replica count, which implies that all replicas are up to date. + requiredReplicas := int32(1) + if statefulSet.Spec.Replicas != nil { + requiredReplicas = *statefulSet.Spec.Replicas + } + if statefulSet.Status.ObservedGeneration == statefulSet.Generation && + statefulSet.Status.AvailableReplicas == requiredReplicas && + statefulSet.Status.CurrentReplicas == statefulSet.Status.UpdatedReplicas && + statefulSet.Status.CurrentRevision == statefulSet.Status.UpdateRevision { + klog.V(2).InfoS("StatefulSet is available", "statefulSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Stateful set is not ready yet, will check later to see if it becomes available", "statefulSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackDaemonSetAvailability tracks the availability of a daemon set in the member cluster. +func trackDaemonSetAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var daemonSet appv1.DaemonSet + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &daemonSet); err != nil { + // Normally this branch should never run. + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a daemon set") + } + + // Check if the daemonSet is available. + // + // A daemonSet is available if all if its desired replicas (the count of which is equal to + // the number of applicable nodes in the cluster) are available and the current replica count + // is equal to the updated replica count, which implies that all replicas are up to date. + if daemonSet.Status.ObservedGeneration == daemonSet.Generation && + daemonSet.Status.NumberAvailable == daemonSet.Status.DesiredNumberScheduled && + daemonSet.Status.CurrentNumberScheduled == daemonSet.Status.UpdatedNumberScheduled { + klog.V(2).InfoS("DaemonSet is available", "daemonSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Daemon set is not ready yet, will check later to see if it becomes available", "daemonSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackServiceAvailability tracks the availability of a service in the member cluster. +func trackServiceAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var svc corev1.Service + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &svc); err != nil { + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a service") + } + switch svc.Spec.Type { + case "": + fallthrough // The default service type is ClusterIP. + case corev1.ServiceTypeClusterIP: + fallthrough + case corev1.ServiceTypeNodePort: + // KubeFleet considers a ClusterIP or NodePort service to be available if it has at least one + // IP assigned. + if len(svc.Spec.ClusterIPs) > 0 && len(svc.Spec.ClusterIPs[0]) > 0 { + klog.V(2).InfoS("Service is available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Service is not ready yet, will check later to see if it becomes available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeNotYetAvailable, nil + case corev1.ServiceTypeLoadBalancer: + // KubeFleet considers a loadBalancer service to be available if it has at least one load + // balancer IP or hostname assigned. + if len(svc.Status.LoadBalancer.Ingress) > 0 && + (len(svc.Status.LoadBalancer.Ingress[0].IP) > 0 || len(svc.Status.LoadBalancer.Ingress[0].Hostname) > 0) { + klog.V(2).InfoS("Service is available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Service is not ready yet, will check later to see if it becomes available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeNotYetAvailable, nil + } + + // KubeFleet does not know how to track the availability of an externalName service. + klog.V(2).InfoS("Cannot determine the availability of external name services; untrack its availability", "service", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotTrackable, nil +} + +// trackCRDAvailability tracks the availability of a custom resource definition in the member cluster. +func trackCRDAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var crd apiextensionsv1.CustomResourceDefinition + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &crd); err != nil { + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a custom resource definition") + } + + // If both conditions are True, the CRD has become available. + if apiextensionshelpers.IsCRDConditionTrue(&crd, apiextensionsv1.Established) && apiextensionshelpers.IsCRDConditionTrue(&crd, apiextensionsv1.NamesAccepted) { + klog.V(2).InfoS("CustomResourceDefinition is available", "customResourceDefinition", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + + klog.V(2).InfoS("Custom resource definition is not ready yet, will check later to see if it becomes available", + "customResourceDefinition", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackPDBAvailability tracks the availability of a pod disruption budget in the member cluster. +func trackPDBAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var pdb policyv1.PodDisruptionBudget + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &pdb); err != nil { + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a pod disruption budget") + } + // Check if conditions are up-to-date. + if poddisruptionbudget.ConditionsAreUpToDate(&pdb) { + klog.V(2).InfoS("PodDisruptionBudget is available", "podDisruptionBudget", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Still need to wait for PodDisruptionBudget to be available", "podDisruptionBudget", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// isDataResource checks if the resource is a data resource; such resources are +// available immediately after creation. +func isDataResource(gvr schema.GroupVersionResource) bool { + switch gvr { + case utils.NamespaceGVR: + return true + case utils.SecretGVR: + return true + case utils.ConfigMapGVR: + return true + case utils.RoleGVR: + return true + case utils.ClusterRoleGVR: + return true + case utils.RoleBindingGVR: + return true + case utils.ClusterRoleBindingGVR: + return true + case utils.ServiceAccountGVR: + return true + case utils.NetworkPolicyGVR: + return true + case utils.CSIDriverGVR: + return true + case utils.CSINodeGVR: + return true + case utils.StorageClassGVR: + return true + case utils.CSIStorageCapacityGVR: + return true + case utils.ControllerRevisionGVR: + return true + case utils.IngressClassGVR: + return true + case utils.LimitRangeGVR: + return true + case utils.ResourceQuotaGVR: + return true + case utils.PriorityClassGVR: + return true + } + return false +} diff --git a/pkg/v1/controllers/workapplier/cleanup.go b/pkg/v1/controllers/workapplier/cleanup.go new file mode 100644 index 000000000..079e5b306 --- /dev/null +++ b/pkg/v1/controllers/workapplier/cleanup.go @@ -0,0 +1,316 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/fieldindexers" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/ownerreferences" +) + +// cleanupWhenBindingDeleted performs cleanup ops when the placement binding that spawns all the linked +// work objects has been deleted, based on the given SyncStrategy. +func (r *Reconciler) cleanupWhenBindingDeleted(ctx context.Context, + primaryWork *placementv1alpha1.Work) (*time.Duration, error) { + primaryWorkCopy := primaryWork.DeepCopy() + setDefaultSyncStrategy(primaryWorkCopy) + + deletionPolicy := metav1.DeletePropagationForeground + if primaryWorkCopy.Spec.SyncStrategy.WhenPlacementDeleted == placementv1alpha1.WhenPlacementDeletedOptionOrphanResources { + deletionPolicy = metav1.DeletePropagationOrphan + } + + // Retrieve all works that derive from the same placement policy. + // + // Note that this list comes from the cache; if the cache has become stale, and there exists a work object + // that is owned by the placement policy but not yet reflected in the cache, it is guaranteed that the work applier + // has never processed it before; as a result, the work object should have no finalizer and needs no cleanup. + ownerNS := primaryWork.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerName := primaryWork.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if ownerName == "" { + return nil, errors.NewUnexpectedError(nil, "no owner placement binding found for the primary work object", "observedOwnerNS", ownerNS) + } + ownedByFieldVal := fmt.Sprintf(fieldindexers.WorkOwnedByPlacementBindingCustomFieldValFmt, ownerNS, ownerName) + workList := &placementv1alpha1.WorkList{} + if err := r.hubClient.List(ctx, workList, + client.InNamespace(primaryWork.Namespace), + client.MatchingFields{fieldindexers.WorkOwnedByPlacementBindingCustomFieldName: ownedByFieldVal}, + ); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list the work objects owned by the placement policy", true, + "ownerPlacementPolicy", ownedByFieldVal) + } + works := workList.Items + + // All the manifests across the linked work objects are owned by the appliedWork object of the primary work. + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: primaryWork.Name}, + } + err := r.spokeClient.Get(ctx, client.ObjectKey{Name: primaryWork.Name}, appliedWork) + switch { + case apierrors.IsNotFound(err): + // The primary appliedWork object is absent; no manifest cleanup is needed. + klog.V(2).InfoS("The primary appliedWork object is not found; no manifest cleanup is needed", "appliedWork", klog.KObj(appliedWork)) + case err != nil: + // An unexpected error has occurred. + return nil, errors.NewAPIServerError(err, "failed to retrieve the appliedWork object", true, "appliedWork", klog.KObj(appliedWork)) + case appliedWork.DeletionTimestamp.IsZero(): + // The appliedWork object exists and has not been marked for deletion yet; delete it. + if err := r.spokeClient.Delete(ctx, appliedWork, &client.DeleteOptions{PropagationPolicy: &deletionPolicy}); err != nil && !apierrors.IsNotFound(err) { + // An unexpected error has occurred. + return nil, errors.NewAPIServerError(err, "failed to delete the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork), "propagationPolicy", deletionPolicy) + } + klog.V(2).InfoS("The primary appliedWork object has been marked for deletion; wait for the deletion to complete", + "appliedWork", klog.KObj(appliedWork), "propagationPolicy", deletionPolicy) + return &r.cleanupRequeueAfter, nil + default: + // The appliedWork object has been marked for deletion; wait for the deletion to complete. + if time.Since(appliedWork.DeletionTimestamp.Time) < r.cleanupWaitTime { + // Keep waiting. + return &r.cleanupRequeueAfter, nil + } + + if _, found := appliedWork.Annotations[appliedWorkForcedDeletedAnnotationKey]; !found { + klog.V(2).InfoS("The primary appliedWork object has been stuck in the pending deletion state; force-remove KubeFleet owner reference from applied manifests as a last resort", + "appliedWork", klog.KObj(appliedWork)) + // The appliedWork object has been stuck in the pending deletion state for longer than the cleanup wait time; + // manually update owner references from all the applied manifests so that they no longer block the deletion + // of the appliedWork object. + if err := r.manuallyUpdateOwnerReferencesOnManifests(ctx, appliedWork, works); err != nil { + return nil, errors.Wraps(err, "failed to manually update owner references from manifests to unblock cleanup") + } + + // Add the force-deleted annotation to the appliedWork object. + appliedWorkPatch := client.MergeFrom(appliedWork.DeepCopy()) + annotations := appliedWork.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + annotations[appliedWorkForcedDeletedAnnotationKey] = "true" + appliedWork.SetAnnotations(annotations) + if err := r.spokeClient.Patch(ctx, appliedWork, appliedWorkPatch); err != nil && !apierrors.IsNotFound(err) { + return nil, errors.NewAPIServerError(err, "failed to add the force-deleted annotation to the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork)) + } + } + + if time.Since(appliedWork.DeletionTimestamp.Time) < 2*r.cleanupWaitTime { + // Keep waiting. + return &r.cleanupRequeueAfter, nil + } + + // If the appliedWork object is still stuck in the pending deletion state after us manually removing the + // owner references from all the applied manifests, the work applier will NOT keep + // waiting for the appliedWork object to disappear. Users should keep an eye on the GC progress (the + // API server/GC controller might not be fast enough) themselves, and manual intervention might be needed + // if the situation persists. + klog.Warning("The appliedWork object has been stuck in the pending deletion state for too long, and KubeFleet will proceed with the cleanup without further waiting; manual intervention might be needed.") + } + + // Remove the cleanup finalizer from non-primary work objects first. + for idx := range works { + work := &works[idx] + if work.Name == primaryWork.Name || !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + continue + } + + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: work.Name}, + } + // No need to specify the propagation policy; these appliedWork objects do not own any child resources. + if err := r.spokeClient.Delete(ctx, appliedWork); err != nil && !apierrors.IsNotFound(err) { + return nil, errors.NewAPIServerError(err, "failed to delete the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + } + klog.V(2).InfoS("Cleaned up the appliedWork object", "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + + controllerutil.RemoveFinalizer(work, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, work); err != nil { + return nil, errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the work", false, + "work", klog.KObj(work)) + } + klog.V(2).InfoS("Removed the cleanup finalizer from the work", "work", klog.KObj(work)) + } + + // Remove the cleanup finalizer from the primary work object. + if controllerutil.ContainsFinalizer(primaryWork, workApplierCleanupFinalizer) { + controllerutil.RemoveFinalizer(primaryWork, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, primaryWork); err != nil { + return nil, errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the primary work", false, + "work", klog.KObj(primaryWork)) + } + klog.V(2).InfoS("Removed the cleanup finalizer from the primary work", "work", klog.KObj(primaryWork)) + } + + return nil, nil +} + +func (r *Reconciler) manuallyUpdateOwnerReferencesOnManifests(ctx context.Context, + appliedWork *placementv1alpha1.AppliedWork, + works []placementv1alpha1.Work) error { + appliedWorkOwnerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "AppliedWork", + Name: appliedWork.Name, + UID: appliedWork.UID, + } + + // Loop through all work objects, identify all applied (or pre-processed) manifests + // in the status, and add them as owner reference removal candidates. + manifestsToRemoveOwnerRef := []placementv1alpha1.ManifestIdentifier{} + for workIdx := range works { + work := &works[workIdx] + for manifestIdx := range work.Status.Manifests { + manifestStatus := &work.Status.Manifests[manifestIdx] + appliedCond := meta.FindStatusCondition(manifestStatus.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond == nil || (appliedCond.Status != metav1.ConditionTrue && appliedCond.Reason != placementv1alpha1.WorkAppliedCondPreparingToProcessReason) { + continue + } + + manifestsToRemoveOwnerRef = append(manifestsToRemoveOwnerRef, manifestStatus.Identifier) + } + } + + // Update the owner references of all the identified manifests so that they no longer block appliedWork object's + // deletion. + errs := make([]error, len(manifestsToRemoveOwnerRef)) + removeOwnerRefFromManifest := func(piece int) { + identifier := manifestsToRemoveOwnerRef[piece] + gvr := schema.GroupVersionResource{ + Group: identifier.APIGroup, + Version: identifier.APIVersion, + Resource: identifier.Resource, + } + manifestRef := klog.KRef(identifier.Namespace, identifier.Name) + manifestObj, err := r.spokeDynamicClient.Resource(gvr).Namespace(identifier.Namespace).Get(ctx, identifier.Name, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + return + } + errs[piece] = errors.NewAPIServerError(err, "failed to retrieve a manifest while unblocking appliedWork deletion", false, + "gvr", gvr, "manifestObj", manifestRef, "appliedWork", klog.KObj(appliedWork)) + return + } + + ownerRefs := manifestObj.GetOwnerReferences() + if len(ownerRefs) == 0 { + // The applied manifest is no longer owned by KubeFleet; no further action is needed. + return + } + updated := false + for ownerIdx := range ownerRefs { + if ownerreferences.AreEqual(&ownerRefs[ownerIdx], appliedWorkOwnerRef) && ptr.Deref(ownerRefs[ownerIdx].BlockOwnerDeletion, false) { + // Set BlockOwnerDeletion to false to allow the appliedWork object to be deleted without being blocked + // by this manifest. + ownerRefs[ownerIdx].BlockOwnerDeletion = ptr.To(false) + updated = true + } + } + if !updated { + return + } + + manifestObj.SetOwnerReferences(ownerRefs) + if _, err := r.spokeDynamicClient.Resource(gvr).Namespace(identifier.Namespace).Update(ctx, manifestObj, metav1.UpdateOptions{}); err != nil && !apierrors.IsNotFound(err) { + errs[piece] = errors.NewAPIServerError(err, "failed to unblock a manifest owner reference", false, + "gvr", gvr, "manifestObj", manifestRef, "appliedWork", klog.KObj(appliedWork)) + } + } + r.parallelizer.ParallelizeUntil(ctx, len(manifestsToRemoveOwnerRef), removeOwnerRefFromManifest, "manuallyRemoveOwnerReferencesFromManifests") + + if aggregatedErrs := utilerrors.NewAggregate(errs); aggregatedErrs != nil { + return errors.Wraps(nil, "failed to update owner references on manifests", + "appliedWork", klog.KObj(appliedWork), "errs", aggregatedErrs) + } + return nil +} + +func (r *Reconciler) cleanupLeftOverWorks(ctx context.Context, + leftOverWorks []*placementv1alpha1.Work, seenManifIDs sets.Set[string], appliedWorkOwnerRef *metav1.OwnerReference) error { + for idx := range leftOverWorks { + leftOverWork := leftOverWorks[idx] + if !controllerutil.ContainsFinalizer(leftOverWork, workApplierCleanupFinalizer) { + // The left-over work does not have the cleanup finalizer; no cleanup is needed. + continue + } + + // Identify manifests that are no longer seen and need to be removed. + manifestsToDelete := make([]placementv1alpha1.ManifestIdentifier, 0) + for manifestIdx := range leftOverWork.Status.Manifests { + manifestStatus := &leftOverWork.Status.Manifests[manifestIdx] + if seenManifIDs.Has(formatManifestIdentifierStr(&manifestStatus.Identifier)) { + // There exists a corner case where a manifest, though present on a left-over work, is still present + // on one of the linked work objects; in this case, such manifests would not be removed. + continue + } + + // Check if the manifest has been applied (or pre-processed) before; if so, attempt to remove it. + appliedCond := meta.FindStatusCondition(manifestStatus.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond != nil && (appliedCond.Status == metav1.ConditionTrue || appliedCond.Reason == placementv1alpha1.WorkAppliedCondPreparingToProcessReason) { + manifestsToDelete = append(manifestsToDelete, manifestStatus.Identifier) + } + } + + errs := make([]error, len(manifestsToDelete)) + removeManifest := func(piece int) { + manifest := manifestsToDelete[piece] + if err := r.removeOneLeftOverManifest(ctx, manifest, appliedWorkOwnerRef); err != nil { + errs[piece] = errors.Wraps(err, "failed to remove a left-over manifest", "manifestId", manifest, "work", klog.KObj(leftOverWork)) + } + } + r.parallelizer.ParallelizeUntil(ctx, len(manifestsToDelete), removeManifest, "cleanupLeftOverWorks") + aggregatedErrs := utilerrors.NewAggregate(errs) + if aggregatedErrs != nil { + return errors.Wraps(nil, "failed to remove manifests on a left-over work", "work", klog.KObj(leftOverWork), "errs", aggregatedErrs) + } + + // Clean up the appliedWork object associated with the left-over work. + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: leftOverWork.Name}, + } + if err := r.spokeClient.Delete(ctx, appliedWork); err != nil && !apierrors.IsNotFound(err) { + return errors.NewAPIServerError(err, "failed to delete the appliedWork object associated with the left-over work", false, + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(leftOverWork)) + } + klog.V(2).InfoS("Cleaned up the appliedWork object associated with the left-over work", + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(leftOverWork)) + + controllerutil.RemoveFinalizer(leftOverWork, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, leftOverWork); err != nil { + return errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the left-over work", false, + "work", klog.KObj(leftOverWork)) + } + klog.V(2).InfoS("Cleaned up a left-over work", "work", klog.KObj(leftOverWork)) + } + return nil +} diff --git a/pkg/v1/controllers/workapplier/controller.go b/pkg/v1/controllers/workapplier/controller.go new file mode 100644 index 000000000..281992844 --- /dev/null +++ b/pkg/v1/controllers/workapplier/controller.go @@ -0,0 +1,374 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "sync/atomic" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/client-go/dynamic" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + ctrloption "sigs.k8s.io/controller-runtime/pkg/controller" + "sigs.k8s.io/controller-runtime/pkg/predicate" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + parallelizerutil "github.com/kubefleet-dev/kubefleet/pkg/utils/parallelizer" +) + +const ( + controllerName = "work-applier" + fieldManagerName = "kubefleet-member-agent" + + workApplierCleanupFinalizer = "placement.kubefleet.dev/work-cleanup" + + appliedWorkForcedDeletedAnnotationKey = "placement.kubefleet.dev/applied-work-forced-deleted" +) + +type ApplyResultType string + +const ( + // The result types for apply op failures. + ApplyResTypeDecodingErred ApplyResultType = "DecodingErred" + ApplyResTypeFoundGenerateName ApplyResultType = "FoundGenerateName" + ApplyResTypeDuplicated ApplyResultType = "Duplicated" + ApplyResTypeFailedToFindObjInMemberCluster ApplyResultType = "FailedToFindObjInMemberCluster" + ApplyResTypeOwnedByOtherKubeFleetAPIObjects ApplyResultType = "OwnedByOtherKubeFleetAPIObjects" + ApplyResTypeFailedToTakeOver ApplyResultType = "FailedToTakeOver" + ApplyResTypeNotTakenOver ApplyResultType = "NotTakenOver" + ApplyResTypeFailedToRunDriftDetection ApplyResultType = "FailedToRunDriftDetection" + ApplyResTypeFoundDrifts ApplyResultType = "FoundDrifts" + ApplyResTypeFoundDriftsInDegradedMode ApplyResultType = "FoundDriftsInDegradedMode" + ApplyResTypeFailedToApply ApplyResultType = "FailedToApply" + + // The result type and description for successful apply ops. + ApplyResTypeApplied ApplyResultType = "Applied" + // The apply op has succeeded, but KubeFleet cannot tell whether the object has drifted. + ApplyResTypeAppliedWithFailedDriftDetection ApplyResultType = "AppliedWithFailedDriftDetection" +) + +var allApplyResTypes = sets.New( + ApplyResTypeDecodingErred, + ApplyResTypeFoundGenerateName, + ApplyResTypeDuplicated, + ApplyResTypeFailedToFindObjInMemberCluster, + ApplyResTypeOwnedByOtherKubeFleetAPIObjects, + ApplyResTypeFailedToTakeOver, + ApplyResTypeNotTakenOver, + ApplyResTypeFailedToRunDriftDetection, + ApplyResTypeFoundDrifts, + ApplyResTypeFoundDriftsInDegradedMode, + ApplyResTypeFailedToApply, + ApplyResTypeApplied, + ApplyResTypeAppliedWithFailedDriftDetection, +) + +// The messages reported on the Applied condition of a manifest. +const ( + ApplyResTypeAppliedDescription = "The manifest has been applied successfully" + ApplyResTypeAppliedWithFailedDriftDetectionDescription = "The manifest has been applied successfully, but KubeFleet cannot determine whether the object has drifted" + ApplyResTypeFailedToApplyDescription = "Failed to apply the manifest (error: %s)" +) + +type AvailabilityCheckResultType string + +const ( + // The result type for availability check being skipped. + AvailabilityResultTypeSkipped AvailabilityCheckResultType = "Skipped" + + // The result type for availability check failures. + AvailabilityResultTypeFailed AvailabilityCheckResultType = "FailedToCheck" + + // The result types for completed availability checks. + AvailabilityResultTypeAvailable AvailabilityCheckResultType = "Available" + AvailabilityResultTypeNotYetAvailable AvailabilityCheckResultType = "Unavailable" + AvailabilityResultTypeNotTrackable AvailabilityCheckResultType = "NotTrackable" +) + +// The messages reported on the Available condition of a manifest. +const ( + AvailabilityResultTypeAvailableDescription = "The manifest is available" + AvailabilityResultTypeNotYetAvailableDescription = "The manifest is not yet available; KubeFleet will check again later" + AvailabilityResultTypeNotTrackableDescription = "The manifest's availability is not trackable; KubeFleet assumes that the applied manifest is available" + AvailabilityResultTypeFailedDescription = "Failed to track the availability of the applied manifest (error: %s)" +) + +type manifestProcessingState struct { + // The manifest data in its raw form (not yet decoded). + manifest *placementv1alpha1.Manifest + + // The manifest identifier. + // + // If the manifest data cannot be decoded as a Kubernetes API object at all, the identifier + // will feature only the ordinal of the manifest data (its rank in the list of the resources). + // If the manifest data can be decoded as a Kubernetes API object, but the API is not available + // on the member cluster, the resource field of the identifier will be empty. + id *placementv1alpha1.ManifestIdentifier + // A string representation of the resource identifier (sans the resources field). + // This is only populated if the manifest data can be successfully decoded. + // + // It is of the format `GV=[API_GROUP]/[VERSION], Kind=[KIND], Namespace=[NAMESPACE], Name=[NAME]`, + // where [API_GROUP], [VERSION], [KIND], [NAMESPACE], and [NAME] are the API group/version/kind of the + // manifest object, and its owner namespace (if applicable)/name, respectively. + idStr string + + // The manifest data, decoded as a Kubernetes API object. + manifestObj *unstructured.Unstructured + + // The object in the member cluster that corresponds to the manifest object. + inMemberClusterObj *unstructured.Unstructured + + // The GVR of the manifest object. + gvr *schema.GroupVersionResource + + // The result of the apply operation. + applyRes ApplyResultType + // The result of the availability check operation. + availabilityCheckRes AvailabilityCheckResultType + // The error that occurred during the apply operation, if any. + applyErr error + // The error that occurred during the availability check operation, if any. + availabilityCheckErr error + // The diffs detected in the apply operation. + diffs []placementv1alpha1.PatchDetail + // A link back to the work object that includes the manifest. + fromWorkObj *placementv1alpha1.Work + // A link back to the primary work object. + fromPrimaryWorkObject *placementv1alpha1.Work + // The expected owner reference for the manifest object. + ownedBy *metav1.OwnerReference +} + +type workObjectProcessingState struct { + // The work object being processed. + work *placementv1alpha1.Work + + // The corresponding appliedWork object for the work object. + appliedWork *placementv1alpha1.AppliedWork + // The owner reference to use for all the manifests within the work object when they are applied. + appliedWorkOwnerRef *metav1.OwnerReference + + // The processing state of each manifest within the work object. + manifestProcessingStates []*manifestProcessingState +} + +type Reconciler struct { + hubClient client.Client + hubUncachedReader client.Reader + spokeClient client.Client + spokeDynamicClient dynamic.Interface + + workNSName string + + restMapper meta.RESTMapper + + parallelizer parallelizerutil.Parallelizer + + concurrentReconciles int + cleanupRequeueAfter time.Duration + periodicRequeueAfter time.Duration + cleanupWaitTime time.Duration + + ready atomic.Bool +} + +func New(workNSName string, + hubClient client.Client, hubUncachedReader client.Reader, + spokeClient client.Client, spokeDynamicClient dynamic.Interface, + restMapper meta.RESTMapper, + parallelizer parallelizerutil.Parallelizer, + concurrentReconciles int, + cleanupRequeueAfter time.Duration, + periodicRequeueAfter time.Duration, + cleanupWaitTime time.Duration, +) *Reconciler { + return &Reconciler{ + workNSName: workNSName, + hubClient: hubClient, + hubUncachedReader: hubUncachedReader, + spokeDynamicClient: spokeDynamicClient, + spokeClient: spokeClient, + restMapper: restMapper, + parallelizer: parallelizer, + concurrentReconciles: concurrentReconciles, + cleanupRequeueAfter: cleanupRequeueAfter, + periodicRequeueAfter: periodicRequeueAfter, + cleanupWaitTime: cleanupWaitTime, + } +} + +func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + if !r.ready.Load() { + klog.V(2).InfoS("Work applier is not yet ready to start; the member agent might still be connecting to the hub cluster", + "work", req.NamespacedName) + return ctrl.Result{RequeueAfter: time.Second * 5}, nil + } + + startTime := time.Now() + klog.V(2).InfoS("Reconciliation starts", "work", req.NamespacedName, "controller", controllerName) + defer func() { + latency := time.Since(startTime).Milliseconds() + klog.V(2).InfoS("Reconciliation ends", "work", req.NamespacedName, "latency", latency, "controller", controllerName) + }() + + // Retrieve the work object. + work := &placementv1alpha1.Work{} + err := r.hubClient.Get(ctx, req.NamespacedName, work) + switch { + case apierrors.IsNotFound(err): + klog.V(2).InfoS("The work object cannot be found", "work", req.NamespacedName, "controller", controllerName) + return ctrl.Result{}, nil + case err != nil: + wrappedErr := errors.NewAPIServerError(err, "", true, "work", req.NamespacedName, "controller", controllerName) + klog.ErrorS(err, "Failed to retrieve the work", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Check if the work object is a primary one, i.e., it has the count annotation set. + _, found := work.GetAnnotations()[placementv1alpha1.LinkedWorkCountAnnotationKey] + if !found { + klog.V(2).InfoS("The work object is not a primary one; skipping reconciliation", "work", req.NamespacedName, "controller", controllerName) + return ctrl.Result{}, nil + } + + // Clean things up if the work object has been marked for deletion. + if !work.DeletionTimestamp.IsZero() { + requeueAfter, err := r.cleanupWhenBindingDeleted(ctx, work) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to clean up linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if requeueAfter != nil { + return ctrl.Result{RequeueAfter: *requeueAfter}, nil + } + return ctrl.Result{}, nil + } + + // Retrieve all linked work objects. + linkedWorks, leftOverWorks, err := r.retrieveLinkedAndLeftOverWorks(ctx, work) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to retrieve linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Add cleanup finalizer to all linked work objects. + if err := r.addCleanupFinalizerTo(ctx, linkedWorks); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to add cleanup finalizer to linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Ensure an appliedWork object for each work object. + // + // The appliedWork object that corresponds to the primary work object will be set as the owner for all + // applied manifests across the linked work objects. + // + // Note that the appliedWork objects are returned in the same order as their corresponding work objects in the + // input. + appliedWorks, err := r.ensureAppliedWorks(ctx, linkedWorks) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to ensure appliedWork objects for linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Prepare the processing states for the work objects and their manifests. + workObjProcessingStates, manifestProcessingStates := prepareWorkObjectAndManifestProcessingStates(linkedWorks, appliedWorks) + + // Pre-process the manifests to apply. + // + // In this step, the work applier will: + // a) decode the manifests; and + // b) write ahead the manifest processing attempts; and + // c) remove any applied manifests left over from previous runs. + seenManifestIDs, err := r.preProcessWorkObjects(ctx, workObjProcessingStates) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to pre-process work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Before processing the manifests in the linked work objects, handle the left-over work objects: check if + // there is any applied manifest on them that needs to be cleaned up, and drop the cleanup finalizer + // from these work objects. + if err := r.cleanupLeftOverWorks(ctx, leftOverWorks, seenManifestIDs, workObjProcessingStates[0].appliedWorkOwnerRef); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to clean up left-over work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Set default values in the work object spec to avoid additional validation logic in the later steps. + setDefaultSyncStrategy(work) + + // Process the manifests. + // + // In this step, the work applier will: + // a) find if there has been a corresponding object in the member cluster for each manifest; + // b) take over the object if applicable; + // c) check for diffs/drifts if applicable; + // e) apply each manifest. + if err := r.processManifests(ctx, manifestProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to process manifests", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Track the availability information. + if err := r.trackInMemberClusterObjAvailability(ctx, manifestProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to check for object availability", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Refresh the status of the Work object. + if err := r.refreshWorkStatus(ctx, workObjProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to refresh work object status", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Refresh the status of the AppliedWork object. + if err := r.refreshAppliedWorkStatus(ctx, workObjProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to refresh appliedWork object status", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Periodically requeue the work object to check and (re-)apply manifests. + return ctrl.Result{RequeueAfter: r.periodicRequeueAfter}, nil +} + +func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + WithOptions(ctrloption.Options{ + MaxConcurrentReconciles: r.concurrentReconciles, + }). + For(&placementv1alpha1.Work{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). + Complete(r) +} diff --git a/pkg/v1/controllers/workapplier/controller_integration_test.go b/pkg/v1/controllers/workapplier/controller_integration_test.go new file mode 100644 index 000000000..db279d9e8 --- /dev/null +++ b/pkg/v1/controllers/workapplier/controller_integration_test.go @@ -0,0 +1,1636 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils" +) + +var _ = Describe("single work object", func() { + Context("core ops (apply manifests)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifests", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // Prepare the status information. + appliedResources := []placementv1alpha1.AppliedResource{ + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Ensure applied manifest has been removed. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("garbage collection (delete work object)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object status update, etc.) are + // omitted in this test node, as the behaviors have been verified elsewhere. + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can delete the work object", func() { + deleteWorkObject(workName, memberReservedNSName1) + }) + + It("should start deleting the appliedWork object in the foreground", func() { + appliedWorkBeingDeletedActual := appliedWorkBeingDeletedInForegroundActual(memberClient1, workName) + Eventually(appliedWorkBeingDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the appliedWork object in the foreground") + }) + + It("should remove the work object", func() { + // As there is no GC controller in the test environment, the appliedWork object will be stuck in the + // pending deletion state; the work applier will unblock the deletion after cleanupWaitTime, and + // proceed to remove the cleanup finalizer after 2 * cleanupWaitTime. + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval*2).Should(Succeed(), "Failed to remove the work object") + }) + + It("should unblock the appliedWork deletion on all applied objects", func() { + // The owner references stay (they are removed by the GC controller in a real cluster), but they should + // no longer block the deletion of the appliedWork object. + nsOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}, appliedWorkOwnerRef) + Eventually(nsOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the namespace object") + + deployOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: deployName}}, appliedWorkOwnerRef) + Eventually(deployOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the deployment object") + }) + + It("should delete the appliedWork object", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + }) + + AfterAll(func() { + // Remove the deployment object manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("leftover cleanup (remove manifests from work object)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularNSJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can update the work object to remove the deployment manifest", func() { + updateWorkObject(workName, memberReservedNSName1, nil, regularNSJSON) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should remove the deployment object", func() { + regularDeployAbsentActual := regularDeployAbsentActual(memberClient1, nsName, deployName) + Eventually(regularDeployAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) +}) + +var _ = Describe("multiple work objects", func() { + Context("core ops (apply manifests)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // Note that normally KubeFleet sets the primary work object as the owner of all secondary work objects; + // this is skipped here as there is no GC controller in the test environment. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularConfigMapJSON, + ) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularDeployJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should add cleanup finalizer to all the work objects", func() { + primaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the primary work object") + + secondaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the secondary work object") + }) + + It("should prepare an appliedWork object for each work object", func() { + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + + secondaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, secondaryWorkName) + Eventually(secondaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the secondary work object") + + // All the manifests across the linked work objects are owned by the appliedWork object of the + // primary work object. + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + }) + + It("should apply the manifests", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should update the work object statuses (deployment not yet available)", func() { + // The primary work object is not yet available, as the deployment has not become available. + primaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, primaryWorkConds, primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // The secondary work object is available. + // + // Note that the ordinals of the manifests are relative to their own work object. + secondaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, secondaryWorkConds, secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + // Each appliedWork object reports the resources applied from its own work object. + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object statuses (all available)", func() { + // The primary work object becomes available. + primaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, primaryWorkConds, primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // The secondary work object stays available. + secondaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, secondaryWorkConds, secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary work object is not owned by + // the primary one in this test), both work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + secondaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("garbage collection (delete primary work object)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // Note that normally KubeFleet sets the primary work object as the owner of all secondary work objects; + // this is skipped here as it has no effect on the work applier, and there is no GC controller in the + // test environment. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularConfigMapJSON, + ) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularDeployJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object of the primary work object; all the manifests + // across the linked work objects are owned by this appliedWork object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the two work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("can delete the primary work object", func() { + deleteWorkObject(primaryWorkName, memberReservedNSName1) + }) + + It("should start deleting the primary appliedWork object in the foreground", func() { + appliedWorkBeingDeletedActual := appliedWorkBeingDeletedInForegroundActual(memberClient1, primaryWorkName) + Eventually(appliedWorkBeingDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the primary appliedWork object in the foreground") + }) + + It("should remove the primary work object", func() { + // As there is no GC controller in the test environment, the primary appliedWork object will be stuck in + // the pending deletion state; the work applier will unblock the deletion after cleanupWaitTime, and + // proceed to remove the cleanup finalizers after 2 * cleanupWaitTime. + workRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(workRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval*2).Should(Succeed(), "Failed to remove the primary work object") + }) + + It("should remove the cleanup finalizer from the secondary work object", func() { + // In a real cluster the secondary work object is owned by the primary one and will be garbage collected; + // here only the finalizer removal is verified. + workFinalizerRemovedActual := workFinalizerRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(workFinalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer from the secondary work object") + }) + + It("should remove the secondary appliedWork object", func() { + // The secondary appliedWork object owns no resources, and is deleted by the work applier directly. + appliedWorkAbsentActual := appliedWorkAbsentActual(memberClient1, secondaryWorkName) + Eventually(appliedWorkAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + }) + + It("should unblock the primary appliedWork deletion on all applied objects", func() { + // The owner references stay (they are removed by the GC controller in a real cluster), but they should + // no longer block the deletion of the primary appliedWork object. + nsOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}, appliedWorkOwnerRef) + Eventually(nsOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the namespace object") + + deployOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: deployName}}, appliedWorkOwnerRef) + Eventually(deployOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the deployment object") + + configMapOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: configMapName}}, appliedWorkOwnerRef) + Eventually(configMapOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the config map object") + }) + + It("should delete the primary appliedWork object", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + }) + + AfterAll(func() { + // Delete the secondary work object manually, as there is no GC controller in the test environment. + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("leftover cleanup (remove and shuffle manifests from linked work objects)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The name of the new primary placement resource snapshot, as created by a rollout. + newPrimarySnapshotName := fmt.Sprintf("%s-new", primarySnapshotName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + var regularNSJSON []byte + var regularDeployJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON = marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // Note that normally KubeFleet sets the primary work object as the owner of all secondary work objects; + // this is skipped here as it has no effect on the work applier, and there is no GC controller in the + // test environment. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularConfigMapJSON, + ) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularDeployJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object of the primary work object; all the manifests + // across the linked work objects are owned by this appliedWork object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the two work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("can update the primary work object to keep only the namespace manifest (rollout in progress)", func() { + // Mimic a rollout: the primary placement resource snapshot has been bumped, and the primary work object + // is refreshed first; the deployment manifest is to be moved to the secondary work object, which has not + // been refreshed yet. + updateLinkedWorkObject(primaryWorkName, memberReservedNSName1, newPrimarySnapshotName, nil, regularNSJSON) + }) + + It("should not remove the deployment object", func() { + // The linked work objects are now in an inconsistent state (the secondary work object is still linked + // to the old primary placement resource snapshot); the work applier should hold off any processing, + // including the removal of the deployment object. + Consistently(func() error { + gotDeploy := &appsv1.Deployment{} + if err := memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy); err != nil { + return fmt.Errorf("failed to retrieve the Deployment object: %w", err) + } + if !gotDeploy.DeletionTimestamp.IsZero() { + return fmt.Errorf("deployment object has been marked for deletion") + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The deployment object has been removed unexpectedly") + }) + + It("can update the secondary work object to add the deployment manifest and drop the config map manifest", func() { + updateLinkedWorkObject(secondaryWorkName, memberReservedNSName1, newPrimarySnapshotName, nil, regularDeployJSON) + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // While the linked work objects were in an inconsistent state, the work applier kept requeueing the + // primary work object with exponential backoff; allow more time for the next attempt to happen. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // Note that the ordinals of the manifests are relative to their own work object. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + // The deployment object is the same one as before (i.e., it has not been re-created). + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + ManifestIdentifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + It("should keep the deployment object", func() { + // The deployment object should still be owned by the appliedWork object of the primary work object. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep the deployment object") + + gotDeploy := &appsv1.Deployment{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + Expect(gotDeploy.UID).To(Equal(regularDeploy.UID), "The deployment object has been re-created unexpectedly") + }) + + It("should remove the config map object", func() { + regularConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary work object is not owned by + // the primary one in this test), both work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + secondaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/defaults.go b/pkg/v1/controllers/workapplier/defaults.go new file mode 100644 index 000000000..1638eff78 --- /dev/null +++ b/pkg/v1/controllers/workapplier/defaults.go @@ -0,0 +1,47 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// setDefaultSyncStrategy fills in a default sync strategy on a work object that carries none. +// +// The individual fields of a sync strategy are defaulted by the API server; this only covers the case where +// the sync strategy itself is absent, which leaves those defaults unapplied. +func setDefaultSyncStrategy(work *placementv1alpha1.Work) { + if work.Spec.SyncStrategy != nil { + if work.Spec.SyncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodServerSideApply && + work.Spec.SyncStrategy.ServerSideApplyOptions == nil { + work.Spec.SyncStrategy.ServerSideApplyOptions = &placementv1alpha1.ServerSideApplyOptions{ + ForceConflicts: false, + } + } + return + } + + work.Spec.SyncStrategy = &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodClientSideApply, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionReportError, + WhenDrifted: placementv1alpha1.WhenDriftedOptionApplyAnyway, + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionReportError, + WhenPlacementDeleted: placementv1alpha1.WhenPlacementDeletedOptionCleanUpResources, + WhenNamespaceDoesNotExist: placementv1alpha1.WhenNamespaceDoesNotExistOptionCreateNamespace, + ComparisonOption: placementv1alpha1.ComparisonOptionPartialComparison, + } +} diff --git a/pkg/v1/controllers/workapplier/join_leave.go b/pkg/v1/controllers/workapplier/join_leave.go new file mode 100644 index 000000000..d3ac8f1dc --- /dev/null +++ b/pkg/v1/controllers/workapplier/join_leave.go @@ -0,0 +1,103 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +// Join sets the work applier to work. +// +// As a member cluster side controller, work applier can only start reconciling work objects after the +// member cluster has successfully joined the fleet. +func (r *Reconciler) Join() error { + if !r.ready.Load() { + klog.InfoS("Setting the work applier as ready") + } + r.ready.Store(true) + return nil +} + +// Leave performs cleanup ops when a member cluster leaves the fleet. +// +// For availability reasons, when a member cluster leaves the fleet, all the resources are left intact on the member +// cluster. +func (r *Reconciler) Leave(ctx context.Context) error { + if r.ready.Load() { + klog.InfoS("Setting the work applier as not ready") + } + r.ready.Store(false) + + // List all the work objects. + workList := &placementv1alpha1.WorkList{} + if err := r.hubUncachedReader.List(ctx, workList, client.InNamespace(r.workNSName)); err != nil { + return errors.NewAPIServerError(err, "failed to list the work objects", false) + } + + // For each work object, if it has the cleanup finalizer, find the corresponding appliedWork object, and + // delete it with the Orphan propagation policy. + // + // It is OK if the appliedWork object hasn't been created yet. + deletePolicy := metav1.DeletePropagationOrphan + for idx := range workList.Items { + work := &workList.Items[idx] + if !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + continue + } + + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: work.Name}, + } + if err := r.spokeClient.Delete(ctx, appliedWork, &client.DeleteOptions{PropagationPolicy: &deletePolicy}); err != nil { + if apierrors.IsNotFound(err) { + klog.V(2).InfoS("The appliedWork object does not exist; skipping deletion", + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + continue + } + return errors.NewAPIServerError(err, "failed to delete the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + } + klog.V(2).InfoS("Deleted the appliedWork object with the Orphan propagation policy", + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + } + + // Drop the cleanup finalizer from all the work objects. + for idx := range workList.Items { + work := &workList.Items[idx] + if !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + continue + } + + controllerutil.RemoveFinalizer(work, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, work); err != nil { + return errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the work", false, + "work", klog.KObj(work)) + } + } + + klog.V(2).InfoS("Removed the cleanup finalizer from all the work objects", "workCount", len(workList.Items)) + return nil +} diff --git a/pkg/v1/controllers/workapplier/preprocess.go b/pkg/v1/controllers/workapplier/preprocess.go new file mode 100644 index 000000000..59ff60a5b --- /dev/null +++ b/pkg/v1/controllers/workapplier/preprocess.go @@ -0,0 +1,582 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "sync/atomic" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/ownerreferences" +) + +func prepareWorkObjectAndManifestProcessingStates(works []*placementv1alpha1.Work, + appliedWorks []*placementv1alpha1.AppliedWork) ([]*workObjectProcessingState, []*manifestProcessingState) { + // Pre-allocate the slices. + workObjectStates := make([]*workObjectProcessingState, 0, len(works)) + manifestProcessingStates := make([]*manifestProcessingState, 0, len(works[0].Spec.Manifests)) + + primaryWork := works[0] + primaryAppliedWorkOwnerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "AppliedWork", + Name: appliedWorks[0].GetName(), + UID: appliedWorks[0].GetUID(), + BlockOwnerDeletion: ptr.To(true), + } + + for i := range works { + work := works[i] + + perWorkManifestProcessingStates := make([]*manifestProcessingState, 0, len(work.Spec.Manifests)) + for j := range work.Spec.Manifests { + manifest := &work.Spec.Manifests[j] + manifestProcessingState := &manifestProcessingState{ + manifest: manifest, + fromWorkObj: work, + fromPrimaryWorkObject: primaryWork, + // Use the appliedWork object that corresponds to the primary work object as the owner for + // all manifests in this work object once they are applied. + ownedBy: primaryAppliedWorkOwnerRef, + } + perWorkManifestProcessingStates = append(perWorkManifestProcessingStates, manifestProcessingState) + manifestProcessingStates = append(manifestProcessingStates, manifestProcessingState) + } + workObjectState := &workObjectProcessingState{ + work: work, + appliedWork: appliedWorks[i], + appliedWorkOwnerRef: primaryAppliedWorkOwnerRef, + manifestProcessingStates: perWorkManifestProcessingStates, + } + workObjectStates = append(workObjectStates, workObjectState) + } + + return workObjectStates, manifestProcessingStates +} + +func (r *Reconciler) preProcessWorkObjects(ctx context.Context, workObjProcessingStates []*workObjectProcessingState) (sets.Set[string], error) { + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + + // Pre-process the manifests one work object at a time, so that the ordinals assigned to the + // manifest identifiers stay relative to their own work object. + if err := r.preProcessPerWorkObjManifests(ctx, workObjProcessingState.manifestProcessingStates); err != nil { + return nil, kferrors.Wraps(err, "failed to pre-process work object", "work", klog.KObj(workObjProcessingState.work)) + } + + klog.V(2).InfoS("Pre-processed a work object", + "work", klog.KObj(workObjProcessingState.work), + "manifestCount", len(workObjProcessingState.manifestProcessingStates)) + } + + seenManifestIDs := markDuplicatedManifests(workObjProcessingStates) + + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + + // Write-ahead the manifest processing attempts in the work object status so that + // KubeFleet can always track applied manifests, even upon untimely crashes. This method will + // also check for any leftover apply attempts from previous runs and clean them up (if the + // corresponding manifest object has been applied). + if err := r.writeAheadPerWorkObjManifestProcessingAttempts(ctx, workObjProcessingState, seenManifestIDs); err != nil { + return nil, kferrors.Wraps(err, "failed to write ahead manifest processing states", "work", klog.KObj(workObjProcessingState.work)) + } + + klog.V(2).InfoS("Wrote ahead manifest processing states for a work object", + "work", klog.KObj(workObjProcessingState.work)) + } + return seenManifestIDs, nil +} + +func (r *Reconciler) preProcessPerWorkObjManifests(ctx context.Context, manifestProcessingStates []*manifestProcessingState) error { + childCtx, childCancel := context.WithCancel(ctx) + defer childCancel() + + var innerErr atomic.Value + preProcessOneManifest := func(pieces int) { + // At this moment the processing state is just initialized. + processingState := manifestProcessingStates[pieces] + ownerWorkObj := processingState.fromWorkObj + + gvr, manifestObj, err := r.decodeManifest(processingState.manifest) + // There exists a corner case where the decoding might have failed due to transient errors (e.g., the + // API discovery endpoint is not responding); in this case, if we register the error as a user-end + // decoding error and skip the manifest, and if the manifest has been applied before, the error might + // trigger the later left-over cleanup step, which would delete the previously applied manifest. It should + // be corrected in the subsequent reconciliation; however, it might trigger fluctuations. To be on the safer + // side, the work applier is set to requeue upon transient decoding errors. + // + // Note that with aggregated API discovery enabled by default in most Kubernetes clusters now, + // normally if the first decoding attempt succeeds, all subsequent decoding attempts will be resolved via + // cache. + if err != nil && kferrors.Category(err) == kferrors.ErrCategoryTransient { + wrappedErr := kferrors.Wraps(err, "failed to decode the manifest", "ordinal", pieces) + innerErr.Store(wrappedErr) + childCancel() + return + } + // Build the manifest identifier. Note that this would return an identifier even if the decoding fails. + processingState.id = buildManifestIdentifier(pieces, gvr, manifestObj) + if err != nil { + wrappedErr := kferrors.Wraps(err, "failed to decode the manifest", "ordinal", pieces) + processingState.applyErr = wrappedErr + processingState.applyRes = ApplyResTypeDecodingErred + return + } + + // Reject objects with a generate name but no name. + if len(manifestObj.GetGenerateName()) > 0 && len(manifestObj.GetName()) == 0 { + wrappedErr := kferrors.NewUserError(nil, "rejected an object with only generate name", + "ordinal", pieces, "manifestObj", klog.KObj(manifestObj), "work", klog.KObj(ownerWorkObj)) + processingState.applyErr = wrappedErr + processingState.applyRes = ApplyResTypeFoundGenerateName + return + } + + processingState.manifestObj = manifestObj + processingState.gvr = gvr + + // Add the string representation of the manifest identifier to the bundle. + // + // Note that the string representation ignores the ordinal number in the identifier. + idStr := formatManifestIdentifierStr(processingState.id) + processingState.idStr = idStr + + klog.V(2).InfoS("Decoded a manifest", + "manifestObj", klog.KObj(manifestObj), + "GVR", *gvr, + "work", klog.KObj(ownerWorkObj)) + } + r.parallelizer.ParallelizeUntil(childCtx, len(manifestProcessingStates), preProcessOneManifest, "preProcessManifests") + + if err := innerErr.Load(); err != nil { + return kferrors.Wraps(err.(error), "failed to pre-process manifests") + } + // If the context has been cancelled, return an error immediately so that the reconciliation loop would exit now. + if ctx.Err() != nil { + return kferrors.NewTransientError(ctx.Err(), "context was canceled or timed out during manifest pre-processing") + } + return nil +} + +// Decodes the manifest JSON into a Kubernetes unstructured object. +func (r *Reconciler) decodeManifest(manifest *placementv1alpha1.Manifest) (*schema.GroupVersionResource, *unstructured.Unstructured, error) { + unstructuredObj := &unstructured.Unstructured{} + if err := unstructuredObj.UnmarshalJSON(manifest.Raw); err != nil { + return &schema.GroupVersionResource{}, nil, kferrors.NewUserError(err, "failed to unmarshal JSON") + } + + mapping, err := r.restMapper.RESTMapping(unstructuredObj.GroupVersionKind().GroupKind(), unstructuredObj.GroupVersionKind().Version) + if err != nil { + if meta.IsNoMatchError(err) { + return &schema.GroupVersionResource{}, unstructuredObj, + kferrors.NewUserError(err, "failed to find GVR from member cluster client REST mapping; the resource might not be available on the member cluster side") + } + return &schema.GroupVersionResource{}, unstructuredObj, + kferrors.NewTransientError(err, "failed to complete REST mapping for the manifest") + } + + return &mapping.Resource, unstructuredObj, nil +} + +func buildManifestIdentifier( + manifestIdx int, + gvr *schema.GroupVersionResource, + manifestObj *unstructured.Unstructured, +) *placementv1alpha1.ManifestIdentifier { + // The ordinal field is always set. + identifier := &placementv1alpha1.ManifestIdentifier{ + Ordinal: manifestIdx, + } + + // Set the GVK, name, namespace, and generate name information if the manifest can be decoded + // as a Kubernetes unstructured object. + // + // Note that for cluster-scoped objects, the namespace field will be empty. + if manifestObj != nil { + identifier.APIGroup = manifestObj.GroupVersionKind().Group + identifier.APIVersion = manifestObj.GroupVersionKind().Version + identifier.Kind = manifestObj.GetKind() + identifier.Name = manifestObj.GetName() + identifier.Namespace = manifestObj.GetNamespace() + } + + // Set the GVR information if the manifest object can be REST mapped. + if gvr != nil { + identifier.Resource = gvr.Resource + } + + return identifier +} + +func formatManifestIdentifierStr(id *placementv1alpha1.ManifestIdentifier) string { + // API version is omitted; KubeFleet will report an error if one attempts to place two manifests with the same + // GK but different versions. + return fmt.Sprintf("GK=%s/%s, Namespace=%s, Name=%s", + id.APIGroup, id.Kind, id.Namespace, id.Name) +} + +// markDuplicatedManifests rejects any manifest that shares an identifier with a manifest seen earlier, so that +// only its first occurrence is applied; duplicates would otherwise overwrite one another on the member cluster. +func markDuplicatedManifests(workObjProcessingStates []*workObjectProcessingState) sets.Set[string] { + // The set spans all the work objects, as linked work objects apply to the same member cluster. + seenManifestIDs := sets.New[string]() + + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + + for i := range workObjProcessingState.manifestProcessingStates { + processingState := workObjProcessingState.manifestProcessingStates[i] + if processingState.applyErr != nil { + // The manifest has already failed pre-processing; it has no usable identifier string. + klog.V(2).InfoS("Skipped a manifest in the duplicated manifest checking process as it has failed in the decoding process", + "work", klog.KObj(workObjProcessingState.work), "ordinal", i, + "applyErr", processingState.applyErr, "applyResTyp", processingState.applyRes) + continue + } + + if seenManifestIDs.Has(processingState.idStr) { + processingState.applyErr = kferrors.NewUserError(nil, "rejected a duplicated manifest", + "ordinal", processingState.id.Ordinal, "manifest", processingState.idStr, + "work", klog.KObj(processingState.fromWorkObj)) + processingState.applyRes = ApplyResTypeDuplicated + continue + } + seenManifestIDs.Insert(processingState.idStr) + } + + klog.V(2).InfoS("Completed the check for duplicated manifests", "work", klog.KObj(workObjProcessingState.work)) + } + return seenManifestIDs +} + +func (r *Reconciler) writeAheadPerWorkObjManifestProcessingAttempts(ctx context.Context, + workObjProcessingState *workObjectProcessingState, seenManifestIDs sets.Set[string]) error { + work := workObjProcessingState.work + manifestProcessingStates := workObjProcessingState.manifestProcessingStates + + // As a shortcut, if there's no spec change in the Work object and the status indicates that + // a previous apply attempt has been recorded (**successful or not**), KubeFleet will skip the write-ahead + // op. + appliedCond := meta.FindStatusCondition(work.Status.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond != nil && appliedCond.ObservedGeneration == work.Generation { + klog.V(2).InfoS("Attempt to apply the current set of manifests has been made before and the results have been recorded; will skip the write-ahead process", + "work", klog.KObj(work)) + return nil + } + + // Prepare the status update (the new manifest conditions) for the write-ahead process. + perManifestStatusesToWriteAhead := make([]placementv1alpha1.PerManifestStatus, 0, len(manifestProcessingStates)) + + // Build an index of existing manifest statuses for quicker lookups. + existingManifestStatusIdx := prepareExistingManifestStatusIndex(work.Status.Manifests) + + for idx := range manifestProcessingStates { + manifestProcessingState := manifestProcessingStates[idx] + if manifestProcessingState.applyErr != nil { + // Skip a manifest if it cannot be pre-processed, e.g., it cannot be decoded. + // + // Such manifests would still be reported in the status (see the later parts of the + // reconciliation loop), it is just that they are not relevant in the write-ahead + // process. + klog.V(2).InfoS("Skipped a manifest in the write-ahead process as it has failed the decoding process or the duplicated manifest checking process", + "work", klog.KObj(work), "ordinal", idx, + "applyErr", manifestProcessingState.applyErr, "applyResTyp", manifestProcessingState.applyRes) + continue + } + + manifestStatusToWriteAhead := buildManifestStatusToWriteAhead(manifestProcessingState, work, existingManifestStatusIdx) + perManifestStatusesToWriteAhead = append(perManifestStatusesToWriteAhead, manifestStatusToWriteAhead) + + klog.V(2).InfoS("Prepared write-ahead information for a manifest", + "manifestObj", klog.KObj(manifestProcessingState.manifestObj), "manifestID", manifestProcessingState.id, "work", klog.KObj(work)) + } + + // Identify any manifests from previous runs that might have been applied and are now left + // over in the member cluster. + leftOverManifests := findLeftOverManifests(perManifestStatusesToWriteAhead, work, existingManifestStatusIdx, seenManifestIDs) + if err := r.removeLeftOverManifests(ctx, leftOverManifests, workObjProcessingState); err != nil { + return kferrors.Wraps(err, "failed to remove leftover manifests", + "work", klog.KObj(work), "leftOverManifestCount", len(leftOverManifests), "removalFailedCount", len(err.Errors())) + } + klog.V(2).InfoS("Left-over manifests are found and removed", + "leftOverManifestCount", len(leftOverManifests), "work", klog.KObj(work)) + + // Update the status. + // + // Note that the Work object might have been refreshed by controllers on the hub cluster + // before this step runs; in this case the current reconciliation loop must be abandoned. + if work.Status.Conditions == nil { + // As a sanity check, set an empty set of conditions. Currently the API definition does + // not allow nil conditions. + work.Status.Conditions = []metav1.Condition{} + } + work.Status.Manifests = perManifestStatusesToWriteAhead + if err := r.hubClient.Status().Update(ctx, work); err != nil { + return kferrors.NewAPIServerError(err, "failed to update work object status", false) + } + + klog.V(2).InfoS("Write-ahead process completed", "work", klog.KObj(work)) + return nil +} + +// prepareExistingManifestStatusIndex indexes the given manifest statuses by their identifier strings, so that +// the status of a specific manifest can be looked up without scanning the whole list. +func prepareExistingManifestStatusIndex(manifestStatuses []placementv1alpha1.PerManifestStatus) map[string]int { + idx := make(map[string]int, len(manifestStatuses)) + for i := range manifestStatuses { + idStr := formatManifestIdentifierStr(&manifestStatuses[i].Identifier) + idx[idStr] = i + } + return idx +} + +// buildManifestStatusToWriteAhead prepares the status of a manifest for the write-ahead process. +func buildManifestStatusToWriteAhead(manifestProcessingState *manifestProcessingState, + work *placementv1alpha1.Work, + existingManifestStatusIdx map[string]int, +) placementv1alpha1.PerManifestStatus { + // If the manifest has an entry in the existing set of manifest statuses, port the previously + // recorded processing results back. + if idx, found := existingManifestStatusIdx[manifestProcessingState.idStr]; found { + return work.Status.Manifests[idx] + } + + // The manifest has not been processed before; report that KubeFleet is preparing to process it. + return placementv1alpha1.PerManifestStatus{ + Identifier: *manifestProcessingState.id, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondPreparingToProcessReason, + Message: "The manifest is being prepared for processing", + LastTransitionTime: metav1.Now(), + }, + }, + } +} + +// findLeftOverManifests returns the manifests that have been left over on the member cluster side. +func findLeftOverManifests( + perManifestStatusesToWriteAhead []placementv1alpha1.PerManifestStatus, + work *placementv1alpha1.Work, + existingManifestStatusIdx map[string]int, + seenManifestIDs sets.Set[string], +) []placementv1alpha1.ManifestIdentifier { + // Build an index for quicker lookup in the newly prepared write-ahead manifest conditions. + // Here the work applier uses the string representations as map keys; ordinals are omitted from any lookup. + // + // Note that before this step, the work applier has already filtered out duplicate manifests. + manifestStatusesToWriteAheadIdx := make(map[string]int, len(perManifestStatusesToWriteAhead)) + for idx := range perManifestStatusesToWriteAhead { + idStr := formatManifestIdentifierStr(&perManifestStatusesToWriteAhead[idx].Identifier) + manifestStatusesToWriteAheadIdx[idStr] = idx + } + + // For each manifest condition in the existing set of manifest conditions, check if + // there is a corresponding entry in the set of manifest conditions prepared for the write-ahead + // process. If not, the work applier will consider that the manifest has been left over on the member + // cluster side and should be removed. + + // An existing manifest status without a counterpart in the write-ahead set refers to a manifest that + // is no longer part of the work object, and that might still be present on the member cluster side. + leftOverManifests := []placementv1alpha1.ManifestIdentifier{} + for existingManifestIDStr, existingManifestStatusIdx := range existingManifestStatusIdx { + if _, found := manifestStatusesToWriteAheadIdx[existingManifestIDStr]; found { + // The current manifest condition does not have a corresponding entry in the set of manifest + // conditions prepared for the write-ahead process. + continue + } + + if seenManifestIDs.Has(existingManifestIDStr) { + // There exists a corner case where a manifest might have changed its location, i.e., it was previously + // seen on work object A but now lives on work object B. In this case the manifest should not be + // categorized as a left-over manifest. + klog.V(2).InfoS("A manifest has moved to another work object; it will not be considered a left-over manifest", + "manifestID", existingManifestIDStr, "work", klog.KObj(work)) + continue + } + + existingManifestStatus := work.Status.Manifests[existingManifestStatusIdx] + + // The work applier assumes that the manifest has been applied if: + // a) it has an Applied condition set to the True status; or + // b) it has an Applied condition which signals that the object is preparing to be processed. + // + // Note that the manifest status might not be up-to-date, so the work applier will not check on the + // generation information. + appliedCond := meta.FindStatusCondition(existingManifestStatus.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond == nil { + continue + } + if appliedCond.Status == metav1.ConditionTrue || appliedCond.Reason == placementv1alpha1.WorkAppliedCondPreparingToProcessReason { + leftOverManifests = append(leftOverManifests, existingManifestStatus.Identifier) + } + } + return leftOverManifests +} + +// removeLeftOverManifests removes applied left-over manifests from the member cluster. +func (r *Reconciler) removeLeftOverManifests( + ctx context.Context, + leftOverManifests []placementv1alpha1.ManifestIdentifier, + workObjProcessingState *workObjectProcessingState, +) utilerrors.Aggregate { + // Remove all the manifests in parallel. + // + // This is concurrency safe as each worker processes its own applied manifest and writes + // to its own error slot. + + // Cancel the child context anyway to avoid leaks. + childCtx, cancel := context.WithCancel(ctx) + defer cancel() + + // Pre-allocate the slice. + errs := make([]error, len(leftOverManifests)) + doWork := func(pieces int) { + leftOverManifest := leftOverManifests[pieces] + + if err := r.removeOneLeftOverManifest(childCtx, leftOverManifest, workObjProcessingState.appliedWorkOwnerRef); err != nil { + errs[pieces] = kferrors.Wraps(err, "failed to remove a left-over manifest", "manifestId", leftOverManifest) + } + } + r.parallelizer.ParallelizeUntil(childCtx, len(leftOverManifests), doWork, "removeLeftOverManifests") + + return utilerrors.NewAggregate(errs) +} + +// removeOneLeftOverManifest removes an applied manifest object that is left over in the member cluster. +func (r *Reconciler) removeOneLeftOverManifest( + ctx context.Context, + leftOverManifest placementv1alpha1.ManifestIdentifier, + expectedAppliedWorkOwnerRef *metav1.OwnerReference, +) error { + // Build the GVR. + gvr := schema.GroupVersionResource{ + Group: leftOverManifest.APIGroup, + Version: leftOverManifest.APIVersion, + Resource: leftOverManifest.Resource, + } + manifestNamespace := leftOverManifest.Namespace + manifestName := leftOverManifest.Name + manifestRef := klog.KRef(manifestNamespace, manifestName) + + inMemberClusterObj, err := r.spokeDynamicClient. + Resource(gvr). + Namespace(manifestNamespace). + Get(ctx, manifestName, metav1.GetOptions{}) + switch { + case err != nil && apierrors.IsNotFound(err): + // The object has been deleted from the member cluster; no further action is needed. + return nil + case err != nil: + // Failed to retrieve the object from the member cluster. + // + // The cached flag is set to false as the dynamic client is non-caching. + return kferrors.NewAPIServerError(err, "failed to retrieve the object from the member cluster", false, + "gvr", gvr, "manifestObj", manifestRef) + case inMemberClusterObj.GetDeletionTimestamp() != nil: + // The object has been marked for deletion; no further action is needed. + return nil + } + + // There are occasions, though rare, where the object has the same GVR + namespace + name + // combo but is not the applied manifest object the work applier tries to find. This could happen if the object + // has been deleted and then re-created manually without the work applier's acknowledgement. In such cases + // the work applier would ignore the object, and this is not registered as an error. + if !isInMemberClusterObjectDerivedFromManifestObj(inMemberClusterObj, expectedAppliedWorkOwnerRef) { + klog.V(2).InfoS("The object to remove is not derived from the manifest object; will not proceed with the removal", + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + return nil + } + + switch { + case len(inMemberClusterObj.GetOwnerReferences()) > 1: + // KubeFleet is not the sole owner of the object; in this case, KubeFleet will only drop the + // ownership. + klog.V(2).InfoS("The object to remove is co-owned by other sources; will drop the ownership", + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + ownerreferences.Disown(inMemberClusterObj, expectedAppliedWorkOwnerRef) + if _, err := r.spokeDynamicClient.Resource(gvr).Namespace(manifestNamespace).Update(ctx, inMemberClusterObj, metav1.UpdateOptions{}); err != nil && !apierrors.IsNotFound(err) { + return kferrors.NewAPIServerError(err, "failed to drop the ownership of the object", false, + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + } + default: + // KubeFleet is the sole owner of the object; in this case, KubeFleet will delete the object. + klog.V(2).InfoS("The object to remove is solely owned by KubeFleet; will delete the object", + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + inMemberClusterObjUID := inMemberClusterObj.GetUID() + deleteOpts := metav1.DeleteOptions{ + Preconditions: &metav1.Preconditions{ + // Add a UID pre-condition to guard against the case where the object has changed + // right before the deletion request is sent. + // + // Technically speaking resource version based concurrency control should also be + // enabled here; the work applier drops the check to avoid conflicts; this is safe as the KubeFleet + // ownership is considered to be a reserved field and other changes on the object are + // irrelevant to this step. + UID: &inMemberClusterObjUID, + }, + } + if err := r.spokeDynamicClient.Resource(gvr).Namespace(manifestNamespace).Delete(ctx, manifestName, deleteOpts); err != nil && !apierrors.IsNotFound(err) { + return kferrors.NewAPIServerError(err, "failed to delete the object", false, + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + } + } + return nil +} + +// isInMemberClusterObjectDerivedFromManifestObj checks if an object in the member cluster is derived +// from a specific manifest object. +func isInMemberClusterObjectDerivedFromManifestObj(inMemberClusterObj *unstructured.Unstructured, expectedAppliedWorkOwnerRef *metav1.OwnerReference) bool { + // Do a sanity check. + if inMemberClusterObj == nil { + return false + } + + // Verify if the owner reference still stands. + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + if ownerreferences.AreEqual(&curOwners[idx], expectedAppliedWorkOwnerRef) { + return true + } + } + return false +} diff --git a/pkg/v1/controllers/workapplier/process.go b/pkg/v1/controllers/workapplier/process.go new file mode 100644 index 000000000..739323e95 --- /dev/null +++ b/pkg/v1/controllers/workapplier/process.go @@ -0,0 +1,491 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/utils/resource" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/ownerreferences" +) + +func (r *Reconciler) processManifests(ctx context.Context, manifestProcessingStates []*manifestProcessingState) error { + // Process all manifests in parallel. + // + // There are cases where certain groups of manifests should not be processed in parallel with + // each other (e.g., a config map must be applied after its owner namespace is applied); + // to address this situation, manifests are processed in waves: manifests in the same wave are + // processed in parallel, while different waves are processed sequentially. + + // Organize the manifests into different waves for parallel processing based on their + // GVR information. + processingWaves := organizeManifestsIntoProcessingWaves(manifestProcessingStates) + for idx := range processingWaves { + statesInWave := processingWaves[idx].states + + // TO-DO (chenyu1): evaluate if there is a need to avoid repeated closure + // assignment just for capturing variables. + doWork := func(piece int) { + state := statesInWave[piece] + if state.applyErr != nil { + // Skip a manifest if it has failed pre-processing. + // + // This added as a sanity check as the organization step normally + // would have already skipped all the manifests with processing failures. + return + } + + r.processOneManifest(ctx, state) + klog.V(2).InfoS("Processed a manifest", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "work", klog.KObj(state.fromWorkObj), "primaryWork", klog.KObj(state.fromPrimaryWorkObject)) + } + + r.parallelizer.ParallelizeUntil(ctx, len(statesInWave), doWork, fmt.Sprintf("processingManifestsInWave%d", idx)) + + // Unlike some other steps in the reconciliation loop, the manifest processing step does not end + // with a contextual API call; consequently, if the context has been cancelled during this step, + // some manifest might not get processed at all, and passing such states to the next step may trigger + // unexpected behaviors. To address this, at the end of this step the work applier checks for context + // cancellation directly. + if err := ctx.Err(); err != nil { + klog.V(2).InfoS("Manifest processing has been interrupted as the main context has been cancelled") + return errors.NewTransientError(err, "manifest processing has been interrupted") + } + } + return nil +} + +func (r *Reconciler) processOneManifest(ctx context.Context, manifestProcessingState *manifestProcessingState) { + manifestObj := manifestProcessingState.manifestObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + // Firstly, attempt to find if an object has been created in the member cluster based on the manifest object. + if shouldSkipProcessing := r.findInMemberClusterObjectFor(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + // Take over the object in the member cluster that corresponds to the manifest object + // if applicable. + // + // KubeFleet will perform the takeover if: + // a) KubeFleet can find an object in the member cluster that corresponds to the manifest object, and + // it is not currently owned by KubeFleet (specifically the expected appliedWork object); and + // b) takeover is allowed (i.e., the WhenAlreadyExists option in the sync strategy is not set to ReportError). + if shouldSkipProcessing := r.takeOverInMemberClusterObjectIfApplicable(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + // For both the ClientSideApply and ServerSideApply methods, ownership is a hard requirement. + // Skip the rest of the processing step if the resource has been created in the member cluster + // but KubeFleet is not listed as an owner of the resource in the member cluster yet (i.e., the + // takeover process does not run). This check is necessary as KubeFleet supports the + // WhenAlreadyExists option ReportError. + if !canApplyWithOwnership(manifestProcessingState) { + wrappedErr := errors.NewUserError(nil, "no ownership of the object in the member cluster; takeover is needed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeNotTakenOver + klog.V(2).InfoS("Ownership is not established yet; skip the apply op", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return + } + + // Perform a round of drift detection before running the apply op, if the ApplyStrategy + // dictates that an apply op can only be run when there are no drifts found. + if shouldSkipProcessing := r.performPreApplyDriftDetectionIfApplicable(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + // Perform the apply op. + appliedObj, err := r.apply(ctx, manifestProcessingState) + if err != nil { + wrappedErr := errors.Wraps(err, "failed to apply the manifest", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToApply + klog.ErrorS(err, "Failed to apply the manifest", errors.Args(wrappedErr)...) + return + } + if appliedObj != nil { + // Track the newly applied object, if an apply op has been run. + manifestProcessingState.inMemberClusterObj = appliedObj + } + klog.V(2).InfoS("Apply process completed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + + // Perform another round of drift detection after the apply op, if the ApplyStrategy dictates + // that drift detection should be done in full comparison mode. + // + // Drift detection is currently always enabled in KubeFleet. At this stage of execution, it is + // safe for us to assume that all the managed fields have been overwritten by the just + // completed apply op (or no apply op is necessary); consequently, no further drift + // detection is necessary if the partial comparison mode is used. However, for the full + // comparison mode, the apply op might not to able to resolve all the drifts, should there + // be any change made on the unmanaged fields; and KubeFleet would need to perform another + // round of drift detection. + if shouldSkipProcessing := r.performPostApplyDriftDetectionIfApplicable(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + manifestProcessingState.applyRes = ApplyResTypeApplied + klog.V(2).InfoS("Manifest processing completed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) +} + +// findInMemberClusterObjectFor attempts to find the corresponding object in the member cluster +// for a given manifest object. +// +// Note that it is possible that the object has not been created yet in the member cluster. +func (r *Reconciler) findInMemberClusterObjectFor( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + inMemberClusterObj, err := r.spokeDynamicClient. + Resource(*gvr). + Namespace(manifestObj.GetNamespace()). + Get(ctx, manifestObj.GetName(), metav1.GetOptions{}) + switch { + case err == nil: + // An object derived from the manifest object has been found in the member cluster. + klog.V(2).InfoS("Found the corresponding object for the manifest object in the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.inMemberClusterObj = inMemberClusterObj + return false + case apierrors.IsNotFound(err): + // The manifest object has never been applied before. + klog.V(2).InfoS("The manifest object has not been created in the member cluster yet", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + default: + // An unexpected error has occurred. + wrappedErr := errors.NewAPIServerError(err, + "failed to find the corresponding object for the manifest object in the member cluster", + false, // false as the dynamic client is non-caching. + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToFindObjInMemberCluster + klog.ErrorS(err, + "Failed to find the corresponding object for the manifest object in the member cluster", + errors.Args(wrappedErr)...) + return true + } +} + +// takeOverInMemberClusterObjectIfApplicable attempts to take over an object in the member cluster +// as needed. +func (r *Reconciler) takeOverInMemberClusterObjectIfApplicable( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + if isOwnedByOlderKubeFleetAPIObjects(inMemberClusterObj) { + wrappedErr := errors.NewUserError(nil, + "the object to process is already owned by another KubeFleet placement API object", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeOwnedByOtherKubeFleetAPIObjects + klog.ErrorS(wrappedErr, "Failed to check if takeover is needed", errors.Args(wrappedErr)...) + return true + } + + if !shouldInitiateTakeOverAttempt(manifestProcessingState) { + // Takeover is not necessary; proceed with the processing. + klog.V(2).InfoS("Takeover is not needed; skip the step", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } + + // Take over the object. Note that this steps adds only the owner reference; no other + // fields are modified (on the object from the member cluster). + takenOverInMemberClusterObj, configDiffs, diffCalculatedInDegradedMode, err := r.takeOverPreExistingObject(ctx, manifestProcessingState) + switch { + case err != nil: + // An unexpected error has occurred. + wrappedErr := errors.Wraps(err, "failed to take over a pre-existing object", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToTakeOver + klog.ErrorS(err, "Failed to complete the takeover process", errors.Args(wrappedErr)...) + return true + case len(configDiffs) > 0 && diffCalculatedInDegradedMode: + // Takeover cannot be performed as configuration differences are found between the manifest + // object and the object in the member cluster in degraded mode. + // + // Note that though degraded diff calculation itself is not considered as an error, the + // presence of diffs is indeed an error. + manifestProcessingState.diffs = configDiffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot take over object: configuration differences are found between the manifest object and the corresponding object in the member cluster in degraded mode (full comparison is performed instead of partial comparison, as the manifest object is considered to be invalid by the member cluster API server)", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFailedToTakeOver + klog.V(2).InfoS("Cannot take over object as configuration differences are found between the manifest object and the corresponding object in the member cluster in degraded mode", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + case len(configDiffs) > 0: + // Takeover cannot be performed as configuration differences are found between the manifest + // object and the object in the member cluster. + manifestProcessingState.diffs = configDiffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot take over object: configuration differences are found between the manifest object and the corresponding object in the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFailedToTakeOver + klog.V(2).InfoS("Cannot take over object as configuration differences are found between the manifest object and the corresponding object in the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + } + + // Takeover process is completed; track the newly refreshed object from the member cluster. + manifestProcessingState.inMemberClusterObj = takenOverInMemberClusterObj + klog.V(2).InfoS("The corresponding object has been taken over", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false +} + +// canApplyWithOwnership checks if KubeFleet can perform an apply op, knowing that KubeFleet has +// acquired the ownership of the object, or that the object has not been created yet. +// +// Note that this function does not concern co-ownership; such checks are executed elsewhere. +func canApplyWithOwnership(manifestProcessingState *manifestProcessingState) bool { + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + if inMemberClusterObj == nil { + // The object has not been created yet; KubeFleet can apply the object. + return true + } + + // Verify if the object is owned by KubeFleet. + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + if ownerreferences.AreEqual(&curOwners[idx], manifestProcessingState.ownedBy) { + return true + } + } + return false +} + +// performPreApplyDriftDetectionIfApplicable checks if pre-apply drift detection is needed and +// runs the drift detection process if applicable. +func (r *Reconciler) performPreApplyDriftDetectionIfApplicable( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + isPreApplyDriftDetectionNeeded, err := shouldPerformPreApplyDriftDetection(manifestProcessingState) + switch { + case err != nil: + // KubeFleet cannot determine if pre-apply drift detection is needed; this will only + // happen if the hash calculation process fails, specifically when KubeFleet cannot + // marshal the manifest object into its JSON representation. This should never happen, + // especially considering that the manifest object itself has been + // successfully decoded at this point of execution. + wrappedErr := errors.NewUnexpectedError(err, "failed to determine if pre-apply drift detection is needed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToRunDriftDetection + klog.ErrorS(err, "Failed to determine if pre-apply drift detection is needed", errors.Args(wrappedErr)...) + return true + case !isPreApplyDriftDetectionNeeded: + // Drift detection is not needed; proceed with the processing. + klog.V(2).InfoS("Pre-apply drift detection is not needed; skip the step", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } + + // Run the drift detection process. + diffs, driftsCalculatedInDegradedMode, err := r.diffBetweenManifestAndInMemberClusterObjects(ctx, + gvr, manifestObj, inMemberClusterObj, + manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy.ComparisonOption) + switch { + case err != nil: + // An unexpected error has occurred. + wrappedErr := errors.Wraps(err, "failed to calculate pre-apply drifts between the manifest and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToRunDriftDetection + klog.ErrorS(err, "Failed to complete pre-apply drift detection", errors.Args(wrappedErr)...) + return true + case len(diffs) > 0 && driftsCalculatedInDegradedMode: + // Configuration drifts are found in degraded mode. + // + // Note that though degraded drift calculation itself is not considered as an error, the + // presence of drifts in the pre-apply drift detection step is indeed an error. + manifestProcessingState.diffs = diffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot apply manifest: drifts are found between the manifest and the object from the member cluster in degraded mode (full comparison is performed instead of partial comparison, as the manifest object is considered to be invalid by the member cluster API server)", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFoundDriftsInDegradedMode + klog.V(2).InfoS("Cannot apply manifest: drifts are found between the manifest and the object from the member cluster in degraded mode", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + case len(diffs) > 0: + // Drifts are found in the pre-apply drift detection process. + manifestProcessingState.diffs = diffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot apply manifest: drifts are found between the manifest and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFoundDrifts + klog.V(2).InfoS("Cannot apply manifest: drifts are found between the manifest and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + default: + // No drifts are found in the pre-apply drift detection process; carry on with the apply op. + klog.V(2).InfoS("Pre-apply drift detection completed; no drifts are found", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } +} + +// shouldPerformPreApplyDriftDetection checks if pre-apply drift detection should be performed. +func shouldPerformPreApplyDriftDetection(manifestProcessingState *manifestProcessingState) (bool, error) { + // Drift detection is performed before the apply op if (and only if): + // * KubeFleet reports that the manifest has been applied before (i.e., inMemberClusterObj exists); and + // * The sync strategy dictates that a drift should be reported rather than overwritten; and + // * The hash of the manifest object is consistent with the last applied manifest object hash + // annotation on the corresponding resource in the member cluster (i.e., the same manifest + // object has been applied before). + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + if syncStrategy.WhenDrifted != placementv1alpha1.WhenDriftedOptionReportError || inMemberClusterObj == nil { + // A shortcut to save some overhead. + return false, nil + } + + cleanedManifestObj := discardFieldsIrrelevantInComparisonFrom(manifestProcessingState.manifestObj) + manifestObjHash, err := resource.HashOf(cleanedManifestObj.Object) + if err != nil { + return false, err + } + + inMemberClusterObjLastAppliedManifestObjHash := inMemberClusterObj.GetAnnotations()[placementv1alpha1.LastAppliedManifestHashAnnotationKey] + return manifestObjHash == inMemberClusterObjLastAppliedManifestObjHash, nil +} + +// performPostApplyDriftDetectionIfApplicable checks if post-apply drift detection is needed and +// runs the drift detection process if applicable. +func (r *Reconciler) performPostApplyDriftDetectionIfApplicable( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + + if !shouldPerformPostApplyDriftDetection(syncStrategy) { + // Post-apply drift detection is not needed; proceed with the processing. + klog.V(2).InfoS("Post-apply drift detection is not needed; skip the step", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } + + diffs, driftsCalculatedInDegradedMode, err := r.diffBetweenManifestAndInMemberClusterObjects(ctx, + gvr, manifestObj, inMemberClusterObj, syncStrategy.ComparisonOption) + switch { + case err != nil: + // An unexpected error has occurred. + // + // This case counts as a partial error; the apply op has been completed, but KubeFleet + // cannot determine if there are any drifts. + wrappedErr := errors.Wraps(err, "failed to calculate post-apply drifts between the manifest object and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeAppliedWithFailedDriftDetection + klog.ErrorS(err, + "Failed to complete post-apply drift detection", + errors.Args(wrappedErr)...) + return true + case len(diffs) > 0 && driftsCalculatedInDegradedMode: + // Configuration drifts are found, but they were calculated in degraded mode. + // + // Normally this should never happen, as post-apply drift detection will only run if the full + // comparison mode is enabled, but degraded mode only applies to partial comparison mode. + manifestProcessingState.diffs = diffs + klog.V(2).InfoS("Post-apply drift detection completed; drifts are found in degraded mode", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + // The presence of such drifts is not considered as an error. + return false + case len(diffs) > 0: + // Drifts are found in the post-apply drift detection process. + manifestProcessingState.diffs = diffs + klog.V(2).InfoS("Post-apply drift detection completed; drifts are found", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + // The presence of such drifts is not considered as an error. + return false + default: + // No drifts are found in the post-apply drift detection process. + klog.V(2).InfoS("Post-apply drift detection completed; no drifts are found", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } +} + +// shouldPerformPostApplyDriftDetection checks if post-apply drift detection should be performed. +func shouldPerformPostApplyDriftDetection(syncStrategy *placementv1alpha1.SyncStrategy) bool { + // Post-apply drift detection is performed if (and only if): + // * The sync strategy dictates that drift detection should run in full comparison mode. + return syncStrategy.ComparisonOption == placementv1alpha1.ComparisonOptionFullComparison +} diff --git a/pkg/v1/controllers/workapplier/retrieval.go b/pkg/v1/controllers/workapplier/retrieval.go new file mode 100644 index 000000000..8466e123e --- /dev/null +++ b/pkg/v1/controllers/workapplier/retrieval.go @@ -0,0 +1,201 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "strconv" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/fieldindexers" +) + +// retrieveLinkedAndLeftOverWorks retrieves two sets of work objects: +// +// a) linked work objects: these work objects share the same owner placement binding as the primary work object, and +// +// are all linked to the same primary placement resource snapshot as the primary work object. In other words, +// these work objects together form a consistent state of manifests that the work applier should process. +// +// b) leftover work objects: these work objects are also owned by the same placement binding as the primary work +// +// object, but are no longer linked to the current primary placement resource snapshot and are considered stale. +// All manifests in these work objects should be cleaned up, unless they are also referenced by another linked work +// object. +// +// This method will return an error if it observed any incomplete/inconsistent state among the two sets. Specifically, +// it will return a transient error when: +// +// a) the number of linked work objects does not match the linked work count recorded on the primary work object, or +// +// any of the linked work objects is not linked to the same primary placement resource snapshot as the primary +// work object. +// +// b) any of the leftover work objects hasn't been marked for deletion. +// +// Note that the first work object in the array of linked work objects is always the primary work object. +func (r *Reconciler) retrieveLinkedAndLeftOverWorks(ctx context.Context, + primaryWork *placementv1alpha1.Work) ([]*placementv1alpha1.Work, []*placementv1alpha1.Work, error) { + ownedBy := primaryWork.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if ownedBy == "" { + return nil, nil, errors.NewUnexpectedError(nil, "the primary work is missing the owner placement binding label") + } + // An empty owner namespace signals a cluster-scoped placement binding. + ownerNS := primaryWork.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + + wantLinkedWorkCountVal := primaryWork.GetAnnotations()[placementv1alpha1.LinkedWorkCountAnnotationKey] + wantLinkedWorkCount, err := strconv.Atoi(wantLinkedWorkCountVal) + if err != nil { + return nil, nil, errors.NewUnexpectedError(err, "failed to parse the linked work count annotation on the primary work", + "linkedWorkCount", wantLinkedWorkCountVal) + } + + wantPrimarySnapshotName := primaryWork.GetAnnotations()[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] + if wantPrimarySnapshotName == "" { + return nil, nil, errors.NewUnexpectedError(nil, "the primary work is missing the primary placement resource snapshot annotation") + } + + // List all work objects owned by the same placement binding as the primary work. + ownedByFieldVal := fmt.Sprintf(fieldindexers.WorkOwnedByPlacementBindingCustomFieldValFmt, ownerNS, ownedBy) + workList := &placementv1alpha1.WorkList{} + if err := r.hubClient.List(ctx, workList, + client.InNamespace(primaryWork.Namespace), + client.MatchingFields{fieldindexers.WorkOwnedByPlacementBindingCustomFieldName: ownedByFieldVal}, + ); err != nil { + return nil, nil, errors.NewAPIServerError(err, "failed to list the work objects owned by the placement binding", true, + "ownerPlacementBinding", ownedByFieldVal) + } + + var linkedWorks []*placementv1alpha1.Work + var leftOverWorks []*placementv1alpha1.Work + + linkedWorks = append(linkedWorks, primaryWork) + for idx := range workList.Items { + linkedWork := &workList.Items[idx] + primarySnapshotName := linkedWork.GetAnnotations()[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] + if primarySnapshotName != wantPrimarySnapshotName { + // This work is not linked to the same primary placement resource snapshot as the primary work; add it + // to the set of leftover works. + leftOverWorks = append(leftOverWorks, linkedWork) + continue + } + + if linkedWork.Name != primaryWork.Name { + // This work is linked to the same primary placement resource snapshot as the primary work; add it + // to the set of linked works. + linkedWorks = append(linkedWorks, linkedWork) + } + } + + // Verify that the number of linked work objects matches the count recorded on the primary work. + if len(linkedWorks) != wantLinkedWorkCount { + return nil, nil, errors.NewTransientError(nil, "the number of linked work objects found is inconsistent with the count recorded on the primary work", + "ownerPlacementBinding", ownedByFieldVal, + "observedLinkedWorkCount", len(linkedWorks), "wantLinkedWorkCount", wantLinkedWorkCount) + } + + // Verify that all leftover work objects have been marked for deletion. + for idx := range leftOverWorks { + leftOverWork := leftOverWorks[idx] + if leftOverWork.DeletionTimestamp == nil { + return nil, nil, errors.NewTransientError(nil, "a leftover work object has not been marked for deletion as expected", + "leftOverWork", klog.KObj(leftOverWork)) + } + } + + return linkedWorks, leftOverWorks, nil +} + +// addCleanupFinalizerTo adds the cleanup finalizer to the given work objects, so that the applied resources +// on the member cluster side can be cleaned up before the work objects are removed. +func (r *Reconciler) addCleanupFinalizerTo(ctx context.Context, linkedWorks []*placementv1alpha1.Work) error { + for idx := range linkedWorks { + linkedWork := linkedWorks[idx] + if controllerutil.ContainsFinalizer(linkedWork, workApplierCleanupFinalizer) { + continue + } + + controllerutil.AddFinalizer(linkedWork, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, linkedWork); err != nil { + // Reset the finalizer list to its previous state. + controllerutil.RemoveFinalizer(linkedWork, workApplierCleanupFinalizer) + return errors.NewAPIServerError(err, "failed to add the cleanup finalizer to the work", false, + "work", klog.KObj(linkedWork)) + } + klog.V(2).InfoS("Added the cleanup finalizer to the work", "work", klog.KObj(linkedWork)) + } + return nil +} + +func (r *Reconciler) ensureAppliedWorks(ctx context.Context, works []*placementv1alpha1.Work) ([]*placementv1alpha1.AppliedWork, error) { + appliedWorks := make([]*placementv1alpha1.AppliedWork, 0, len(works)) + for idx := range works { + work := works[idx] + + // Check if an appliedWork object already exists for the primary work object. + // + // Since we only create an appliedWork object after adding the finalizer to the primary work object, + // it might appear that if the finalizer is absent, the appliedWork object should not exist. This + // is not the case with the work applier though, as the controller features a + // Leave method that will strip all work objects off their finalizers, which is called when the + // member cluster leaves the fleet. If the member cluster chooses to later re-join the fleet, the controller + // might see a work object with no finalizer but with an appliedWork object. Because of this, here we always + // check for the existence of the appliedWork object, with or without the finalizer. + appliedWork := &placementv1alpha1.AppliedWork{} + err := r.spokeClient.Get(ctx, types.NamespacedName{Name: work.Name}, appliedWork) + switch { + case err == nil: + // The AppliedWork already exists; no further action is needed. + klog.V(2).InfoS("Found an appliedWork object for the work object", "appliedWork", klog.KObj(appliedWork)) + appliedWorks = append(appliedWorks, appliedWork) + continue + case !apierrors.IsNotFound(err): + return nil, errors.NewAPIServerError(err, "failed to retrieve the appliedWork object", true) + } + + // The appliedWork object does not exist; create one. + appliedWork = &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{ + Name: work.Name, + }, + Spec: placementv1alpha1.AppliedWorkSpec{ + WorkName: work.Name, + WorkNamespace: work.Namespace, + }, + } + if err := r.spokeClient.Create(ctx, appliedWork); err != nil { + // Note that the controller must retry on AppliedWork AlreadyExists errors; otherwise the + // controller will run the reconciliation loop with an AppliedWork that has no UID, + // which might lead to takeover failures in later steps. + return nil, errors.NewAPIServerError(err, "failed to create an appliedWork object for the work object", false, + "appliedWork", klog.KObj(appliedWork)) + } + klog.V(2).InfoS("Created an appliedWork object for the work object", "appliedWork", klog.KObj(appliedWork)) + appliedWorks = append(appliedWorks, appliedWork) + } + + return appliedWorks, nil +} diff --git a/pkg/v1/controllers/workapplier/status.go b/pkg/v1/controllers/workapplier/status.go new file mode 100644 index 000000000..b08b55dfd --- /dev/null +++ b/pkg/v1/controllers/workapplier/status.go @@ -0,0 +1,528 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + + "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +// refreshWorkStatus refreshes the status of a Work object based on the processing results of its manifests. +func (r *Reconciler) refreshWorkStatus( + ctx context.Context, + workObjProcessingStates []*workObjectProcessingState, +) error { + // Note (chenyu1): it is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of work objects would be low, currently + // KubeFleet still does the status refresh sequentially. + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + if err := r.refreshOneWorkStatus(ctx, workObjProcessingState); err != nil { + wrappedErr := errors.Wraps(err, "", "work", klog.KObj(workObjProcessingState.work)) + return wrappedErr + } + } + return nil +} + +func (r *Reconciler) refreshOneWorkStatus( + ctx context.Context, + workObjProcessingState *workObjectProcessingState, +) error { + work := workObjProcessingState.work + statusCopy := work.Status.DeepCopy() + + // Note (chenyu1): it is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of manifests would be low, currently + // KubeFleet still does the status refresh sequentially. + + // Set up the counters. + manifestCount := len(workObjProcessingState.manifestProcessingStates) + appliedManifestsCount := 0 + availableAppliedObjectsCount := 0 + untrackableAppliedObjectsCount := 0 + + // Use the now timestamp as the observation time. + now := metav1.Now() + + // Build an empty list of per-manifest statuses and port back existing per-manifest statuses to the list + // as applicable. + // + // This step is necessary at the moment primarily because the manifest status features `metav1.Condition`, + // the LastTransitionTime field of which requires that KubeFleet track the last known condition. + // + // Note that the refreshed manifest status array is pre-allocated and is of the same length as the number of + // manifest processing states. + refreshedManifestStatuses, refreshedManifestStatusesIndex := prepareRefreshedManifestStatusWithIndex(workObjProcessingState.manifestProcessingStates) + portBackExistingManifestStatuses(statusCopy.Manifests, refreshedManifestStatuses, refreshedManifestStatusesIndex) + + // Refresh per-manifest statuses. + for idx := range workObjProcessingState.manifestProcessingStates { + manifestProcessingState := workObjProcessingState.manifestProcessingStates[idx] + refreshOneManifestStatus(manifestProcessingState, &refreshedManifestStatuses[idx], now) + + // Tally the stats. + if isManifestObjectApplied(manifestProcessingState.applyRes) { + appliedManifestsCount++ + } + if isAppliedObjectAvailable(manifestProcessingState.availabilityCheckRes) { + availableAppliedObjectsCount++ + } + if manifestProcessingState.availabilityCheckRes == AvailabilityResultTypeNotTrackable { + untrackableAppliedObjectsCount++ + } + } + statusCopy.Manifests = refreshedManifestStatuses + + // Set work object status conditions. + + // Do a sanity check. + if appliedManifestsCount > manifestCount || + availableAppliedObjectsCount > manifestCount || + untrackableAppliedObjectsCount > manifestCount { + // Normally this should never happen. + return errors.NewUnexpectedError(nil, + "the number of applied manifests, available applied objects, or untrackable applied objects exceeds the total number of manifests", + "appliedManifestsCount", appliedManifestsCount, + "availableAppliedObjectsCount", availableAppliedObjectsCount, + "untrackableAppliedObjectsCount", untrackableAppliedObjectsCount, + "manifestCount", manifestCount) + } + if statusCopy.Conditions == nil { + statusCopy.Conditions = []metav1.Condition{} + } + setWorkAppliedCondition(statusCopy, manifestCount, appliedManifestsCount, work.Generation) + setWorkAvailableCondition(statusCopy, manifestCount, availableAppliedObjectsCount, untrackableAppliedObjectsCount, work.Generation) + + // TO-DO (chenyu1): if needed, trim the reported status so that the work object will not grow over Kubernetes' + // object size limit. + if shouldSkipStatusUpdate(&work.Status, statusCopy) { + // No status change found; skip the update. + klog.V(2).InfoS("No status change found for Work object; skip the status update", "work", klog.KObj(work)) + } else { + klog.V(2).InfoS("Refreshing work object status", "work", klog.KObj(work)) + + work.Status = *statusCopy + if err := r.hubClient.Status().Update(ctx, work); err != nil { + return errors.NewAPIServerError(err, "failed to update work object status", false, "work", klog.KObj(work)) + } + } + + return nil +} + +func prepareRefreshedManifestStatusWithIndex(manifestProcessingStates []*manifestProcessingState) ( + []placementv1alpha1.PerManifestStatus, map[string]int) { + // Pre-allocate the status slice. + refreshedManifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestProcessingStates)) + + refreshedManifestStatusesIndex := make(map[string]int, len(manifestProcessingStates)) + for idx := range manifestProcessingStates { + manifestProcessingState := manifestProcessingStates[idx] + idStr := manifestProcessingState.idStr + + if len(manifestProcessingState.idStr) == 0 { + // There might be manifests without a valid identifier in the bundle set + // (e.g., a decoding error has occurred when processing a bundle). + // KubeFleet will skip these bundles, as there is no need to port back + // information for such manifests any way for obvious reasons (manifest itself is not + // identifiable). This is not considered as an error. + continue + } + refreshedManifestStatusesIndex[idStr] = idx + } + return refreshedManifestStatuses, refreshedManifestStatusesIndex +} + +func portBackExistingManifestStatuses( + existingManifestStatuses []placementv1alpha1.PerManifestStatus, + refreshedManifestStatuses []placementv1alpha1.PerManifestStatus, + refreshedManifestStatusesIndex map[string]int, +) { + for idx := range existingManifestStatuses { + existingManifestStatus := existingManifestStatuses[idx] + existingManifestId := &existingManifestStatus.Identifier + if existingManifestId.Kind == "" || + existingManifestId.Resource == "" || + existingManifestId.Name == "" { + // It is OK for an existing manifest status to not have a valid identifier; this + // happens when the manifest status was previously associated with a manifest + // that cannot be decoded. For obvious reasons KubeFleet does not need to port back + // such manifest conditions any way. + continue + } + existingManifestIdStr := formatManifestIdentifierStr(existingManifestId) + + if idx, found := refreshedManifestStatusesIndex[existingManifestIdStr]; found { + // A match can be found via the index; port back the existing manifest status. + refreshedManifestStatuses[idx] = *existingManifestStatus.DeepCopy() + } + } +} + +func refreshOneManifestStatus(manifestProcessingState *manifestProcessingState, + manifestStatusToRefresh *placementv1alpha1.PerManifestStatus, + now metav1.Time, +) { + // Populate the identifier in the manifest status. + // + // If the status has been ported back, the identifier written here is guaranteed to match the existing one. + manifestStatusToRefresh.Identifier = *manifestProcessingState.id.DeepCopy() + + // Prepare the list of conditions in the manifest status. + if manifestStatusToRefresh.Conditions == nil { + manifestStatusToRefresh.Conditions = []metav1.Condition{} + } + + // Set the conditions. + // + // Note that per API definition, the observed generation of a manifest condition is that + // of the applied resource, not that of the Work object. + var inMemberClusterObjGeneration *int64 + if manifestProcessingState.inMemberClusterObj != nil { + inMemberClusterObjGeneration = ptr.To(manifestProcessingState.inMemberClusterObj.GetGeneration()) + } + setManifestStatusAppliedCondition(manifestStatusToRefresh, manifestProcessingState.applyRes, manifestProcessingState.applyErr, inMemberClusterObjGeneration) + setManifestStatusAvailableCondition(manifestStatusToRefresh, manifestProcessingState.availabilityCheckRes, manifestProcessingState.availabilityCheckErr, inMemberClusterObjGeneration) + setManifestStatusDiffDetails(manifestStatusToRefresh, manifestProcessingState.diffs, inMemberClusterObjGeneration, now) +} + +// setManifestStatusAppliedCondition sets the Applied condition on a manifest status. +func setManifestStatusAppliedCondition( + manifestStatus *placementv1alpha1.PerManifestStatus, + applyRes ApplyResultType, + applyErr error, + inMemberClusterObjGeneration *int64, +) { + // If the in-cluster object generation is not available (i.e., there is no in-cluster object), use the default + // value 0. + observedGeneration := int64(0) + if inMemberClusterObjGeneration != nil { + observedGeneration = *inMemberClusterObjGeneration + } + + var appliedCond metav1.Condition + switch { + case applyRes == ApplyResTypeApplied: + // The manifest has been successfully applied. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + Message: ApplyResTypeAppliedDescription, + ObservedGeneration: observedGeneration, + } + case applyRes == ApplyResTypeAppliedWithFailedDriftDetection: + // The manifest has been successfully applied, but drift detection has failed. + // + // At this moment KubeFleet does not prepare a dedicated condition for drift detection outcomes. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeAppliedWithFailedDriftDetection), + Message: ApplyResTypeAppliedWithFailedDriftDetectionDescription, + ObservedGeneration: observedGeneration, + } + case !allApplyResTypes.Has(applyRes): + // Do a sanity check; verify if the returned result type is a valid one. + // Normally this branch should never run. + wrappedErr := errors.NewUnexpectedError(applyErr, "found an unexpected apply result type", + "manifestID", manifestStatus.Identifier, "applyRes", applyRes) + klog.ErrorS(wrappedErr, "Failed to set the Applied condition", errors.Args(wrappedErr)...) + // The work applier will consider this to be an apply failure. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToApply), + Message: fmt.Sprintf("An unexpected apply result is returned (%s) with error (%s)", applyRes, applyErr.Error()), + ObservedGeneration: observedGeneration, + } + default: + // The apply op fails. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(applyRes), + Message: fmt.Sprintf(ApplyResTypeFailedToApplyDescription, applyErr), + ObservedGeneration: observedGeneration, + } + } + + meta.SetStatusCondition(&manifestStatus.Conditions, appliedCond) +} + +// setManifestStatusAvailableCondition sets the Available condition on a manifest status. +func setManifestStatusAvailableCondition( + manifestStatus *placementv1alpha1.PerManifestStatus, + availabilityCheckRes AvailabilityCheckResultType, + availabilityCheckErr error, + inMemberClusterObjGeneration *int64, +) { + // If the in-cluster object generation is not available (i.e., there is no in-cluster object), use the default + // value 0. + observedGeneration := int64(0) + if inMemberClusterObjGeneration != nil { + observedGeneration = *inMemberClusterObjGeneration + } + + var availableCond *metav1.Condition + switch availabilityCheckRes { + case AvailabilityResultTypeSkipped: + // The availability check has been skipped as the manifest has not been applied yet; in this + // case no Available condition is set. + case AvailabilityResultTypeFailed: + // The availability check has failed. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeFailed), + Message: fmt.Sprintf(AvailabilityResultTypeFailedDescription, availabilityCheckErr), + ObservedGeneration: observedGeneration, + } + case AvailabilityResultTypeNotYetAvailable: + // The manifest is not yet available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + Message: AvailabilityResultTypeNotYetAvailableDescription, + ObservedGeneration: observedGeneration, + } + case AvailabilityResultTypeNotTrackable: + // KubeFleet cannot track the availability of the manifest. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeNotTrackable), + Message: AvailabilityResultTypeNotTrackableDescription, + ObservedGeneration: observedGeneration, + } + default: + // The manifest is available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + Message: AvailabilityResultTypeAvailableDescription, + ObservedGeneration: observedGeneration, + } + } + + if availableCond == nil { + // The conditions might be ported back from the previous run; a stale Available condition must be + // removed when no new one is set. + meta.RemoveStatusCondition(&manifestStatus.Conditions, placementv1alpha1.ManifestCondTypeAvailable) + return + } + meta.SetStatusCondition(&manifestStatus.Conditions, *availableCond) +} + +// setManifestStatusDiffDetails sets the diff details on a manifest status. +func setManifestStatusDiffDetails( + manifestStatus *placementv1alpha1.PerManifestStatus, + diffs []placementv1alpha1.PatchDetail, + observedInMemberClusterGeneration *int64, + now metav1.Time, +) { + // Check if a first diffed timestamp has been set; if not, set it to the current time. + firstDiffedTimestamp := now + if manifestStatus.DiffDetails != nil && !manifestStatus.DiffDetails.FirstDiffedObservedTimestamp.IsZero() { + firstDiffedTimestamp = manifestStatus.DiffDetails.FirstDiffedObservedTimestamp + } + + // Reset the diff details (such details need no port-back). + manifestStatus.DiffDetails = nil + if len(diffs) == 0 { + return + } + + // Populate the diff details. + manifestStatus.DiffDetails = &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: observedInMemberClusterGeneration, + FirstDiffedObservedTimestamp: firstDiffedTimestamp, + ObservedDiffs: diffs, + } +} + +// isAppliedObjectAvailable returns whether an availability check result type indicates that an +// applied manifest object is available. +func isAppliedObjectAvailable(availabilityCheckRes AvailabilityCheckResultType) bool { + return availabilityCheckRes == AvailabilityResultTypeAvailable || availabilityCheckRes == AvailabilityResultTypeNotTrackable +} + +// setWorkAppliedCondition sets the Applied condition on a work object status. +// +// A work object is considered to be applied if all of its manifests have been successfully applied. +func setWorkAppliedCondition( + workStatus *placementv1alpha1.WorkStatus, + manifestCount, appliedManifestCount int, + workGeneration int64, +) { + var appliedCond metav1.Condition + switch { + case appliedManifestCount == manifestCount: + // All manifests have been successfully applied. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + // Here KubeFleet reuses the same reason for individual manifests. + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + Message: fmt.Sprintf("All %d manifest(s) have been successfully applied", manifestCount), + ObservedGeneration: workGeneration, + } + default: + // Not all manifests have been successfully applied. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + Message: fmt.Sprintf("%d out of %d manifest(s) have not yet been applied", manifestCount-appliedManifestCount, manifestCount), + ObservedGeneration: workGeneration, + } + } + + meta.SetStatusCondition(&workStatus.Conditions, appliedCond) + klog.V(2).InfoS("Applied condition set on the work object status", + "appliedManifestCount", appliedManifestCount, "manifestCount", manifestCount) +} + +// setWorkAvailableCondition sets the Available condition on a work object status. +// +// A work object is considered to be available if all of its applied manifests are available. +func setWorkAvailableCondition( + workStatus *placementv1alpha1.WorkStatus, + manifestCount, availableManifestCount, untrackableAppliedObjectsCount int, + workGeneration int64, +) { + var availableCond *metav1.Condition + switch { + case availableManifestCount == manifestCount && untrackableAppliedObjectsCount == 0: + // All manifests are available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + Message: fmt.Sprintf("All %d manifest(s) are available", manifestCount), + ObservedGeneration: workGeneration, + } + case availableManifestCount == manifestCount: + // Some manifests are not trackable. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + Message: fmt.Sprintf("All %d manifest(s) are available (with %d manifests untrackable)", manifestCount, untrackableAppliedObjectsCount), + ObservedGeneration: workGeneration, + } + default: + // Not all manifests are available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + Message: fmt.Sprintf("%d out of %d manifest(s) are not yet available", manifestCount-availableManifestCount, manifestCount), + ObservedGeneration: workGeneration, + } + } + + meta.SetStatusCondition(&workStatus.Conditions, *availableCond) + klog.V(2).InfoS("Available condition set on the work object status", + "availableManifestCount", availableManifestCount, + "untrackableAppliedObjectsCount", untrackableAppliedObjectsCount, "manifestCount", manifestCount) +} + +func shouldSkipStatusUpdate(oldStatus, newStatus *placementv1alpha1.WorkStatus) bool { + // Skip status update if there is no change in the status. + return equality.Semantic.DeepEqual(oldStatus, newStatus) +} + +// refreshAppliedWorkStatus refreshes the status of appliedWork objects. +func (r *Reconciler) refreshAppliedWorkStatus( + ctx context.Context, + workObjProcessingStates []*workObjectProcessingState, +) error { + // Note (chenyu1): it is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of work objects would be low, currently + // KubeFleet still does the status refresh sequentially. + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + if err := r.refreshOneAppliedWorkStatus(ctx, workObjProcessingState); err != nil { + wrappedErr := errors.Wraps(err, "", "appliedWork", klog.KObj(workObjProcessingState.appliedWork)) + return wrappedErr + } + } + return nil +} + +func (r *Reconciler) refreshOneAppliedWorkStatus( + ctx context.Context, + workObjProcessingState *workObjectProcessingState, +) error { + appliedWork := workObjProcessingState.appliedWork + appliedWorkStatusCopy := appliedWork.Status.DeepCopy() + + manifestProcessingStates := workObjProcessingState.manifestProcessingStates + + // Note (chenyu1): It is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of manifests would be low, currently + // KubeFleet still does the status refresh sequentially. + + // Pre-allocate the slice. + // + // Manifests that failed to get applied are not included in this list, hence + // empty length. + appliedResources := make([]placementv1alpha1.AppliedResource, 0, len(manifestProcessingStates)) + + // Build the list of applied resources. + for idx := range manifestProcessingStates { + manifestProcessingState := manifestProcessingStates[idx] + + if isManifestObjectApplied(manifestProcessingState.applyRes) { + appliedResources = append(appliedResources, placementv1alpha1.AppliedResource{ + ManifestIdentifier: *manifestProcessingState.id, + UID: manifestProcessingState.inMemberClusterObj.GetUID(), + }) + } + } + + // Update the AppliedWork object status. + appliedWorkStatusCopy.AppliedResources = appliedResources + + // Skip the status update if no change found. + if equality.Semantic.DeepEqual(&appliedWork.Status, appliedWorkStatusCopy) { + klog.V(2).InfoS("No status change found for appliedWork object; skip the status update", "appliedWork", klog.KObj(appliedWork)) + } else { + klog.V(2).InfoS("Refreshing appliedWork object status", "appliedWork", klog.KObj(appliedWork)) + + appliedWork.Status = *appliedWorkStatusCopy + if err := r.spokeClient.Status().Update(ctx, appliedWork); err != nil { + klog.ErrorS(err, "Failed to update appliedWork status", + "appliedWork", klog.KObj(appliedWork)) + return errors.NewAPIServerError(err, "failed to update appliedWork object status", false, "appliedWork", klog.KObj(appliedWork)) + } + } + return nil +} diff --git a/pkg/v1/controllers/workapplier/suite_test.go b/pkg/v1/controllers/workapplier/suite_test.go new file mode 100644 index 000000000..2ce6f9b1b --- /dev/null +++ b/pkg/v1/controllers/workapplier/suite_test.go @@ -0,0 +1,216 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "flag" + "path/filepath" + "sync" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/cache" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/parallelizer" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/fieldindexers" + testv1alpha1 "github.com/kubefleet-dev/kubefleet/test/apis/v1alpha1" +) + +// These tests use Ginkgo (BDD-style Go testing framework). Refer to +// http://onsi.github.io/ginkgo/ to learn more about Ginkgo. +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubClient client.Client + + memberCfg1 *rest.Config + memberEnv1 *envtest.Environment + hubMgr1 manager.Manager + memberClient1 client.Client + memberDynamicClient1 dynamic.Interface + workApplier1 *Reconciler + + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup +) + +const ( + // The number of max. concurrent reconciliations for the work applier controller. + maxConcurrentReconciles = 5 + // The count of workers for the work applier controller. + workerCount = 4 + + // The requeue delay when the work applier waits for the cleanup (appliedWork deletion) to complete. + cleanupRequeueAfter = time.Second * 2 + // The requeue delay for the periodic re-processing of work objects. + periodicRequeueAfter = time.Second * 3 + // The max. time the work applier waits for an appliedWork object to be deleted before it unblocks + // the deletion manually. + // + // Note that envtest runs no GC controller; foreground deletions of appliedWork objects never complete + // in the test environments, and the work applier will always wait for 2 * cleanupWaitTime before + // it removes the cleanup finalizers. A short wait time is used here to keep the tests fast. + cleanupWaitTime = time.Second * 5 + + memberReservedNSName1 = "fleet-member-experimental-1" +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Work Applier Integration Test Suite") +} + +func setupResources() { + ns1 := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: memberReservedNSName1, + }, + } + Expect(hubClient.Create(ctx, ns1)).To(Succeed()) +} + +// Note: each Ginkgo process must do the same setup; unlike our E2E tests, the integration +// tests uses in-memory testing environments, and as a result cannot be shared across processes. +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + logger := zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)) + klog.SetLogger(logger) + ctrl.SetLogger(logger) + + By("Bootstrapping test environments") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{ + filepath.Join("../../../../", "config", "crd", "bases"), + filepath.Join("../../../../", "test", "manifests"), + }, + ErrorIfCRDPathMissing: true, + } + // memberEnv1 is the test environment for verifying most work applier behaviors. + memberEnv1 = &envtest.Environment{ + CRDDirectoryPaths: []string{ + filepath.Join("../../../../", "config", "crd", "bases"), + filepath.Join("../../../../", "test", "manifests"), + }, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).ToNot(HaveOccurred()) + Expect(hubCfg).ToNot(BeNil()) + + memberCfg1, err = memberEnv1.Start() + Expect(err).ToNot(HaveOccurred()) + Expect(memberCfg1).ToNot(BeNil()) + + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed()) + Expect(testv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed()) + + By("Building the K8s clients") + hubClient, err = client.New(hubCfg, client.Options{Scheme: scheme.Scheme}) + Expect(err).ToNot(HaveOccurred()) + Expect(hubClient).ToNot(BeNil()) + + memberClient1, err = client.New(memberCfg1, client.Options{Scheme: scheme.Scheme}) + Expect(err).ToNot(HaveOccurred()) + Expect(memberClient1).ToNot(BeNil()) + + memberDynamicClient1, err = dynamic.NewForConfig(memberCfg1) + Expect(err).ToNot(HaveOccurred()) + + By("Setting up the resources") + setupResources() + + By("Setting up the controller and the controller manager for member cluster 1") + hubMgr1, err = ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: server.Options{ + BindAddress: "0", + }, + Cache: cache.Options{ + DefaultNamespaces: map[string]cache.Config{ + memberReservedNSName1: {}, + }, + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).ToNot(HaveOccurred()) + + // The work applier lists work objects via custom field indexes; the indexes must be set up + // (with the manager's cache) before the manager starts. + Expect(fieldindexers.SetupWithMemberAgentControllerManager(ctx, hubMgr1)).To(Succeed()) + + workApplier1 = New( + memberReservedNSName1, + // Use the cached client from the manager, as the work applier relies on the field indexes. + hubMgr1.GetClient(), + hubMgr1.GetAPIReader(), + memberClient1, + memberDynamicClient1, + memberClient1.RESTMapper(), + parallelizer.NewParallelizer(workerCount), + maxConcurrentReconciles, + cleanupRequeueAfter, + periodicRequeueAfter, + cleanupWaitTime, + ) + Expect(workApplier1.SetupWithManager(hubMgr1)).To(Succeed()) + + wg = sync.WaitGroup{} + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + Expect(workApplier1.Join()).To(Succeed()) + Expect(hubMgr1.Start(ctx)).To(Succeed()) + }() +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + wg.Wait() + By("Tearing down the test environment") + Expect(hubEnv.Stop()).To(Succeed()) + Expect(memberEnv1.Stop()).To(Succeed()) +}) diff --git a/pkg/v1/controllers/workapplier/takeover_drift_detection.go b/pkg/v1/controllers/workapplier/takeover_drift_detection.go new file mode 100644 index 000000000..433cc3973 --- /dev/null +++ b/pkg/v1/controllers/workapplier/takeover_drift_detection.go @@ -0,0 +1,573 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "strings" + + "github.com/qri-io/jsonpointer" + "github.com/wI2L/jsondiff" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + kfioplacementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/ownerreferences" +) + +const ( + patchDetailPerObjLimit = 100 +) + +const ( + k8sReservedLabelAnnotationFullDomain = "kubernetes.io/" + k8sReservedLabelAnnotationAbbrDomain = "k8s.io/" + kubefleetReservedLabelAnnotationDomain = "kubefleet.dev/" +) + +// isOwnedByOlderKubeFleetAPIObjects checks if the object in the member cluster is owned by older KubeFleet API +// objects, specifically the appliedWork objects in the `placement.kubernetes-fleet.io` API group. +// +// This is added as a safeguard to avoid scenarios where a manifest is applied via both the newer PlacementPolicy +// APIs and the older ResourcePlacement APIs. And the safeguard always runs regardless of the sync strategy in use. +func isOwnedByOlderKubeFleetAPIObjects(inMemberClusterObj *unstructured.Unstructured) bool { + if inMemberClusterObj == nil { + return false + } + + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + owner := &curOwners[idx] + if owner.APIVersion == kfioplacementv1beta1.GroupVersion.String() && owner.Kind == "AppliedWork" { + return true + } + } + return false +} + +// shouldInitiateTakeOverAttempt checks if KubeFleet should initiate the takeover process for an object. +// +// A takeover process is initiated when: +// +// - An object that matches with the given manifest has been created; but +// - The object is not owned by KubeFleet (more specifically, the object is not owned by the +// expected AppliedWork object). +func shouldInitiateTakeOverAttempt(manifestProcessingState *manifestProcessingState) bool { + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + if inMemberClusterObj == nil { + // Obviously, if the corresponding live object is not found, no takeover is + // needed. + return false + } + + // Skip the takeover process if the sync strategy forbids so. + if manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy.WhenAlreadyExists == placementv1alpha1.WhenAlreadyExistsOptionReportError { + return false + } + + // Check if the live object is owned by KubeFleet. + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + if ownerreferences.AreEqual(&curOwners[idx], manifestProcessingState.ownedBy) { + // The live object is owned by KubeFleet; no takeover is needed. + return false + } + } + return true +} + +// takeOverPreExistingObject takes over a pre-existing object in the member cluster. +func (r *Reconciler) takeOverPreExistingObject( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (*unstructured.Unstructured, []placementv1alpha1.PatchDetail, bool, error) { + gvr := manifestProcessingState.gvr + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + expectedAppliedWorkOwnerRef := manifestProcessingState.ownedBy + + inMemberClusterObjCopy := manifestProcessingState.inMemberClusterObj.DeepCopy() + existingOwnerRefs := inMemberClusterObjCopy.GetOwnerReferences() + + // At this moment KubeFleet is set to leave applied resources in the member cluster if the cluster + // decides to leave its fleet; when this happens, the resources will be owned by an AppliedWork + // object that might not have a corresponding Work object anymore as the cluster might have + // been de-selected. If the cluster decides to re-join the fleet, and the same set of manifests + // are being applied again, one may encounter unexpected errors due to the presence of the + // left behind AppliedWork object. To address this issue, KubeFleet here will perform a cleanup, + // removing any owner reference that points to an orphaned AppliedWork object. + existingOwnerRefs, err := r.removeLeftBehindAppliedWorkOwnerRefs(ctx, existingOwnerRefs) + if err != nil { + return nil, nil, false, errors.Wraps(err, "failed to remove left-behind AppliedWork owner references") + } + + // Check this object is already owned by another object (or controller); if so, KubeFleet will only + // add itself as an additional owner if co-ownership is allowed. + if len(existingOwnerRefs) >= 1 && syncStrategy.WhenOwnedByOthers != placementv1alpha1.WhenOwnedByOthersOptionShareOwnership { + // The object is already owned by another object, and co-ownership is forbidden. + // No takeover will be performed. + // + // Note that this will be registered as an (apply) error. + return nil, nil, false, errors.NewUserError(nil, + "the object is already owned by some other source(s) and co-ownership is disallowed", + "existingOwnerRefs", existingOwnerRefs) + } + + // Check if the object is already owned by KubeFleet, but the owner is a different AppliedWork + // object, i.e., the object has been placed in duplicate. + // + // Originally KubeFleet does allow placing the same object multiple times; each placement + // attempt might have its own object spec (envelopes might be used and in each envelope + // the object looks different). This would lead to the object being constantly overwritten, + // yet no error would be raised on the user-end. With the drift detection feature now added to KubeFleet, however, + // this scenario would repeatedly trigger drift-related alarms, which could lead to user confusion. + // To address this corner case, KubeFleet would now refuse to place the same object twice. + if isPlacedByKubeFleetInDuplicate(existingOwnerRefs, expectedAppliedWorkOwnerRef) { + return nil, nil, false, errors.NewUserError(nil, "the object is already owned by another KubeFleet AppliedWork object") + } + + // Check if the takeover action requires additional steps (configuration difference inspection). + if syncStrategy.WhenAlreadyExists == placementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff { + configDiffs, diffCalculatedInDegradedMode, err := r.diffBetweenManifestAndInMemberClusterObjects(ctx, + gvr, manifestProcessingState.manifestObj, inMemberClusterObjCopy, syncStrategy.ComparisonOption) + switch { + case err != nil: + return nil, nil, false, errors.Wraps(err, "failed to calculate configuration diffs between the manifest object and the object from the member cluster") + case len(configDiffs) > 0: + return nil, configDiffs, diffCalculatedInDegradedMode, nil + } + } + + // Take over the object. + updatedOwnerRefs := append(existingOwnerRefs, *expectedAppliedWorkOwnerRef) + inMemberClusterObjCopy.SetOwnerReferences(updatedOwnerRefs) + takenOverInMemberClusterObj, err := r.spokeDynamicClient. + Resource(*gvr).Namespace(inMemberClusterObjCopy.GetNamespace()). + Update(ctx, inMemberClusterObjCopy, metav1.UpdateOptions{}) + if err != nil { + // false as the dynamic client is non-caching. + return nil, nil, false, errors.NewAPIServerError(err, "failed to take over the object", false) + } + + return takenOverInMemberClusterObj, nil, false, nil +} + +// removeLeftBehindAppliedWorkOwnerRefs removes owner references that point to orphaned AppliedWork objects. +func (r *Reconciler) removeLeftBehindAppliedWorkOwnerRefs(ctx context.Context, ownerRefs []metav1.OwnerReference) ([]metav1.OwnerReference, error) { + updatedOwnerRefs := make([]metav1.OwnerReference, 0, len(ownerRefs)) + for idx := range ownerRefs { + ownerRef := ownerRefs[idx] + if ownerRef.APIVersion != placementv1alpha1.GroupVersion.String() || ownerRef.Kind != placementv1alpha1.AppliedWorkKind { + // Skip non-appliedWork owner references. + updatedOwnerRefs = append(updatedOwnerRefs, ownerRef) + continue + } + + // Check if the appliedWork object has a corresponding work object. + workObj := &placementv1alpha1.Work{} + err := r.hubClient.Get(ctx, types.NamespacedName{Namespace: r.workNSName, Name: ownerRef.Name}, workObj) + switch { + case err != nil && !apierrors.IsNotFound(err): + // An unexpected error occurred. + return nil, errors.NewAPIServerError(err, "failed to retrieve the work object", true, + "workNamespace", r.workNSName, "workName", ownerRef.Name) + case err == nil: + // The appliedWork owner reference is valid; no need for removal. + // + // Note that no UID check is performed here; KubeFleet can (and will) reuse the same appliedWork + // as long as it has the same name as a work object, even if the appliedWork object is not + // originally derived from it. This is safe as the appliedWork object is in essence a delegate + // and does not keep any additional information. + updatedOwnerRefs = append(updatedOwnerRefs, ownerRef) + continue + default: + // The appliedWork owner reference is invalid; the work object does not exist. + // + // Remove the owner reference. + klog.V(2).InfoS("Found an owner reference that points to an orphaned AppliedWork object", "ownerRef", ownerRef) + continue + } + } + + return updatedOwnerRefs, nil +} + +// isPlacedByKubeFleetInDuplicate checks if the object has already been placed by KubeFleet via another +// placement policy. +func isPlacedByKubeFleetInDuplicate(ownerRefs []metav1.OwnerReference, expectedAppliedWorkOwnerRef *metav1.OwnerReference) bool { + for idx := range ownerRefs { + ownerRef := ownerRefs[idx] + if ownerRef.APIVersion == placementv1alpha1.GroupVersion.String() && + ownerRef.Kind == placementv1alpha1.AppliedWorkKind && + ownerRef.UID != expectedAppliedWorkOwnerRef.UID { + return true + } + } + return false +} + +// diffBetweenManifestAndInMemberClusterObjects calculates the differences between the manifest object +// and its corresponding object in the member cluster. +func (r *Reconciler) diffBetweenManifestAndInMemberClusterObjects( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, + cmpOption placementv1alpha1.ComparisonOption, +) ([]placementv1alpha1.PatchDetail, bool, error) { + switch cmpOption { + case placementv1alpha1.ComparisonOptionPartialComparison: + return r.partialDiffBetweenManifestAndInMemberClusterObjects(ctx, gvr, manifestObj, inMemberClusterObj) + case placementv1alpha1.ComparisonOptionFullComparison: + // For the full comparison, KubeFleet compares directly the JSON representations of the + // manifest object and the object in the member cluster. + patchDetails, err := preparePatchDetails(manifestObj, inMemberClusterObj) + return patchDetails, false, err + default: + return nil, false, errors.NewUnexpectedError(nil, "an invalid comparison option is specified", "comparisonOption", cmpOption) + } +} + +// partialDiffBetweenManifestAndInMemberClusterObjects calculates the differences between the +// manifest object and its corresponding object in the member cluster by performing a dry-run +// apply op; this would ignore differences in the unmanaged fields and report only those +// in the managed fields. +func (r *Reconciler) partialDiffBetweenManifestAndInMemberClusterObjects( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, +) ([]placementv1alpha1.PatchDetail, bool, error) { + // KubeFleet calculates the partial diff between two objects by running apply ops in the dry-run + // mode. + appliedObj, err := r.applyInDryRunMode(ctx, gvr, manifestObj, inMemberClusterObj) + + // After the dry-run apply op, all the managed fields should have been overwritten using the + // values from the manifest object, while leaving all the unmanaged fields untouched. This + // would allow KubeFleet to compare the object returned by the dry-run apply op with the object + // that is currently in the member cluster; if all the fields are consistent, it is safe + // for us to assume that there are no drifts, otherwise, any fields that are different + // imply that running an actual apply op would lead to unexpected changes, which signifies + // the presence of partial drifts (drifts in managed fields). + + switch { + case err == nil: + // The dry-run apply op has succeeded. All managed fields should have been overwritten using the + // values from the manifest object, while leaving all the unmanaged fields untouched. This + // would allow KubeFleet to compare the object returned by the dry-run apply op with the object + // that is currently in the member cluster; if all the fields are consistent, it is safe + // for us to assume that there are no drifts, otherwise, any fields that are different + // imply that running an actual apply op would lead to unexpected changes, which signifies + // the presence of partial drifts (drifts in managed fields). + patchDetails, err := preparePatchDetails(appliedObj, inMemberClusterObj) + return patchDetails, false, err + case apierrors.IsInvalid(err): + // The dry-run apply op has failed as the manifest object provided is not valid. This could + // happen when the apply op involves fields that are immutable or the apply op attempts to + // set invalid values. This error implies that the in-cluster object has already been modified, + // and any change that the user supplies right now cannot be accepted. + // + // In this case, fall back to full comparison. Report that the diff is being calculated in a + // degraded manner. + // + // This is not considered as a diff calculation error. + klog.V(2).InfoS("Calculate diffs in degraded mode as the manifest object cannot be server-side applied in dry-run mode", + "gvr", gvr, "manifestObj", klog.KObj(manifestObj), "serverErr", err) + patchDetails, err := preparePatchDetails(manifestObj, inMemberClusterObj) + return patchDetails, true, err + default: + // An unexpected error has occurred. + return nil, false, errors.Wraps(err, "failed to apply the manifest in dry-run mode") + } +} + +// preparePatchDetails calculates the differences between two objects in the form +// of KubeFleet patch details. +func preparePatchDetails(srcObj, destObj *unstructured.Unstructured) ([]placementv1alpha1.PatchDetail, error) { + // Discard certain fields from both objects before comparison. + srcObjCopy := discardFieldsIrrelevantInComparisonFrom(srcObj) + destObjCopy := discardFieldsIrrelevantInComparisonFrom(destObj) + + // Marshal the objects into JSON. + srcObjJSONBytes, err := srcObjCopy.MarshalJSON() + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to marshal the source object into JSON") + } + + destObjJSONBytes, err := destObjCopy.MarshalJSON() + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to marshal the destination object into JSON") + } + + // Compare the JSON representations. + patch, err := jsondiff.CompareJSON(srcObjJSONBytes, destObjJSONBytes) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to compare the JSON representations of the source and destination objects") + } + + // Prepare KubeFleet patch details from the JSON patches. + details, err := organizeJSONPatchIntoPatchDetails(patch, srcObjCopy.Object) + if err != nil { + return nil, errors.Wraps(err, "failed to organize JSON patch operations into patch details") + } + + // Obscure sensitive fields in the patch details. + // + // This currently only concerns Secret objects (core API group); all the drift/diff outputs regarding + // a Secret object's data (`.data` or `.stringData` fields) are obscured. + details = obscureSensitiveFieldsInPatchDetails(srcObj, details) + return details, nil +} + +// discardFieldsIrrelevantInComparisonFrom discards fields that are irrelevant when comparing +// the manifest and live objects (or two manifest objects). +// +// Note that this method will return an object copy; the original object will be left untouched. +func discardFieldsIrrelevantInComparisonFrom(obj *unstructured.Unstructured) *unstructured.Unstructured { + // Create a deep copy of the object. + objCopy := obj.DeepCopy() + + // Remove object meta fields that are irrelevant in comparison. + + // Clear out the object's name/namespace. For regular objects, the names/namespaces will + // always be same between the manifest and the live objects, as guaranteed by the object + // retrieval step earlier, so a comparison on these fields is unnecessary anyway. + objCopy.SetName("") + objCopy.SetNamespace("") + + // Clear out the object's generate name. This is a field that is irrelevant in comparison. + objCopy.SetGenerateName("") + + // Remove certain labels and annotations. + // + // KubeFleet will remove labels/annotations that are reserved for KubeFleet own use cases, plus + // well-known Kubernetes labels and annotations, as these cannot (should not) be set by users + // directly. + annotations := objCopy.GetAnnotations() + cleanedAnnotations := map[string]string{} + for k, v := range annotations { + if strings.Contains(k, k8sReservedLabelAnnotationFullDomain) { + // Skip Kubernetes reserved annotations. + continue + } + + if strings.Contains(k, k8sReservedLabelAnnotationAbbrDomain) { + // Skip Kubernetes reserved annotations. + continue + } + + if strings.Contains(k, kubefleetReservedLabelAnnotationDomain) { + // Skip KubeFleet reserved annotations. + continue + } + cleanedAnnotations[k] = v + } + objCopy.SetAnnotations(cleanedAnnotations) + + labels := objCopy.GetLabels() + cleanedLabels := map[string]string{} + for k, v := range labels { + if strings.Contains(k, k8sReservedLabelAnnotationFullDomain) { + // Skip Kubernetes reserved labels. + continue + } + + if strings.Contains(k, k8sReservedLabelAnnotationAbbrDomain) { + // Skip Kubernetes reserved labels. + continue + } + + if strings.Contains(k, kubefleetReservedLabelAnnotationDomain) { + // Skip KubeFleet reserved labels. + continue + } + cleanedLabels[k] = v + } + objCopy.SetLabels(cleanedLabels) + + // Fields below are system-reserved fields in object meta. Technically speaking they can be + // set in the manifests, but this is a very uncommon practice, and currently KubeFleet will clear + // these fields (except for the finalizers) before applying the manifests. + // As a result, for now KubeFleet will ignore them in the comparison process as well. + // + // TO-DO (chenyu1): evaluate if this is a correct assumption for most (if not all) KubeFleet + // users. + objCopy.SetFinalizers([]string{}) + objCopy.SetManagedFields([]metav1.ManagedFieldsEntry{}) + objCopy.SetOwnerReferences([]metav1.OwnerReference{}) + + // Fields below are read-only fields in object meta. KubeFleet will ignore them in the comparison + // process. + // Deleted objects are handled separately in the apply process; for comparison purposes, + // KubeFleet will ignore the deletion timestamp and grace period seconds. + objCopy.SetDeletionTimestamp(nil) + objCopy.SetDeletionGracePeriodSeconds(nil) + objCopy.SetGeneration(0) + objCopy.SetResourceVersion("") + objCopy.SetSelfLink("") + objCopy.SetUID("") + + // Remove the status field. + unstructured.RemoveNestedField(objCopy.Object, "status") + + // Remove all creationTimestamp fields. + unstructured.RemoveNestedField(objCopy.Object, "metadata", "creationTimestamp") + // Resources that have .spec.template.metadata include: Deployment, Job, StatefulSet, + // DaemonSet, ReplicaSet, and CronJob. + unstructured.RemoveNestedField(objCopy.Object, "spec", "template", "metadata", "creationTimestamp") + // Also handle CronJob's .spec.jobTemplate.spec.template.metadata + unstructured.RemoveNestedField(objCopy.Object, "spec", "jobTemplate", "spec", "template", "metadata", "creationTimestamp") + + return objCopy +} + +// organizeJSONPatchIntoPatchDetails organizes the JSON patch operations into KubeFleet patch details. +func organizeJSONPatchIntoPatchDetails(patch jsondiff.Patch, manifestObjMap map[string]interface{}) ([]placementv1alpha1.PatchDetail, error) { + // Pre-allocate the slice for the patch details. The organization procedure typically will yield + // the same number of PatchDetail items as the JSON patch operations. + details := make([]placementv1alpha1.PatchDetail, 0, len(patch)) + + // A side note: here KubeFleet takes an expedient approach processing null JSON paths, by treating + // null paths as empty strings. + + // Process only the first 100 ops. + // TO-DO (chenyu1): Impose additional size limits. + for idx := 0; idx < len(patch) && idx < patchDetailPerObjLimit; idx++ { + op := patch[idx] + pathPtr, err := jsonpointer.Parse(op.Path) + if err != nil { + // An invalid path is found; normally this should not happen. + return nil, errors.NewUnexpectedError(err, "failed to parse the JSON path", "path", op.Path) + } + fromPtr, err := jsonpointer.Parse(op.From) + if err != nil { + // An invalid path is found; normally this should not happen. + return nil, errors.NewUnexpectedError(err, "failed to parse the JSON path", "path", op.From) + } + + switch op.Type { + case jsondiff.OperationAdd: + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprint(op.Value), + }) + case jsondiff.OperationRemove: + // KubeFleet here skips validation as the JSON data is just marshalled. + hubValue, err := pathPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.Path) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInHub: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationReplace: + // KubeFleet here skips validation as the JSON data is just marshalled. + hubValue, err := pathPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.Path) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprint(op.Value), + ValueInHub: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationMove: + // Normally the Move operation will not be returned as factorization is disabled + // for the JSON patch calculation process; however, KubeFleet here still processes them + // just in case. + // + // Each Move operation will be parsed into two separate operations. + hubValue, err := fromPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.From) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.From, + ValueInHub: fmt.Sprintf("%v", hubValue), + }) + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationCopy: + // Normally the Copy operation will not be returned as factorization is disabled + // for the JSON patch calculation process; however, KubeFleet here still processes them + // just in case. + // + // Each Copy operation will be parsed into an Add operation. + hubValue, err := fromPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.From) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationTest: + // The Test op is a no-op in KubeFleet's use case. Normally it will not be returned, either. + default: + // An unexpected op is returned. + return nil, errors.NewUnexpectedError(nil, "an unexpected JSON patch operation is returned", "op", op) + } + } + + return details, nil +} + +// obscureSensitiveFieldsInPatchDetails obscures sensitive fields from the patch details so that +// such information will not be included in the drift/diff outputs. +// +// At this moment fields in the following API objects are discarded: +// +// * `.data` and `.stringData` field (and their children) in all Secret objects (`core` API group). +// +// Note (chenyu1): there are other Kubernetes API objects that also feature sensitive data, +// such as TokenRequest and CertificateSigningRequest; these objects are not included in the list +// as they have the sensitive information in the status, which are not accounted for in the +// drift/diff calculation in the very beginning. +func obscureSensitiveFieldsInPatchDetails( + srcObj *unstructured.Unstructured, details []placementv1alpha1.PatchDetail, +) (sanitizedPatchDetails []placementv1alpha1.PatchDetail) { + // Verify if the object is a Secret. + if srcObj.GetAPIVersion() != "v1" || srcObj.GetKind() != "Secret" { + return details + } + + for idx := range details { + pd := &details[idx] + // Note (chenyu1): the string data field in the Secret object is provided by Kubernetes + // as a write-only field for convenience reasons; all entries shall be merged into the + // data field. Here the code still processes the field just for completeness reasons; + // in practice it will never be included as part of the patch details. + if strings.HasPrefix(pd.Path, "/data") || strings.HasPrefix(pd.Path, "/stringData") { + // Obscure all patch details that concerns the Secret object's data. + + if len(pd.ValueInHub) > 0 { + pd.ValueInHub = "(redacted for security reasons)" + } + if len(pd.ValueInMember) > 0 { + pd.ValueInMember = "(redacted for security reasons)" + } + } + } + return details +} diff --git a/pkg/v1/controllers/workapplier/utils_test.go b/pkg/v1/controllers/workapplier/utils_test.go new file mode 100644 index 000000000..bb1af5cd6 --- /dev/null +++ b/pkg/v1/controllers/workapplier/utils_test.go @@ -0,0 +1,701 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + testutilsresource "github.com/kubefleet-dev/kubefleet/test/utils/resource" +) + +const ( + workNameTemplate = "work-%s" + nsNameTemplate = "ns-%s" + placementPolicyNameTemplate = "policy-%s" + placementBindingNameTemplate = "binding-%s" + primarySnapshotNameTemplate = "snapshot-%s-0" + deployName = "deploy-1" + configMapName = "configmap-1" + singleWorkLinkedWorkCountAnno = "1" +) + +const ( + eventuallyDuration = time.Second * 10 + eventuallyInterval = time.Second * 1 + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Millisecond * 500 +) + +var ( + ignoreFieldObjectMetaAutoGenFields = cmpopts.IgnoreFields(metav1.ObjectMeta{}, "CreationTimestamp", "Generation", "ResourceVersion", "SelfLink", "UID", "ManagedFields") + ignoreFieldAppliedWorkStatus = cmpopts.IgnoreFields(placementv1alpha1.AppliedWork{}, "Status") + ignoreFieldConditionLTTMsg = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message") + ignoreFieldDiffDetailsObsTime = cmpopts.IgnoreFields(placementv1alpha1.DiffDetails{}, "FirstDiffedObservedTimestamp") +) + +var ( + ns = &corev1.Namespace{ + TypeMeta: metav1.TypeMeta{ + Kind: "Namespace", + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: "ns-1", + }, + } + + configMap = &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{ + Kind: "ConfigMap", + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: "ns-1", + }, + Data: map[string]string{ + "foo": "bar", + }, + } + + deploy = &appsv1.Deployment{ + TypeMeta: metav1.TypeMeta{ + Kind: "Deployment", + APIVersion: "apps/v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: deployName, + Namespace: "ns-1", + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(1)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "app": "nginx", + }, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": "nginx", + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "nginx", + Image: "nginx", + Ports: []corev1.ContainerPort{ + { + ContainerPort: 80, + }, + }, + }, + }, + }, + }, + }, + } +) + +// buildWorkObject builds a work object with the given work name/namespace, owner placement policy/binding, +// primary placement resource snapshot, sync strategy, and raw manifest JSONs. +// +// Note that the returned work object does not have the linked work count annotation set; the annotation +// is only present on primary work objects. +func buildWorkObject( + workName, memberClusterReservedNSName string, + placementPolicyName, placementBindingName, primarySnapshotName string, + syncStrategy *placementv1alpha1.SyncStrategy, + rawManifestJSON ...[]byte, +) *placementv1alpha1.Work { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + return &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberClusterReservedNSName, + Labels: map[string]string{ + // An empty owner namespace signals cluster-scoped owners. + placementv1alpha1.WorkOwnerNamespaceLabelKey: "", + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: placementBindingName, + }, + Annotations: map[string]string{ + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: placementBindingName, + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: primarySnapshotName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: manifests, + SyncStrategy: syncStrategy, + }, + } +} + +// createWorkObject creates a new (primary) work object with the given work name/namespace, sync strategy, +// and raw manifest JSONs. +// +// The work object is set up as the only work object linked to a dedicated (cluster-scoped) placement policy and +// placement binding, so that work objects created in different test cases never get linked with each other. +func createWorkObject(workName, memberClusterReservedNSName string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + work := buildWorkObject( + workName, memberClusterReservedNSName, + fmt.Sprintf(placementPolicyNameTemplate, workName), + fmt.Sprintf(placementBindingNameTemplate, workName), + fmt.Sprintf(primarySnapshotNameTemplate, workName), + syncStrategy, + rawManifestJSON..., + ) + work.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = singleWorkLinkedWorkCountAnno + Expect(hubClient.Create(ctx, work)).To(Succeed(), "Failed to create the Work object") +} + +func marshalK8sObjJSON(obj runtime.Object) []byte { + json, err := testutilsresource.MarshalRuntimeObjToJSONForTest(obj) + Expect(err).To(BeNil(), "Failed to marshal the k8s object to JSON") + return json +} + +func workFinalizerAddedActual(workNS, workName string) func() error { + return func() error { + // Retrieve the Work object. + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + // Check that the cleanup finalizer has been added. + if !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer has not been added") + } + return nil + } +} + +func appliedWorkCreatedActual(memberClient client.Client, workNS, workName string) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + wantAppliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + }, + Spec: placementv1alpha1.AppliedWorkSpec{ + WorkName: workName, + WorkNamespace: workNS, + }, + } + if diff := cmp.Diff( + appliedWork, wantAppliedWork, + ignoreFieldObjectMetaAutoGenFields, + ignoreFieldAppliedWorkStatus, + ); diff != "" { + return fmt.Errorf("appliedWork diff (-got +want):\n%s", diff) + } + return nil + } +} + +func prepareAppliedWorkOwnerRef(memberClient client.Client, workName string) *metav1.OwnerReference { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + Expect(memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork)).To(Succeed(), "Failed to retrieve the AppliedWork object") + + // Prepare the expected OwnerReference. + return &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "AppliedWork", + Name: appliedWork.Name, + UID: appliedWork.GetUID(), + BlockOwnerDeletion: ptr.To(true), + } +} + +func regularNSObjectAppliedActual(memberClient client.Client, nsName string, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + // Retrieve the NS object. + gotNS := &corev1.Namespace{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: nsName}, gotNS); err != nil { + return fmt.Errorf("failed to retrieve the NS object: %w", err) + } + + // Check that the NS object has been created as expected. + + // To ignore default values automatically, here the test suite rebuilds the objects. + wantNS := ns.DeepCopy() + wantNS.TypeMeta = metav1.TypeMeta{} + wantNS.Name = nsName + wantNS.OwnerReferences = []metav1.OwnerReference{ + *appliedWorkOwnerRef, + } + + rebuiltGotNS := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: gotNS.Name, + OwnerReferences: gotNS.OwnerReferences, + }, + } + + if diff := cmp.Diff(rebuiltGotNS, wantNS); diff != "" { + return fmt.Errorf("namespace diff (-got +want):\n%s", diff) + } + return nil + } +} + +func regularDeploymentObjectAppliedActual(memberClient client.Client, nsName, deployName string, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + // Retrieve the Deployment object. + gotDeploy := &appsv1.Deployment{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy); err != nil { + return fmt.Errorf("failed to retrieve the Deployment object: %w", err) + } + + // Check that the Deployment object has been created as expected. + + // To ignore default values automatically, here the test suite rebuilds the objects. + wantDeploy := deploy.DeepCopy() + wantDeploy.TypeMeta = metav1.TypeMeta{} + wantDeploy.Namespace = nsName + wantDeploy.Name = deployName + wantDeploy.OwnerReferences = []metav1.OwnerReference{ + *appliedWorkOwnerRef, + } + + if len(gotDeploy.Spec.Template.Spec.Containers) != 1 { + return fmt.Errorf("number of containers in the Deployment object, got %d, want %d", len(gotDeploy.Spec.Template.Spec.Containers), 1) + } + if len(gotDeploy.Spec.Template.Spec.Containers[0].Ports) != 1 { + return fmt.Errorf("number of ports in the first container, got %d, want %d", len(gotDeploy.Spec.Template.Spec.Containers[0].Ports), 1) + } + rebuiltGotDeploy := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: gotDeploy.Namespace, + Name: gotDeploy.Name, + OwnerReferences: gotDeploy.OwnerReferences, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: gotDeploy.Spec.Replicas, + Selector: gotDeploy.Spec.Selector, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": gotDeploy.Spec.Template.Labels["app"], + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: gotDeploy.Spec.Template.Spec.Containers[0].Name, + Image: gotDeploy.Spec.Template.Spec.Containers[0].Image, + Ports: []corev1.ContainerPort{ + { + ContainerPort: gotDeploy.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort, + }, + }, + }, + }, + }, + }, + }, + } + if diff := cmp.Diff(rebuiltGotDeploy, wantDeploy); diff != "" { + return fmt.Errorf("deployment diff (-got +want):\n%s", diff) + } + return nil + } +} + +func regularConfigMapObjectAppliedActual(memberClient client.Client, nsName, configMapName string, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + // Retrieve the ConfigMap object. + gotConfigMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, gotConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap object: %w", err) + } + + // Check that the ConfigMap object has been created as expected. + + // To ignore default values automatically, here the test suite rebuilds the objects. + wantConfigMap := configMap.DeepCopy() + wantConfigMap.TypeMeta = metav1.TypeMeta{} + wantConfigMap.Namespace = nsName + wantConfigMap.Name = configMapName + wantConfigMap.OwnerReferences = []metav1.OwnerReference{ + *appliedWorkOwnerRef, + } + + rebuiltGotConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: gotConfigMap.Namespace, + Name: gotConfigMap.Name, + OwnerReferences: gotConfigMap.OwnerReferences, + }, + Data: gotConfigMap.Data, + } + if diff := cmp.Diff(rebuiltGotConfigMap, wantConfigMap); diff != "" { + return fmt.Errorf("configMap diff (-got +want):\n%s", diff) + } + return nil + } +} + +func markDeploymentAsAvailable(memberClient client.Client, nsName, deployName string) { + // Retrieve the Deployment object. + gotDeploy := &appsv1.Deployment{} + Expect(memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + + // Mark the Deployment object as available. + now := metav1.Now() + requiredReplicas := int32(1) + if gotDeploy.Spec.Replicas != nil { + requiredReplicas = *gotDeploy.Spec.Replicas + } + gotDeploy.Status = appsv1.DeploymentStatus{ + ObservedGeneration: gotDeploy.Generation, + Replicas: requiredReplicas, + UpdatedReplicas: requiredReplicas, + ReadyReplicas: requiredReplicas, + AvailableReplicas: requiredReplicas, + UnavailableReplicas: 0, + Conditions: []appsv1.DeploymentCondition{ + { + Type: appsv1.DeploymentAvailable, + Status: corev1.ConditionTrue, + Reason: "MarkedAsAvailable", + Message: "Deployment has been marked as available", + LastUpdateTime: now, + LastTransitionTime: now, + }, + }, + } + Expect(memberClient.Status().Update(ctx, gotDeploy)).To(Succeed(), "Failed to mark the Deployment object as available") +} + +func workStatusUpdated( + workNS, workName string, + workConds []metav1.Condition, + manifestStatuses []placementv1alpha1.PerManifestStatus, +) func() error { + return func() error { + // Retrieve the Work object. + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + // Prepare the expected Work object status. + + // Update the conditions with the observed generation. + // + // Note that the observed generation of a manifest condition is that of an applied + // resource, not that of the Work object. + for idx := range workConds { + workConds[idx].ObservedGeneration = work.Generation + } + wantWorkStatus := placementv1alpha1.WorkStatus{ + Conditions: workConds, + Manifests: manifestStatuses, + } + + // Check that the Work object status has been updated as expected. + if diff := cmp.Diff( + work.Status, wantWorkStatus, + ignoreFieldConditionLTTMsg, + ignoreFieldDiffDetailsObsTime, + ); diff != "" { + return fmt.Errorf("work status diff (-got, +want):\n%s", diff) + } + return nil + } +} + +func appliedWorkStatusUpdated(memberClient client.Client, workName string, appliedResources []placementv1alpha1.AppliedResource) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + // Prepare the expected AppliedWork object status. + wantAppliedWorkStatus := placementv1alpha1.AppliedWorkStatus{ + AppliedResources: appliedResources, + } + if diff := cmp.Diff(appliedWork.Status, wantAppliedWorkStatus); diff != "" { + return fmt.Errorf("appliedWork status diff (-got, +want):\n%s", diff) + } + return nil + } +} + +func deleteWorkObject(workName, memberClusterReservedNSName string) { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberClusterReservedNSName, + }, + } + Expect(hubClient.Delete(ctx, work)).To(Succeed(), "Failed to delete the Work object") +} + +// updateWorkObject replaces the manifests and the sync strategy of an existing work object. +func updateWorkObject(workName, memberClusterReservedNSName string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: memberClusterReservedNSName}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + work.Spec.Manifests = manifests + work.Spec.SyncStrategy = syncStrategy + // The work applier might update the work object (e.g., its status) at the same time; retry on conflicts. + return hubClient.Update(ctx, work) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the Work object") +} + +// updateLinkedWorkObject replaces the manifests and the sync strategy of an existing work object, and links it +// to the given primary placement resource snapshot; this mimics how KubeFleet refreshes work objects during a +// rollout. +func updateLinkedWorkObject(workName, memberClusterReservedNSName, primarySnapshotName string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: memberClusterReservedNSName}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + work.Annotations[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] = primarySnapshotName + work.Spec.Manifests = manifests + work.Spec.SyncStrategy = syncStrategy + // The work applier might update the work object (e.g., its status) at the same time; retry on conflicts. + return hubClient.Update(ctx, work) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the Work object") +} + +func appliedWorkBeingDeletedInForegroundActual(memberClient client.Client, workName string) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + // Check that the AppliedWork object has been marked for deletion with the foreground propagation policy. + // + // Note that as there are no real built-in controllers (the GC controller) in this test environment, the + // AppliedWork object will stay in the pending deletion state with the foreground deletion finalizer. + if appliedWork.DeletionTimestamp.IsZero() { + return fmt.Errorf("appliedWork object has not been marked for deletion") + } + if !controllerutil.ContainsFinalizer(appliedWork, metav1.FinalizerDeleteDependents) { + return fmt.Errorf("appliedWork object is not being deleted in the foreground (finalizers: %v)", appliedWork.Finalizers) + } + return nil + } +} + +// appliedWorkOwnerRefUnblockedActual checks that the given object is still owned by the AppliedWork object, +// but the owner reference no longer blocks the deletion of the AppliedWork object. +func appliedWorkOwnerRefUnblockedActual(memberClient client.Client, obj client.Object, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil { + return fmt.Errorf("failed to retrieve the object: %w", err) + } + + wantOwnerRef := appliedWorkOwnerRef.DeepCopy() + wantOwnerRef.BlockOwnerDeletion = ptr.To(false) + wantOwnerRefs := []metav1.OwnerReference{*wantOwnerRef} + if diff := cmp.Diff(obj.GetOwnerReferences(), wantOwnerRefs); diff != "" { + return fmt.Errorf("owner references diff (-got +want):\n%s", diff) + } + return nil + } +} + +// workFinalizerRemovedActual checks that the work object still exists, but no longer has the cleanup finalizer. +func workFinalizerRemovedActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + if controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer has not been removed") + } + return nil + } +} + +// appliedWorkAbsentActual checks that the AppliedWork object does not exist in the member cluster. +// +// Unlike appliedWorkRemovedActual, this function does not attempt to remove any finalizer from the object. +func appliedWorkAbsentActual(memberClient client.Client, workName string) func() error { + return func() error { + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); !apierrors.IsNotFound(err) { + return fmt.Errorf("appliedWork object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func appliedWorkRemovedActual(memberClient client.Client, workName string) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + if apierrors.IsNotFound(err) { + // The AppliedWork object has been deleted, which is expected. + return nil + } + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + if !appliedWork.DeletionTimestamp.IsZero() && controllerutil.ContainsFinalizer(appliedWork, metav1.FinalizerDeleteDependents) { + // The AppliedWork object is being deleted, but the finalizer is still present. Remove the finalizer as there + // are no real built-in controllers in this test environment to handle garbage collection. + controllerutil.RemoveFinalizer(appliedWork, metav1.FinalizerDeleteDependents) + Expect(memberClient.Update(ctx, appliedWork)).To(Succeed(), "Failed to remove the finalizer from the AppliedWork object") + } + return fmt.Errorf("appliedWork object still exists") + } +} + +func regularDeployRemovedActual(memberClient client.Client, nsName, deployName string) func() error { + return func() error { + // Delete the Deployment object manually, as there are no real built-in controllers in this + // test environment to handle garbage collection. + deploy := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: deployName, + }, + } + if err := memberClient.Delete(ctx, deploy); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the Deployment object: %w", err) + } + + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, deploy); !apierrors.IsNotFound(err) { + return fmt.Errorf("deployment object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func regularConfigMapRemovedActual(memberClient client.Client, nsName, configMapName string) func() error { + return func() error { + // Delete the ConfigMap object manually, as there are no real built-in controllers in this + // test environment to handle garbage collection. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: configMapName, + }, + } + if err := memberClient.Delete(ctx, configMap); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the ConfigMap object: %w", err) + } + + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, configMap); !apierrors.IsNotFound(err) { + return fmt.Errorf("configMap object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// regularConfigMapAbsentActual checks that the ConfigMap object does not exist in the member cluster. +// +// Unlike regularConfigMapRemovedActual, this function does not attempt to delete the object by itself. +func regularConfigMapAbsentActual(memberClient client.Client, nsName, configMapName string) func() error { + return func() error { + configMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, configMap); !apierrors.IsNotFound(err) { + return fmt.Errorf("configMap object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// regularDeployAbsentActual checks that the Deployment object does not exist in the member cluster. +// +// Unlike regularDeployRemovedActual, this function does not attempt to delete the object by itself. +func regularDeployAbsentActual(memberClient client.Client, nsName, deployName string) func() error { + return func() error { + deploy := &appsv1.Deployment{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, deploy); !apierrors.IsNotFound(err) { + return fmt.Errorf("deployment object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func workRemovedActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + if apierrors.IsNotFound(err) { + return nil + } + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + return fmt.Errorf("work object still exists (finalizers: %v)", work.Finalizers) + } +} diff --git a/pkg/v1/controllers/workapplier/waves.go b/pkg/v1/controllers/workapplier/waves.go new file mode 100644 index 000000000..16074cbf4 --- /dev/null +++ b/pkg/v1/controllers/workapplier/waves.go @@ -0,0 +1,180 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "slices" + + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" +) + +type waveNumber int + +const ( + lastWave waveNumber = 999 +) + +var ( + // The default wave number for all known Kubernetes resource type. + // + // Note (chenyu1): the waves below are based on the Helm resource installation + // order (see also the Helm source code). Similar objects are grouped together + // to achieve best performance. + defaultWaveNumberByResourceType = map[string]waveNumber{ + // Apply namespaces and priority classes first. + "namespaces": 0, + "priorityclasses": 0, + // Apply policies, configuration data, and other static resources second. + "networkpolicies": 1, + "resourcequotas": 1, + "limitranges": 1, + "podsecuritypolicies": 1, + "poddisruptionbudgets": 1, + "serviceaccounts": 1, + "secrets": 1, + "configmaps": 1, + "storageclasses": 1, + "persistentvolumes": 1, + "persistentvolumeclaims": 1, + "customresourcedefinitions": 1, + "ingressclasses": 1, + // Apply RBAC resources (cluster roles and roles). + "clusterroles": 2, + "roles": 2, + // Apply RBAC resources (cluster role bindings and role bindings). + "clusterrolebindings": 3, + "rolebindings": 3, + // Apply workloads and services. + "services": 4, + "daemonsets": 4, + "pods": 4, + "replicationcontrollers": 4, + "replicasets": 4, + "deployments": 4, + "horizontalpodautoscalers": 4, + "statefulsets": 4, + "jobs": 4, + "cronjobs": 4, + "ingresses": 4, + // Apply API services and webhooks. + "apiservices": 5, + "validatingwebhookconfigurations": 5, + "mutatingwebhookconfigurations": 5, + } + + // The API groups for all known Kubernetes resource types. + knownAPIGroups = sets.New( + // The core API group. + "", + // The networking API group (`networking.k8s.io`). + "networking.k8s.io", + // The scheduling API group (`scheduling.k8s.io`). + "scheduling.k8s.io", + // The policy API group (`policy.k8s.io`). + "policy", + // The storage API group (`storage.k8s.io`). + "storage.k8s.io", + // The API extensions API group (`apiextensions.k8s.io`). + "apiextensions.k8s.io", + // The RBAC authorization API group (`rbac.authorization.k8s.io`). + "rbac.authorization.k8s.io", + // The apps API group (`apps`). + "apps", + // The autoscaling API group (`autoscaling`). + "autoscaling", + // The API registration API group (`apiregistration.k8s.io`). + "apiregistration.k8s.io", + // The admission registration API group (`admissionregistration.k8s.io`). + "admissionregistration.k8s.io", + // The batch API group (`batch`). + "batch", + ) +) + +// manifestProcessingWave is a wave of manifests that can be processed in parallel. +type manifestProcessingWave struct { + num waveNumber + states []*manifestProcessingState +} + +// organizeManifestsIntoProcessingWaves organizes the list of manifest processing states into +// different waves for parallel processing based on their GVR information. +func organizeManifestsIntoProcessingWaves(manifestProcessingStates []*manifestProcessingState) []*manifestProcessingWave { + // Pre-allocate the map; 7 is the total count of default wave numbers, though + // not all wave numbers might be used. + waveByNum := make(map[waveNumber]*manifestProcessingWave, 7) + + getOrAddWave := func(num waveNumber) *manifestProcessingWave { + wave, ok := waveByNum[num] + if !ok { + wave = &manifestProcessingWave{ + num: num, + // Pre-allocate with a reasonable size. + states: make([]*manifestProcessingState, 0, 5), + } + waveByNum[num] = wave + } + return wave + } + + // For simplicity reasons, the organization itself runs in sequential order. + // Considering that the categorization itself is quick and the total number of manifests + // should be limited in most cases, this should not introduce significant overhead. + for idx := range manifestProcessingStates { + state := manifestProcessingStates[idx] + if state.gvr == nil { + // For manifest data that cannot be decoded, there might not be any available GVR + // information. Skip such manifests; this is not considered as an error. + klog.V(2).InfoS("Skipping a manifest with no GVR; no wave is assigned", + "ordinal", idx, "work", klog.KObj(state.fromWorkObj)) + continue + } + + if state.applyErr != nil { + // An error has occurred before this step; such manifests need no further processing, + // skip them. This is not considered as an error. + klog.V(2).InfoS("Skipping a manifest with prior processing error; no wave is assigned", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, "work", klog.KObj(state.fromWorkObj)) + continue + } + + waveNum := lastWave + defaultWaveNum, foundInDefaultWaveNumber := defaultWaveNumberByResourceType[state.gvr.Resource] + if foundInDefaultWaveNumber && knownAPIGroups.Has(state.gvr.Group) { + // The resource is a known one; assign the manifest to its default wave. + waveNum = defaultWaveNum + } + + wave := getOrAddWave(waveNum) + wave.states = append(wave.states, state) + klog.V(2).InfoS("Assigned manifest to a wave", + "waveNumber", waveNum, + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, "work", klog.KObj(state.fromWorkObj)) + } + + // Retrieve all the waves and sort them by their wave number. + waves := make([]*manifestProcessingWave, 0, len(waveByNum)) + for _, w := range waveByNum { + waves = append(waves, w) + } + // Sort the waves in ascending order. + slices.SortFunc(waves, func(a, b *manifestProcessingWave) int { + return int(a.num) - int(b.num) + }) + return waves +} diff --git a/pkg/v1/utils/fieldindexers/hub.go b/pkg/v1/utils/fieldindexers/hub.go index a4838ee94..089b12407 100644 --- a/pkg/v1/utils/fieldindexers/hub.go +++ b/pkg/v1/utils/fieldindexers/hub.go @@ -14,13 +14,13 @@ See the License for the specific language governing permissions and limitations under the License. */ +// Package fieldindexers provides utilities for setting up and using field-based indexes for KubeFleet API objects on the KubeFleet agent side. package fieldindexers import ( "context" "fmt" - "k8s.io/klog/v2" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -29,13 +29,13 @@ import ( ) const ( - // The field-based indexes set up for KubeFleet API objects. + // The field-based indexes set up for KubeFleet API objects (on the hub agent side). // - // Important: many KubeFleet components run under the assumption that proper custom fields + // Important: many KubeFleet components on the hub agent side run under the assumption that proper custom fields // have been added and indexed in the cache when running. Failure to complete such prior setup **before - // the manager starts** will result in unexpected behaviors. Make sure that all applicable components - // are properly set up using the client provided by the hub controller manager, and `SetupWithManager` is - // called before the manager starts. + // the manager starts** will result in unexpected behaviors/failures. Make sure that all applicable components + // are properly set up using the client provided by the hub controller manager, and + // `SetupWithHubAgentControllerManager` is called before the manager starts. // PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldName is the name of the custom field that indexes // placement resource snapshots by their owner placement policies and their sub-indices. @@ -78,28 +78,6 @@ const ( WorkOwnedByBindingCustomFieldValFmt = "%s/%s" ) -type fieldValueExtractor func(obj client.Object) ([]string, error) - -func indexCompositeField(ctx context.Context, - fieldIdxer client.FieldIndexer, - obj client.Object, - fieldName string, fieldValueExt fieldValueExtractor) error { - if err := fieldIdxer.IndexField(ctx, obj, fieldName, func(rawObj client.Object) []string { - fieldVals, extErr := fieldValueExt(rawObj) - if extErr != nil { - wrappedErr := errors.NewUnexpectedError(extErr, "failed to extract field value", "object", klog.KObj(rawObj)) - klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) - return nil - } - return fieldVals - }); err != nil { - wrappedErr := errors.NewUnexpectedError(err, "", "fieldName", fieldName, "object", klog.KObj(obj)) - klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) - return wrappedErr - } - return nil -} - var ( placementResourceSnapshotOwnedByAndSubIdxedFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { ownedBy := obj.GetLabels()[placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey] @@ -132,9 +110,10 @@ var ( } ) -// SetupWithHubControllerManager sets up the indices that controllers from the KubeFleet hub agent need to run properly. +// SetupWithHubAgentControllerManager sets up the indices that controllers from the KubeFleet hub agent need to run properly. +// // It must be called before the manager starts. -func SetupWithHubControllerManager(ctx context.Context, mgr ctrl.Manager) error { +func SetupWithHubAgentControllerManager(ctx context.Context, mgr ctrl.Manager) error { fieldIdxer := mgr.GetFieldIndexer() if err := indexCompositeField(ctx, fieldIdxer, diff --git a/pkg/v1/utils/fieldindexers/member.go b/pkg/v1/utils/fieldindexers/member.go new file mode 100644 index 000000000..98991a230 --- /dev/null +++ b/pkg/v1/utils/fieldindexers/member.go @@ -0,0 +1,115 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package fieldindexers + +import ( + "context" + "fmt" + + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +const ( + // The field-based indexes set up for KubeFleet API objects (on the member agent side). + // + // Important: many KubeFleet components on the member agent side run under the assumption that proper custom fields + // have been added and indexed in the cache when running. Failure to complete such prior setup **before + // the manager starts** will result in unexpected behaviors/failures. Make sure that all applicable components + // are properly set up using the client provided by the member controller manager, and + // `SetupWithMemberAgentControllerManager` is called before the manager starts. + + // WorkOwnedByPlacementPolicyCustomFieldName is the name of the custom field that indexes + // work objects by their owner placement policies. + // + // This is added to help the work applier retrieve all work objects associated with a placement policy. + WorkOwnedByPlacementPolicyCustomFieldName = "ownedByPlacementPolicy" + + // WorkOwnedByPlacementBindingCustomFieldName is the name of the custom field that indexes + // work objects by their owner placement bindings. + // + // This is added to help the work applier retrieve all the work objects associated with a placement binding + // in one batch. + WorkOwnedByPlacementBindingCustomFieldName = "ownedByPlacementBinding" +) + +const ( + // The format of the custom field values for the field-based indexes defined above. + + // WorkOwnedByPlacementPolicyCustomFieldValFmt is used to format the value for the custom field, + // `WorkOwnedByPlacementPolicyCustomFieldName`, in the form of `[OWNER-NAMESPACE]/[OWNER-NAME]`. The owner + // namespace is empty for cluster-scoped placement policies. + // + // Note that slashes are used to avoid unexpected collisions. + WorkOwnedByPlacementPolicyCustomFieldValFmt = "%s/%s" + + // WorkOwnedByPlacementBindingCustomFieldValFmt is used to format the value for the custom field, + // `WorkOwnedByPlacementBindingCustomFieldName`, in the form of `[OWNER-NAMESPACE]/[OWNER-NAME]`. The owner + // namespace is empty for cluster-scoped placement bindings. + // + // Note that slashes are used to avoid unexpected collisions. + WorkOwnedByPlacementBindingCustomFieldValFmt = "%s/%s" +) + +var ( + workOwnedByPlacementPolicyFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + ownerNSName := obj.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerName := obj.GetLabels()[placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey] + if ownerName == "" { + wrappedErr := errors.NewUnexpectedError(nil, "work object is missing required labels") + return nil, wrappedErr + } + return []string{fmt.Sprintf(WorkOwnedByPlacementPolicyCustomFieldValFmt, ownerNSName, ownerName)}, nil + } + + workOwnedByPlacementBindingFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + ownerNSName := obj.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerName := obj.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if ownerName == "" { + wrappedErr := errors.NewUnexpectedError(nil, "work object is missing required labels") + return nil, wrappedErr + } + return []string{fmt.Sprintf(WorkOwnedByPlacementBindingCustomFieldValFmt, ownerNSName, ownerName)}, nil + } +) + +// SetupWithMemberAgentControllerManager sets up the indices that controllers from the KubeFleet member agent need to +// run properly. +// +// It must be called before the manager starts. +func SetupWithMemberAgentControllerManager(ctx context.Context, mgr ctrl.Manager) error { + fieldIdxer := mgr.GetFieldIndexer() + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.Work{}, + WorkOwnedByPlacementPolicyCustomFieldName, workOwnedByPlacementPolicyFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up work owner and placement policy field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.Work{}, + WorkOwnedByPlacementBindingCustomFieldName, workOwnedByPlacementBindingFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up work owner and placement binding field index") + } + + return nil +} diff --git a/pkg/v1/utils/fieldindexers/utils.go b/pkg/v1/utils/fieldindexers/utils.go new file mode 100644 index 000000000..0a2366c36 --- /dev/null +++ b/pkg/v1/utils/fieldindexers/utils.go @@ -0,0 +1,48 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package fieldindexers + +import ( + "context" + + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/kubefleet-dev/kubefleet/pkg/utils/errors" +) + +type fieldValueExtractor func(obj client.Object) ([]string, error) + +func indexCompositeField(ctx context.Context, + fieldIdxer client.FieldIndexer, + obj client.Object, + fieldName string, fieldValueExt fieldValueExtractor) error { + if err := fieldIdxer.IndexField(ctx, obj, fieldName, func(rawObj client.Object) []string { + fieldVals, extErr := fieldValueExt(rawObj) + if extErr != nil { + wrappedErr := errors.NewUnexpectedError(extErr, "failed to extract field value", "object", klog.KObj(rawObj)) + klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) + return nil + } + return fieldVals + }); err != nil { + wrappedErr := errors.NewUnexpectedError(err, "", "fieldName", fieldName, "object", klog.KObj(obj)) + klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) + return wrappedErr + } + return nil +} diff --git a/pkg/v1/utils/ownerreferences/refs.go b/pkg/v1/utils/ownerreferences/refs.go new file mode 100644 index 000000000..136e4054e --- /dev/null +++ b/pkg/v1/utils/ownerreferences/refs.go @@ -0,0 +1,63 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package ownerreferences provides utilities for managing owner references of Kubernetes objects. +package ownerreferences + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" +) + +// Disown removes a specific owner reference from the given object. +// +// Note that this function will check for UID matches. To disown an object without checking for UIDs, +// call controllerutil.RemoveOwnerReference instead. +func Disown(obj *unstructured.Unstructured, owner *metav1.OwnerReference) { + ownerRefs := obj.GetOwnerReferences() + updatedOwnerRefs := make([]metav1.OwnerReference, 0, len(ownerRefs)) + + // Re-build the owner references; remove the given one from the list. + for idx := range ownerRefs { + if AreEqual(&ownerRefs[idx], owner) { + // Skip the expected owner reference. + continue + } + updatedOwnerRefs = append(updatedOwnerRefs, ownerRefs[idx]) + } + obj.SetOwnerReferences(updatedOwnerRefs) +} + +// AreEqual checks if two owner references are equal by comparing their UID, Name, Kind, and APIVersion fields. +func AreEqual(a, b *metav1.OwnerReference) bool { + return a.UID == b.UID && + a.Name == b.Name && + a.Kind == b.Kind && + a.APIVersion == b.APIVersion +} + +// Own adds a specific owner reference to the given object if it is not already present. +func Own(obj *unstructured.Unstructured, owner *metav1.OwnerReference) { + ownerRefs := obj.GetOwnerReferences() + for idx := range ownerRefs { + if AreEqual(&ownerRefs[idx], owner) { + // The object is already owned by the given owner. + return + } + } + ownerRefs = append(ownerRefs, *owner) + obj.SetOwnerReferences(ownerRefs) +} diff --git a/test/apis/v1alpha1/zz_generated.deepcopy.go b/test/apis/v1alpha1/zz_generated.deepcopy.go index 081bec913..143bdee7b 100644 --- a/test/apis/v1alpha1/zz_generated.deepcopy.go +++ b/test/apis/v1alpha1/zz_generated.deepcopy.go @@ -21,7 +21,7 @@ limitations under the License. package v1alpha1 import ( - "k8s.io/apimachinery/pkg/apis/meta/v1" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" ) From d5b45d490278b01e62e4662cbad96840083aaee9 Mon Sep 17 00:00:00 2001 From: Simon Waight Date: Wed, 7 Oct 2026 11:00:20 +1100 Subject: [PATCH 10/20] docs: update security vulnerability instructions (#952) * feat: [FEP-0001] add work applier (1/) (#911) Signed-off-by: Simon Waight * Revise security vulnerability reporting instructions Updated the reporting process for security vulnerabilities to direct users to the KubeFleet Security Advisories feature. Signed-off-by: Simon Waight --------- Signed-off-by: Simon Waight Co-authored-by: Chen Yu --- SECURITY.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 3ee4c6508..64d2fd124 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -58,16 +58,18 @@ being finalized: (see the [CNCF TAG-Security `SECURITY.md` template](https://github.com/cncf/tag-security) for the conventional `cncf--distributors-announce@lists.cncf.io` form). Whether KubeFleet stands one up depends on demonstrated downstream demand. -- **GitHub private vulnerability reporting:** to be enabled on this repository as the - preferred reporting channel; the maintainer mailing list (see below) remains the fallback - until it is. +- **GitHub private vulnerability reporting:** report them via the KubeFleet Security Advisories [Report a vulnerability](https://github.com/kubefleet-dev/kubefleet/security/advisories/new) feature.. This section will be updated as each item is decided. ## Reporting Security Issues -**Please do not report security vulnerabilities through public GitHub issues.** Instead, -report them to the [KubeFleet maintainers](mailto:kubefleet@microsoft.com). +**Please do not report security vulnerabilities through public GitHub issues.** + +Instead, report them via the KubeFleet Security Advisories [Report a vulnerability](https://github.com/kubefleet-dev/kubefleet/security/advisories/new) feature. + +You can also follow up with the [KubeFleet maintainers](mailto:kubefleet@microsoft.com). + We prefer all communications to be in English. You should receive a response as soon as possible. If for some reason you do not, please From ea05fcb192320c656fdace030eaabc2e0145cb10 Mon Sep 17 00:00:00 2001 From: Akshita kumari <110122283+akshita317@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:19:43 +0530 Subject: [PATCH 11/20] fix: surface unknown eviction conditions in draincluster (#767) * fix: surface unknown eviction conditions in draincluster The draincluster command waits for each eviction to reach a terminal state but then only handled True/False Valid and Executed conditions. A missing or Unknown condition fell through and was reported as a successful drain, masking an uncertain eviction (addressing the pre-existing TODO in draincluster.go). Add safeguards so a missing or Unknown Valid or Executed condition marks the drain unsuccessful and logs that the drain could not be confirmed, instead of silently proceeding. Fixes #659 Signed-off-by: Akshita <110122283+akshita317@users.noreply.github.com> * test: extract evaluateEviction helper and add table-driven tests Address review feedback by extracting the eviction Valid/Executed condition evaluation into a pure evaluateEviction helper, and cover the True/False/Unknown/missing combinations (including the invalid-but-drained reasons) with table-driven unit tests. No behavior change. Signed-off-by: Akshita <110122283+akshita317@users.noreply.github.com> * fix: handle unexpected and non-True eviction statuses defensively Address review feedback: - add an explicit evictionResultExecuted case and a default to the drain switch, so an unexpected classification is treated as unconfirmed rather than a successful drain - evaluateEviction now treats only an explicit True/False status as definitive; Unknown, empty, or unexpected statuses map to indeterminate - add a test case for an empty/invalid Executed status Signed-off-by: Akshita <110122283+akshita317@users.noreply.github.com> * fix: match invalid eviction detail against condition Message markEvictionInvalid sets Reason to the generic ClusterResourcePlacementEvictionInvalidReason and stores the specific detail in Message, so comparing Reason against the EvictionInvalid*Message constants never matched and the invalid-but-drained branch was unreachable. Compare against Message instead, and log Message rather than the generic Reason when reporting an invalid eviction that still drained. Signed-off-by: Akshita <110122283+akshita317@users.noreply.github.com> --------- Signed-off-by: Akshita <110122283+akshita317@users.noreply.github.com> Co-authored-by: Yetkin Timocin Co-authored-by: Simon Waight --- tools/fleet/cmd/draincluster/draincluster.go | 107 +++++++++++++---- .../cmd/draincluster/draincluster_test.go | 111 ++++++++++++++++++ 2 files changed, 196 insertions(+), 22 deletions(-) diff --git a/tools/fleet/cmd/draincluster/draincluster.go b/tools/fleet/cmd/draincluster/draincluster.go index 7dc6f334b..1968bad57 100644 --- a/tools/fleet/cmd/draincluster/draincluster.go +++ b/tools/fleet/cmd/draincluster/draincluster.go @@ -186,38 +186,101 @@ func (o *drainOptions) drain(ctx context.Context) (bool, error) { return false, fmt.Errorf("failed to wait for eviction %s for CRP %s targeting member cluster %s to reach terminal state: %w", evictionName, crpName, o.clusterName, err) } - // TODO: add safeguards to check if eviction conditions are set to unknown. - validCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeValid)) - if validCondition != nil && validCondition.Status == metav1.ConditionFalse { - // check to see if CRP is missing or CRP is being deleted or CRB is missing. - if validCondition.Reason == condition.EvictionInvalidMissingCRPMessage || - validCondition.Reason == condition.EvictionInvalidDeletingCRPMessage || - validCondition.Reason == condition.EvictionInvalidMissingCRBMessage { - log.Printf("eviction %s is invalid with reason %s for CRP %s targeting member cluster %s, but drain will succeed", evictionName, validCondition.Reason, crpName, o.clusterName) - continue - } - } - executedCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeExecuted)) - if executedCondition == nil || executedCondition.Status == metav1.ConditionFalse { + // Classify the terminal eviction. A missing or Unknown Valid/Executed + // condition is never treated as success, so an uncertain eviction is not + // reported as a completed drain (see evaluateEviction). + switch evaluateEviction(&eviction) { + case evictionResultInvalidButDrained: + validCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeValid)) + log.Printf("eviction %s is invalid: %s for CRP %s targeting member cluster %s, but drain will succeed", evictionName, validCondition.Message, crpName, o.clusterName) + continue + case evictionResultIndeterminate: + isDrainSuccessful = false + log.Printf("eviction %s has a missing or unknown Valid or Executed condition for CRP %s targeting member cluster %s; cannot confirm drain", evictionName, crpName, o.clusterName) + continue + case evictionResultNotExecuted: isDrainSuccessful = false log.Printf("eviction %s was not executed successfully for CRP %s targeting member cluster %s", evictionName, crpName, o.clusterName) continue - } - log.Printf("eviction %s was executed successfully for CRP %s targeting member cluster %s", evictionName, crpName, o.clusterName) - // log each cluster scoped resource evicted for CRP. - clusterScopedResourceIdentifiers, err := o.collectClusterScopedResourcesSelectedByCRP(ctx, crpName) - if err != nil { - log.Printf("failed to collect cluster scoped resources selected by CRP %s: %v", crpName, err) + case evictionResultExecuted: + log.Printf("eviction %s was executed successfully for CRP %s targeting member cluster %s", evictionName, crpName, o.clusterName) + // log each cluster scoped resource evicted for CRP. + clusterScopedResourceIdentifiers, err := o.collectClusterScopedResourcesSelectedByCRP(ctx, crpName) + if err != nil { + log.Printf("failed to collect cluster scoped resources selected by CRP %s: %v", crpName, err) + continue + } + for _, resourceIdentifier := range clusterScopedResourceIdentifiers { + log.Printf("evicted resource %s propagated by CRP %s targeting member cluster %s", generateResourceIdentifierKey(resourceIdentifier), crpName, o.clusterName) + } + default: + // Any unexpected classification is treated as unconfirmed rather than + // success, so a future evictionResult value can never silently pass a drain. + isDrainSuccessful = false + log.Printf("eviction %s returned an unexpected evaluation result for CRP %s targeting member cluster %s; cannot confirm drain", evictionName, crpName, o.clusterName) continue } - for _, resourceIdentifier := range clusterScopedResourceIdentifiers { - log.Printf("evicted resource %s propagated by CRP %s targeting member cluster %s", generateResourceIdentifierKey(resourceIdentifier), crpName, o.clusterName) - } } return isDrainSuccessful, nil } +// evictionResult classifies the terminal state of a ClusterResourcePlacementEviction +// based on its Valid and Executed conditions. +type evictionResult int + +const ( + // evictionResultIndeterminate means the eviction's Valid or Executed condition + // is missing or Unknown, so the drain cannot be confirmed. + evictionResultIndeterminate evictionResult = iota + // evictionResultInvalidButDrained means the eviction is invalid for a reason + // that still leaves the cluster drained (missing/deleting CRP or missing CRB). + evictionResultInvalidButDrained + // evictionResultNotExecuted means the eviction did not execute. + evictionResultNotExecuted + // evictionResultExecuted means the eviction executed successfully. + evictionResultExecuted +) + +// evaluateEviction inspects a terminal eviction's Valid and Executed conditions and +// classifies the outcome. Only an explicit "True"/"False" status is treated as +// definitive; a missing, Unknown, or otherwise unexpected status is never treated as +// a successful drain. +func evaluateEviction(eviction *placementv1beta1.ClusterResourcePlacementEviction) evictionResult { + validCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeValid)) + if validCondition == nil { + return evictionResultIndeterminate + } + if validCondition.Status == metav1.ConditionFalse { + // An invalid eviction whose detail indicates a missing/deleting CRP or a + // missing CRB still means the cluster is drained. markEvictionInvalid stores + // that detail in the condition Message (Reason is a generic invalid reason). + if validCondition.Message == condition.EvictionInvalidMissingCRPMessage || + validCondition.Message == condition.EvictionInvalidDeletingCRPMessage || + validCondition.Message == condition.EvictionInvalidMissingCRBMessage { + return evictionResultInvalidButDrained + } + // Any other invalid eviction falls through to the Executed check. + } else if validCondition.Status != metav1.ConditionTrue { + // Unknown, empty, or any unexpected status: the drain cannot be confirmed. + return evictionResultIndeterminate + } + + executedCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeExecuted)) + if executedCondition == nil { + return evictionResultIndeterminate + } + switch executedCondition.Status { + case metav1.ConditionTrue: + return evictionResultExecuted + case metav1.ConditionFalse: + return evictionResultNotExecuted + default: + // Unknown, empty, or any unexpected status: the drain cannot be confirmed. + return evictionResultIndeterminate + } +} + func (o *drainOptions) cordon(ctx context.Context) error { // add taint to member cluster to ensure resources aren't scheduled on it. return retry.RetryOnConflict(retry.DefaultRetry, func() error { diff --git a/tools/fleet/cmd/draincluster/draincluster_test.go b/tools/fleet/cmd/draincluster/draincluster_test.go index fd97c8346..453b8ab3e 100644 --- a/tools/fleet/cmd/draincluster/draincluster_test.go +++ b/tools/fleet/cmd/draincluster/draincluster_test.go @@ -32,6 +32,7 @@ import ( clusterv1beta1 "github.com/kubefleet-dev/kubefleet/apis/cluster/v1beta1" placementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" + "github.com/kubefleet-dev/kubefleet/pkg/utils/condition" toolsutils "github.com/kubefleet-dev/kubefleet/tools/utils" ) @@ -510,3 +511,113 @@ func serviceScheme(t *testing.T) *runtime.Scheme { } return scheme } + +func TestEvaluateEviction(t *testing.T) { + validType := string(placementv1beta1.PlacementEvictionConditionTypeValid) + executedType := string(placementv1beta1.PlacementEvictionConditionTypeExecuted) + + tests := []struct { + name string + conditions []metav1.Condition + want evictionResult + }{ + { + name: "valid true, executed true", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: metav1.ConditionTrue, Reason: "Executed"}, + }, + want: evictionResultExecuted, + }, + { + name: "valid true, executed false", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: metav1.ConditionFalse, Reason: "NotExecuted"}, + }, + want: evictionResultNotExecuted, + }, + { + name: "valid true, executed unknown", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: metav1.ConditionUnknown, Reason: "Unknown"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid true, executed empty/invalid status", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: "", Reason: "EmptyStatus"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid true, executed missing", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid unknown", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionUnknown, Reason: "Unknown"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid missing", + conditions: nil, + want: evictionResultIndeterminate, + }, + { + name: "valid false, message missing CRP", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: condition.EvictionInvalidMissingCRPMessage}, + }, + want: evictionResultInvalidButDrained, + }, + { + name: "valid false, message deleting CRP", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: condition.EvictionInvalidDeletingCRPMessage}, + }, + want: evictionResultInvalidButDrained, + }, + { + name: "valid false, message missing CRB", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: condition.EvictionInvalidMissingCRBMessage}, + }, + want: evictionResultInvalidButDrained, + }, + { + name: "valid false with unrecognized message, executed true", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: "some other invalid detail"}, + {Type: executedType, Status: metav1.ConditionTrue, Reason: "Executed"}, + }, + want: evictionResultExecuted, + }, + { + name: "valid false with unrecognized message, executed unknown", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: "some other invalid detail"}, + {Type: executedType, Status: metav1.ConditionUnknown, Reason: "Unknown"}, + }, + want: evictionResultIndeterminate, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + eviction := &placementv1beta1.ClusterResourcePlacementEviction{} + eviction.SetConditions(tc.conditions...) + if got := evaluateEviction(eviction); got != tc.want { + t.Errorf("evaluateEviction() = %v, want %v", got, tc.want) + } + }) + } +} From 29f4f73bb51c2407db9757fa37f7e9b6a8f371c9 Mon Sep 17 00:00:00 2001 From: Britania Rodriguez Reyes <145056127+britaniar@users.noreply.github.com> Date: Thu, 8 Oct 2026 14:13:54 -0700 Subject: [PATCH 12/20] fix: preserve update run transition timestamp (#935) * fix: preserve update run transition timestamp Use the selected condition's LastTransitionTime for the UpdateRun status metric so reconciles and hub-agent restarts do not restamp stale statuses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes * test: align update run metric expectations by gauge Keep the shared metrics test helper sorted by gauge value. For UpdateRun status metrics, normalize expected gauge values from the gathered metrics before comparing so the existing gauge-based ordering remains authoritative after the gauge changed from emission time to persisted LastTransitionTime. Add a label tie-breaker only for equal gauge values in this UpdateRun-specific comparison. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes * test: derive expected UpdateRun gauge values from persisted condition time Remove the alignExpectedMetricGaugeValues workaround and its local label-key/gauge-primary comparator introduced on this branch. The generate*Metric test helpers now derive their expected gauge value directly from updateRun.Status.Conditions[...].LastTransitionTime via a new conditionTransitionTimeSeconds helper (re-fetching updateRun and waiting for the condition to be persisted), matching the exact formula production code uses (float64(cond.LastTransitionTime.UnixNano()) / float64(time.Second)) instead of a time.Now() placeholder. validateUpdateRunMetricsEmitted goes back to using the shared metricsutils.MetricsCmpOptions directly, with no per-package workaround. Since gauges now reflect second-precision persisted timestamps, two distinct condition transitions can legitimately tie on gauge value, so pairing metrics by sorting on gauge value is not reliable. The shared comparator in test/utils/metrics/metrics.go now sorts by a stable label-identity key instead, which guarantees correct got/want pairing regardless of gauge value ties. The Gauge comparer itself keeps its existing permissive (value > 0) check, so this does not change behavior for other consumers of the shared file (e.g. the placement package), which still uses time.Now()-based expected values. Signed-off-by: Britania Rodriguez Reyes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test: compare update run metrics with transition times Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes * test: use new expressions for UpdateRun metric pointers Adopt Go 1.26 pointer initialization for the new status metric gauge expectations.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes --------- Signed-off-by: Britania Rodriguez Reyes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- ...2026-09-23-1617-hub-condition-timestamp.md | 26 ++++ .../updaterun/controller_integration_test.go | 142 ++++++++++++++++-- pkg/controllers/updaterun/metrics.go | 25 +-- pkg/controllers/updaterun/metrics_test.go | 124 +++++++++++++++ pkg/metrics/hub/metrics.go | 4 +- 5 files changed, 297 insertions(+), 24 deletions(-) create mode 100644 .github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md diff --git a/.github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md b/.github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md new file mode 100644 index 000000000..4058a809d --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md @@ -0,0 +1,26 @@ +# Fix: Preserve UpdateRun Condition Transition Timestamp + +## Overview + +Prevent hub-agent reconciliation and restarts from restamping the UpdateRun status metric when the selected Kubernetes condition has not transitioned. + +## Plan + +1. Trace UpdateRun condition reconciliation and metric emission. +2. Emit the selected condition's persisted `LastTransitionTime` instead of reconciliation time. +3. Add focused unit tests for unchanged failed conditions and actual status transitions. +4. Run the smallest relevant UpdateRun unit test suite. + +## Success Criteria + +- [x] Re-emitting an unchanged failed condition preserves the metric timestamp. +- [x] A condition status transition emits the new transition timestamp. +- [x] Focused UpdateRun tests pass. + +## Implementation Notes + +- Kubernetes `meta.SetStatusCondition` already preserves `LastTransitionTime` when `Status` is unchanged and advances it when `Status` changes. +- The restamping occurred in metric emission through `SetToCurrentTime()`, independently of the persisted condition. +- `emitUpdateRunStatusMetric` now sets the gauge from the selected condition's `LastTransitionTime`. +- Added table-driven unit coverage for failed, initialized, executing, and succeeded runs. Each case emits twice to prove reconciliation does not restamp the selected condition's timestamp. +- Validation: `go test ./pkg/controllers/updaterun -run 'TestEmitUpdateRunStatusMetricUsesConditionTransitionTime|TestDetermineFailureType|TestIsFailureReason' -count=1`. diff --git a/pkg/controllers/updaterun/controller_integration_test.go b/pkg/controllers/updaterun/controller_integration_test.go index 1eb488602..293d629bc 100644 --- a/pkg/controllers/updaterun/controller_integration_test.go +++ b/pkg/controllers/updaterun/controller_integration_test.go @@ -20,10 +20,13 @@ import ( "context" "encoding/json" "fmt" + "sort" "strconv" + "strings" "time" "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" prometheusclientmodel "github.com/prometheus/client_model/go" @@ -46,7 +49,6 @@ import ( hubmetrics "github.com/kubefleet-dev/kubefleet/pkg/metrics/hub" "github.com/kubefleet-dev/kubefleet/pkg/utils" "github.com/kubefleet-dev/kubefleet/pkg/utils/condition" - metricsutils "github.com/kubefleet-dev/kubefleet/test/utils/metrics" ) const ( @@ -73,6 +75,34 @@ var ( updateRunNamespacedName types.NamespacedName ) +var updateRunStatusMetricCmpOptions = []cmp.Option{ + // Metric slices are pre-sorted deterministically (see sortMetricsByGaugeThenLabel) before + // diffing, so we intentionally do not also sort *Metric here via cmpopts.SortSlices. + cmpopts.SortSlices(func(a, b *prometheusclientmodel.LabelPair) bool { + return a.GetName() < b.GetName() + }), + // No custom Comparer for *Gauge is registered here: leaving it to cmp's default + // (exported-field) comparison means gauge values, which now carry the exact + // persisted condition.LastTransitionTime, are compared for exact equality. + cmpopts.IgnoreUnexported(prometheusclientmodel.Metric{}, prometheusclientmodel.LabelPair{}, prometheusclientmodel.Gauge{}), +} + +// sortMetricsByGaugeThenLabel sorts metrics in place by gauge value, breaking ties by +// label identity. Gauge values now reflect persisted (second-precision) condition +// LastTransitionTime, so two distinct condition transitions can legitimately tie on +// gauge value; the label-identity tie-break guarantees a deterministic, matching order +// between the "got" (scraped) and "want" (test-authored) slices without relying on +// cmp's own slice-sorting/pairing for *Metric. +func sortMetricsByGaugeThenLabel(metrics []*prometheusclientmodel.Metric) { + sort.Slice(metrics, func(i, j int) bool { + gi, gj := metrics[i].GetGauge().GetValue(), metrics[j].GetGauge().GetValue() + if gi != gj { + return gi < gj + } + return metricLabelKey(metrics[i]) < metricLabelKey(metrics[j]) + }) +} + var _ = Describe("Test the clusterStagedUpdateRun controller", func() { BeforeEach(func() { @@ -238,7 +268,7 @@ func resetUpdateRunMetrics() { hubmetrics.FleetUpdateRunApprovalRequestLatencySeconds.Reset() } -// validateUpdateRunMetricsEmitted validates the update run status metrics are emitted and are emitted in the correct order. +// validateUpdateRunMetricsEmitted validates the update run status metrics are emitted with the expected labels and values. func validateUpdateRunMetricsEmitted(wantMetrics ...*prometheusclientmodel.Metric) { Eventually(func() error { metricFamilies, err := ctrlmetrics.Registry.Gather() @@ -251,8 +281,12 @@ func validateUpdateRunMetricsEmitted(wantMetrics ...*prometheusclientmodel.Metri gotMetrics = mf.GetMetric() } } - - if diff := cmp.Diff(gotMetrics, wantMetrics, metricsutils.MetricsCmpOptions...); diff != "" { + // Pre-sort both slices deterministically (by gauge value, then label identity) so + // that got/want are paired up correctly even when multiple metrics tie on gauge + // value; the cmp options below intentionally do not re-sort *Metric slices. + sortMetricsByGaugeThenLabel(gotMetrics) + sortMetricsByGaugeThenLabel(wantMetrics) + if diff := cmp.Diff(gotMetrics, wantMetrics, updateRunStatusMetricCmpOptions...); diff != "" { return fmt.Errorf("update run status metrics mismatch (-got, +want):\n%s", diff) } @@ -260,6 +294,16 @@ func validateUpdateRunMetricsEmitted(wantMetrics ...*prometheusclientmodel.Metri }, timeout, interval).Should(Succeed(), "failed to validate the update run status metrics") } +func metricLabelKey(metric *prometheusclientmodel.Metric) string { + labels := metric.GetLabel() + pairs := make([]string, 0, len(labels)) + for _, label := range labels { + pairs = append(pairs, label.GetName()+"="+label.GetValue()) + } + sort.Strings(pairs) + return strings.Join(pairs, ";") +} + // validateUpdateRunApprovalStageTaskMetric validates the update run approval stage task metric by checking labels and count. func validateUpdateRunApprovalStageTaskMetric(wantMetrics ...*prometheusclientmodel.Metric) { Eventually(func() error { @@ -390,12 +434,84 @@ func generateMetricsLabels( } } +// conditionTransitionTimeSeconds returns the gauge value that emitUpdateRunStatusMetric +// would have set for updateRun's condition of the given type and reason, derived from the +// condition's persisted LastTransitionTime (rather than the time the test happens to run at). +// +// Callers typically build the expected metric before the controller has necessarily +// finished reconciling (validateUpdateRunMetricsEmitted then polls until it converges), +// so this re-fetches updateRun from the API server and waits for the condition to appear +// rather than requiring it to already be present on the (possibly stale) object passed in. +// +// A single condition type (e.g. Progressing) cycles through multiple reasons over an +// updateRun's lifetime (Waiting -> Progressing -> Stuck, etc.), but the live object only +// ever exposes the *current* reason's LastTransitionTime; once the condition moves on to a +// new reason, the previous reason's transition time can no longer be recovered from the +// object (Kubernetes conditions retain only the most recent transition per type). The metric +// emitted while that earlier reason was current, however, remains persisted in the metrics +// registry under its own label set (each reason has a distinct time series that is never +// overwritten by a later reason). So when the requested reason is no longer the condition's +// current reason, this falls back to reading the already-persisted metric for that reason +// straight out of the registry instead of the (now unavailable) live condition. +func conditionTransitionTimeSeconds(updateRun *placementv1beta1.ClusterStagedUpdateRun, conditionType placementv1beta1.StagedUpdateRunConditionType, reason string) float64 { + var value float64 + Eventually(func() error { + if err := k8sClient.Get(ctx, types.NamespacedName{Name: updateRun.Name, Namespace: updateRun.Namespace}, updateRun); err != nil { + return err + } + cond := meta.FindStatusCondition(updateRun.Status.Conditions, string(conditionType)) + if cond == nil { + return fmt.Errorf("condition %s not found on updateRun %s", conditionType, updateRun.Name) + } + if cond.Reason == reason { + value = float64(cond.LastTransitionTime.UnixNano()) / float64(time.Second) + return nil + } + v, found, err := persistedConditionGaugeValue(updateRun, conditionType, reason) + if err != nil { + return err + } + if !found { + return fmt.Errorf("condition %s on updateRun %s currently has reason %q, want %q, and no metric for reason %q was found in the registry", conditionType, updateRun.Name, cond.Reason, reason, reason) + } + value = v + return nil + }, timeout, interval).Should(Succeed(), "failed to determine transition time for condition %s (reason %s) on updateRun %s", conditionType, reason, updateRun.Name) + return value +} + +// persistedConditionGaugeValue looks up the already-emitted update run status gauge for the +// given condition type and reason directly from the metrics registry. It returns found=false +// if no such metric has been emitted (yet). +func persistedConditionGaugeValue(updateRun *placementv1beta1.ClusterStagedUpdateRun, conditionType placementv1beta1.StagedUpdateRunConditionType, reason string) (float64, bool, error) { + metricFamilies, err := ctrlmetrics.Registry.Gather() + if err != nil { + return 0, false, fmt.Errorf("failed to gather metrics: %w", err) + } + for _, mf := range metricFamilies { + if mf.GetName() != "fleet_workload_update_run_status_last_timestamp_seconds" { + continue + } + for _, m := range mf.GetMetric() { + labels := make(map[string]string, len(m.GetLabel())) + for _, l := range m.GetLabel() { + labels[l.GetName()] = l.GetValue() + } + if labels["namespace"] == updateRun.Namespace && labels["name"] == updateRun.Name && + labels["condition"] == string(conditionType) && labels["reason"] == reason { + return m.GetGauge().GetValue(), true, nil + } + } + } + return 0, false, nil +} + func generateInitializationSucceededMetric(state placementv1beta1.State, updateRun *placementv1beta1.ClusterStagedUpdateRun) *prometheusclientmodel.Metric { return &prometheusclientmodel.Metric{ Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionInitialized), string(metav1.ConditionTrue), condition.UpdateRunInitializeSucceededReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionInitialized, condition.UpdateRunInitializeSucceededReason)), }, } } @@ -405,7 +521,7 @@ func generateInitializationFailedMetric(state placementv1beta1.State, updateRun Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionInitialized), string(metav1.ConditionFalse), condition.UpdateRunInitializeFailedReason, failureType), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionInitialized, condition.UpdateRunInitializeFailedReason)), }, } } @@ -415,7 +531,7 @@ func generateProgressingMetric(state placementv1beta1.State, updateRun *placemen Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionTrue), condition.UpdateRunProgressingReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunProgressingReason)), }, } } @@ -425,7 +541,7 @@ func generateWaitingMetric(state placementv1beta1.State, updateRun *placementv1b Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionFalse), condition.UpdateRunWaitingReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunWaitingReason)), }, } } @@ -435,7 +551,7 @@ func generateStuckMetric(state placementv1beta1.State, updateRun *placementv1bet Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionFalse), condition.UpdateRunStuckReason, string(hubmetrics.UpdateRunFailureTypeInternalError)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStuckReason)), }, } } @@ -445,7 +561,7 @@ func generateFailedMetric(state placementv1beta1.State, updateRun *placementv1be Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionSucceeded), string(metav1.ConditionFalse), condition.UpdateRunFailedReason, failureType), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionSucceeded, condition.UpdateRunFailedReason)), }, } } @@ -455,7 +571,7 @@ func generateStoppingMetric(state placementv1beta1.State, updateRun *placementv1 Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionUnknown), condition.UpdateRunStoppingReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStoppingReason)), }, } } @@ -465,7 +581,7 @@ func generateStoppedMetric(state placementv1beta1.State, updateRun *placementv1b Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionFalse), condition.UpdateRunStoppedReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStoppedReason)), }, } } @@ -475,7 +591,7 @@ func generateSucceededMetric(state placementv1beta1.State, updateRun *placementv Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionSucceeded), string(metav1.ConditionTrue), condition.UpdateRunSucceededReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionSucceeded, condition.UpdateRunSucceededReason)), }, } } diff --git a/pkg/controllers/updaterun/metrics.go b/pkg/controllers/updaterun/metrics.go index 1012bcb23..5c1007a9d 100644 --- a/pkg/controllers/updaterun/metrics.go +++ b/pkg/controllers/updaterun/metrics.go @@ -89,25 +89,19 @@ func emitUpdateRunStatusMetric(updateRun placementv1beta1.UpdateRunObj) { updateRunStatus := updateRun.GetUpdateRunStatus() succeedCond := meta.FindStatusCondition(updateRunStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionSucceeded)) if succeedCond != nil && succeedCond.ObservedGeneration == generation { - failureType := determineFailureType(succeedCond) - hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues(updateRun.GetNamespace(), updateRun.GetName(), string(state), - string(placementv1beta1.StagedUpdateRunConditionSucceeded), string(succeedCond.Status), succeedCond.Reason, string(failureType)).SetToCurrentTime() + setUpdateRunStatusMetric(updateRun, state, succeedCond) return } progressingCond := meta.FindStatusCondition(updateRunStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionProgressing)) if progressingCond != nil && progressingCond.ObservedGeneration == generation { - failureType := determineFailureType(progressingCond) - hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues(updateRun.GetNamespace(), updateRun.GetName(), string(state), - string(placementv1beta1.StagedUpdateRunConditionProgressing), string(progressingCond.Status), progressingCond.Reason, string(failureType)).SetToCurrentTime() + setUpdateRunStatusMetric(updateRun, state, progressingCond) return } initializedCond := meta.FindStatusCondition(updateRunStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionInitialized)) if initializedCond != nil && initializedCond.ObservedGeneration == generation { - failureType := determineFailureType(initializedCond) - hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues(updateRun.GetNamespace(), updateRun.GetName(), string(state), - string(placementv1beta1.StagedUpdateRunConditionInitialized), string(initializedCond.Status), initializedCond.Reason, string(failureType)).SetToCurrentTime() + setUpdateRunStatusMetric(updateRun, state, initializedCond) return } @@ -115,6 +109,19 @@ func emitUpdateRunStatusMetric(updateRun placementv1beta1.UpdateRunObj) { klog.V(2).InfoS("There's no valid status condition on updateRun, status updating failed possibly", "updateRun", klog.KObj(updateRun)) } +func setUpdateRunStatusMetric(updateRun placementv1beta1.UpdateRunObj, state placementv1beta1.State, cond *metav1.Condition) { + failureType := determineFailureType(cond) + hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues( + updateRun.GetNamespace(), + updateRun.GetName(), + string(state), + cond.Type, + string(cond.Status), + cond.Reason, + string(failureType), + ).Set(float64(cond.LastTransitionTime.UnixNano()) / float64(time.Second)) +} + // recordApprovalRequestLatency records the time from approval request creation to user approval. func recordApprovalRequestLatency( stageTaskStatus *placementv1beta1.StageTaskStatus, diff --git a/pkg/controllers/updaterun/metrics_test.go b/pkg/controllers/updaterun/metrics_test.go index a1bb15321..bb8636578 100644 --- a/pkg/controllers/updaterun/metrics_test.go +++ b/pkg/controllers/updaterun/metrics_test.go @@ -19,8 +19,10 @@ package updaterun import ( "fmt" "testing" + "time" "github.com/google/go-cmp/cmp" + "github.com/prometheus/client_golang/prometheus/testutil" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" @@ -199,3 +201,125 @@ func TestIsFailureReason(t *testing.T) { }) } } + +func TestEmitUpdateRunStatusMetricUsesConditionTransitionTime(t *testing.T) { + tests := []struct { + name string + state v1beta1.State + conditions []metav1.Condition + wantType v1beta1.StagedUpdateRunConditionType + wantStatus metav1.ConditionStatus + wantReason string + wantFailure hubmetrics.UpdateRunFailureType + wantTime metav1.Time + }{ + { + name: "failed run keeps failure transition time", + state: v1beta1.StateRun, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionSucceeded, metav1.ConditionFalse, condition.UpdateRunFailedReason, "internal error", time.Unix(100, 123456789)), + }, + wantType: v1beta1.StagedUpdateRunConditionSucceeded, + wantStatus: metav1.ConditionFalse, + wantReason: condition.UpdateRunFailedReason, + wantFailure: hubmetrics.UpdateRunFailureTypeInternalError, + wantTime: metav1.NewTime(time.Unix(100, 123456789)), + }, + { + name: "initialized run uses initialization transition time", + state: v1beta1.StateInitialize, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionInitialized, metav1.ConditionTrue, condition.UpdateRunInitializeSucceededReason, "initialized", time.Unix(200, 0)), + }, + wantType: v1beta1.StagedUpdateRunConditionInitialized, + wantStatus: metav1.ConditionTrue, + wantReason: condition.UpdateRunInitializeSucceededReason, + wantFailure: hubmetrics.UpdateRunFailureTypeNone, + wantTime: metav1.NewTime(time.Unix(200, 0)), + }, + { + name: "executing run uses progressing transition time", + state: v1beta1.StateRun, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionInitialized, metav1.ConditionTrue, condition.UpdateRunInitializeSucceededReason, "initialized", time.Unix(200, 0)), + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionTrue, condition.UpdateRunProgressingReason, "progressing", time.Unix(300, 0)), + }, + wantType: v1beta1.StagedUpdateRunConditionProgressing, + wantStatus: metav1.ConditionTrue, + wantReason: condition.UpdateRunProgressingReason, + wantFailure: hubmetrics.UpdateRunFailureTypeNone, + wantTime: metav1.NewTime(time.Unix(300, 0)), + }, + { + name: "succeeded run uses success transition time", + state: v1beta1.StateRun, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionInitialized, metav1.ConditionTrue, condition.UpdateRunInitializeSucceededReason, "initialized", time.Unix(200, 0)), + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionFalse, condition.UpdateRunSucceededReason, "completed", time.Unix(400, 0)), + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionSucceeded, metav1.ConditionTrue, condition.UpdateRunSucceededReason, "succeeded", time.Unix(400, 0)), + }, + wantType: v1beta1.StagedUpdateRunConditionSucceeded, + wantStatus: metav1.ConditionTrue, + wantReason: condition.UpdateRunSucceededReason, + wantFailure: hubmetrics.UpdateRunFailureTypeNone, + wantTime: metav1.NewTime(time.Unix(400, 0)), + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.Reset() + t.Cleanup(hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.Reset) + + updateRun := &v1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-update-run", + Generation: 1, + }, + Spec: v1beta1.UpdateRunSpec{ + State: tc.state, + }, + Status: v1beta1.UpdateRunStatus{ + Conditions: tc.conditions, + }, + } + + metric := hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues( + "", + updateRun.Name, + string(tc.state), + string(tc.wantType), + string(tc.wantStatus), + tc.wantReason, + string(tc.wantFailure), + ) + wantTimestamp := float64(tc.wantTime.UnixNano()) / float64(time.Second) + + emitUpdateRunStatusMetric(updateRun) + if got := testutil.ToFloat64(metric); got != wantTimestamp { + t.Fatalf("emitUpdateRunStatusMetric() timestamp = %v, want %v", got, wantTimestamp) + } + + emitUpdateRunStatusMetric(updateRun) + if got := testutil.ToFloat64(metric); got != wantTimestamp { + t.Fatalf("emitUpdateRunStatusMetric() timestamp after unchanged reconciliation = %v, want %v", got, wantTimestamp) + } + }) + } +} + +func newUpdateRunMetricCondition( + conditionType v1beta1.StagedUpdateRunConditionType, + status metav1.ConditionStatus, + reason, message string, + transitionTime time.Time, +) metav1.Condition { + return metav1.Condition{ + Type: string(conditionType), + Status: status, + ObservedGeneration: 1, + LastTransitionTime: metav1.NewTime(transitionTime), + Reason: reason, + Message: message, + } +} diff --git a/pkg/metrics/hub/metrics.go b/pkg/metrics/hub/metrics.go index 82ff70946..f9edef3d5 100644 --- a/pkg/metrics/hub/metrics.go +++ b/pkg/metrics/hub/metrics.go @@ -54,12 +54,12 @@ var ( }, []string{"name", "isCompleted", "isValid"}) // FleetUpdateRunStatusLastTimestampSeconds is a prometheus metric which holds the - // last update timestamp of update run status in seconds. + // last condition transition timestamp of update run status in seconds. // The failure_type label indicates whether a failure is a user_error (customer configuration issue), // internal_error (requires investigation), or none (no failure). FleetUpdateRunStatusLastTimestampSeconds = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Name: "fleet_workload_update_run_status_last_timestamp_seconds", - Help: "Last update timestamp of update run status in seconds", + Help: "Last transition timestamp of update run status in seconds", }, []string{"namespace", "name", "state", "condition", "status", "reason", "failureType"}) // FleetUpdateRunApprovalRequestLatencySeconds tracks how long users take to approve approval requests. From 8a562ff8bcdbc001d5b140625b957af4f90f9d88 Mon Sep 17 00:00:00 2001 From: Britania Rodriguez Reyes <145056127+britaniar@users.noreply.github.com> Date: Thu, 8 Oct 2026 14:14:09 -0700 Subject: [PATCH 13/20] fix: preserve placement status metric timestamps (#936) * fix: preserve placement status metric timestamps Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes * fix: address placement metric review comments Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes * test: stabilize placement metric comparisons Clarify that synthetic Completed derives its timestamp from the final real condition and compare placement metric series by stable labels when timestamps tie. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Britania Rodriguez Reyes --------- Signed-off-by: Britania Rodriguez Reyes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- pkg/controllers/placement/controller.go | 14 +- .../placement/controller_integration_test.go | 35 +++- pkg/controllers/placement/controller_test.go | 171 ++++++++++++++++++ pkg/metrics/hub/metrics.go | 4 +- 4 files changed, 215 insertions(+), 9 deletions(-) diff --git a/pkg/controllers/placement/controller.go b/pkg/controllers/placement/controller.go index 7dcb1b319..041fdf3d7 100644 --- a/pkg/controllers/placement/controller.go +++ b/pkg/controllers/placement/controller.go @@ -868,16 +868,19 @@ func emitPlacementStatusMetric(placementObj fleetv1beta1.PlacementObj) { // Check Placement Scheduled condition. status := "nil" reason := "nil" + metricTimestamp := placementObj.GetCreationTimestamp() scheduledConditionType := getPlacementScheduledConditionType(placementObj) cond := placementObj.GetCondition(scheduledConditionType) if !condition.IsConditionStatusTrue(cond, placementObj.GetGeneration()) { if cond != nil && cond.ObservedGeneration == placementObj.GetGeneration() { status = string(cond.Status) reason = cond.Reason + metricTimestamp = cond.LastTransitionTime } - hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), scheduledConditionType, status, reason).SetToCurrentTime() + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), scheduledConditionType, status, reason).Set(float64(metricTimestamp.UnixNano()) / float64(time.Second)) return } + completionTimestamp := cond.LastTransitionTime // Check placement expected conditions. expectedCondTypes := determineExpectedPlacementAndResourcePlacementStatusCondType(placementObj) @@ -888,13 +891,16 @@ func emitPlacementStatusMetric(placementObj fleetv1beta1.PlacementObj) { if cond != nil && cond.ObservedGeneration == placementObj.GetGeneration() { status = string(cond.Status) reason = cond.Reason + metricTimestamp = cond.LastTransitionTime } - hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), conditionType, status, reason).SetToCurrentTime() + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), conditionType, status, reason).Set(float64(metricTimestamp.UnixNano()) / float64(time.Second)) return } + completionTimestamp = cond.LastTransitionTime } // Emit the "Completed" condition metric to indicate that the placement has completed. - // This condition is used solely for metric reporting purposes. - hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), "Completed", string(metav1.ConditionTrue), "Completed").SetToCurrentTime() + // Completed is a synthetic metric label, not a persisted condition. Its timestamp is the + // transition time of the final real condition that established completion. + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), "Completed", string(metav1.ConditionTrue), "Completed").Set(float64(completionTimestamp.UnixNano()) / float64(time.Second)) } diff --git a/pkg/controllers/placement/controller_integration_test.go b/pkg/controllers/placement/controller_integration_test.go index 1206a4789..65ff4f8ad 100644 --- a/pkg/controllers/placement/controller_integration_test.go +++ b/pkg/controllers/placement/controller_integration_test.go @@ -17,7 +17,9 @@ package placement import ( "fmt" + "sort" "strconv" + "strings" "time" "github.com/google/go-cmp/cmp" @@ -40,7 +42,6 @@ import ( "github.com/kubefleet-dev/kubefleet/pkg/utils/condition" "github.com/kubefleet-dev/kubefleet/pkg/utils/resource" "github.com/kubefleet-dev/kubefleet/test/utils/crpstatussync" - metricsUtils "github.com/kubefleet-dev/kubefleet/test/utils/metrics" ) const ( @@ -2346,6 +2347,34 @@ func checkPlacementStatusMetric(wantMetrics []*prometheusclientmodel.Metric) { placementStatusMetrics = mf.GetMetric() } } - // Sort the emitted metrics for comparison - Expect(cmp.Diff(placementStatusMetrics, wantMetrics, metricsUtils.MetricsCmpOptions...)).Should(BeEmpty(), "Placement status metrics do not match diff (-got, +want):") + // Persisted condition and creation timestamps can legitimately be identical, so compare + // placement time series by their stable label identity rather than scrape order. + sortPlacementStatusMetricsByLabels(placementStatusMetrics) + sortPlacementStatusMetricsByLabels(wantMetrics) + cmpOptions := []cmp.Option{ + cmpopts.SortSlices(func(a, b *prometheusclientmodel.LabelPair) bool { + return a.GetName() < b.GetName() + }), + cmp.Comparer(func(a, b *prometheusclientmodel.Gauge) bool { + return (a.GetValue() > 0) == (b.GetValue() > 0) + }), + cmpopts.IgnoreUnexported(prometheusclientmodel.Metric{}, prometheusclientmodel.LabelPair{}, prometheusclientmodel.Gauge{}), + } + Expect(cmp.Diff(placementStatusMetrics, wantMetrics, cmpOptions...)).Should(BeEmpty(), "Placement status metrics do not match diff (-got, +want):") +} + +func sortPlacementStatusMetricsByLabels(metrics []*prometheusclientmodel.Metric) { + sort.Slice(metrics, func(i, j int) bool { + return placementStatusMetricLabelKey(metrics[i]) < placementStatusMetricLabelKey(metrics[j]) + }) +} + +func placementStatusMetricLabelKey(metric *prometheusclientmodel.Metric) string { + labels := metric.GetLabel() + pairs := make([]string, 0, len(labels)) + for _, label := range labels { + pairs = append(pairs, label.GetName()+"="+label.GetValue()) + } + sort.Strings(pairs) + return strings.Join(pairs, ";") } diff --git a/pkg/controllers/placement/controller_test.go b/pkg/controllers/placement/controller_test.go index 1a14e27f8..8b79f42ba 100644 --- a/pkg/controllers/placement/controller_test.go +++ b/pkg/controllers/placement/controller_test.go @@ -27,6 +27,7 @@ import ( "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" + "github.com/prometheus/client_golang/prometheus/testutil" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -39,6 +40,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" fleetv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" + hubmetrics "github.com/kubefleet-dev/kubefleet/pkg/metrics/hub" "github.com/kubefleet-dev/kubefleet/pkg/utils" "github.com/kubefleet-dev/kubefleet/pkg/utils/condition" "github.com/kubefleet-dev/kubefleet/pkg/utils/controller" @@ -119,6 +121,175 @@ func clusterResourcePlacementForTest() *fleetv1beta1.ClusterResourcePlacement { } } +func TestEmitPlacementStatusMetric(t *testing.T) { + generation := int64(3) + creationTime := metav1.NewTime(time.Date(2026, time.September, 1, 2, 3, 4, 123456789, time.UTC)) + scheduledTime := metav1.NewTime(time.Date(2026, time.September, 2, 3, 4, 5, 234567890, time.UTC)) + rolloutTime := metav1.NewTime(time.Date(2026, time.September, 3, 4, 5, 6, 345678901, time.UTC)) + overriddenTime := metav1.NewTime(time.Date(2026, time.September, 4, 5, 6, 7, 456789012, time.UTC)) + synchronizedTime := metav1.NewTime(time.Date(2026, time.September, 5, 6, 7, 8, 567890123, time.UTC)) + appliedTime := metav1.NewTime(time.Date(2026, time.September, 6, 7, 8, 9, 678901234, time.UTC)) + availableTime := metav1.NewTime(time.Date(2026, time.September, 7, 8, 9, 10, 789012345, time.UTC)) + + newPlacement := func(clusterScoped bool, conditions ...metav1.Condition) fleetv1beta1.PlacementObj { + objectMeta := metav1.ObjectMeta{ + Name: "test-placement", + Generation: generation, + CreationTimestamp: creationTime, + } + if clusterScoped { + return &fleetv1beta1.ClusterResourcePlacement{ + ObjectMeta: objectMeta, + Status: fleetv1beta1.PlacementStatus{ + Conditions: conditions, + }, + } + } + objectMeta.Namespace = "test-namespace" + return &fleetv1beta1.ResourcePlacement{ + ObjectMeta: objectMeta, + Status: fleetv1beta1.PlacementStatus{ + Conditions: conditions, + }, + } + } + newCondition := func(conditionType string, status metav1.ConditionStatus, reason string, transitionTime metav1.Time) metav1.Condition { + return metav1.Condition{ + Type: conditionType, + Status: status, + Reason: reason, + ObservedGeneration: generation, + LastTransitionTime: transitionTime, + } + } + completedConditions := func(clusterScoped bool) []metav1.Condition { + placementObj := newPlacement(clusterScoped) + return []metav1.Condition{ + newCondition(getPlacementScheduledConditionType(placementObj), metav1.ConditionTrue, "Scheduled", scheduledTime), + newCondition(getPlacementConditionType(placementObj, condition.RolloutStartedCondition), metav1.ConditionTrue, "RolloutStarted", rolloutTime), + newCondition(getPlacementConditionType(placementObj, condition.OverriddenCondition), metav1.ConditionTrue, "Overridden", overriddenTime), + newCondition(getPlacementConditionType(placementObj, condition.WorkSynchronizedCondition), metav1.ConditionTrue, "WorkSynchronized", synchronizedTime), + newCondition(getPlacementConditionType(placementObj, condition.AppliedCondition), metav1.ConditionTrue, "Applied", appliedTime), + newCondition(getPlacementConditionType(placementObj, condition.AvailableCondition), metav1.ConditionTrue, "Available", availableTime), + } + } + + tests := []struct { + name string + placementObj fleetv1beta1.PlacementObj + wantConditionType string + wantStatus string + wantReason string + wantTimestamp metav1.Time + }{ + { + name: "cluster-scoped missing scheduled condition uses creation timestamp", + placementObj: newPlacement(true), + wantConditionType: string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), + wantStatus: "nil", + wantReason: "nil", + wantTimestamp: creationTime, + }, + { + name: "namespace-scoped missing scheduled condition uses creation timestamp", + placementObj: newPlacement(false), + wantConditionType: string(fleetv1beta1.ResourcePlacementScheduledConditionType), + wantStatus: "nil", + wantReason: "nil", + wantTimestamp: creationTime, + }, + { + name: "cluster-scoped scheduling failure uses condition transition timestamp", + placementObj: newPlacement(true, + newCondition(string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), metav1.ConditionFalse, "SchedulingFailed", scheduledTime), + ), + wantConditionType: string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), + wantStatus: string(metav1.ConditionFalse), + wantReason: "SchedulingFailed", + wantTimestamp: scheduledTime, + }, + { + name: "namespace-scoped scheduling pending uses condition transition timestamp", + placementObj: newPlacement(false, + newCondition(string(fleetv1beta1.ResourcePlacementScheduledConditionType), metav1.ConditionUnknown, "SchedulingPending", scheduledTime), + ), + wantConditionType: string(fleetv1beta1.ResourcePlacementScheduledConditionType), + wantStatus: string(metav1.ConditionUnknown), + wantReason: "SchedulingPending", + wantTimestamp: scheduledTime, + }, + { + name: "cluster-scoped missing intermediate condition uses creation timestamp", + placementObj: newPlacement(true, + newCondition(string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), metav1.ConditionTrue, "Scheduled", scheduledTime), + ), + wantConditionType: string(fleetv1beta1.ClusterResourcePlacementRolloutStartedConditionType), + wantStatus: "nil", + wantReason: "nil", + wantTimestamp: creationTime, + }, + { + name: "namespace-scoped selects first non-true intermediate condition", + placementObj: newPlacement(false, + newCondition(string(fleetv1beta1.ResourcePlacementScheduledConditionType), metav1.ConditionTrue, "Scheduled", scheduledTime), + newCondition(string(fleetv1beta1.ResourcePlacementRolloutStartedConditionType), metav1.ConditionTrue, "RolloutStarted", rolloutTime), + newCondition(string(fleetv1beta1.ResourcePlacementOverriddenConditionType), metav1.ConditionFalse, "OverrideFailed", overriddenTime), + newCondition(string(fleetv1beta1.ResourcePlacementWorkSynchronizedConditionType), metav1.ConditionFalse, "SynchronizationFailed", synchronizedTime), + ), + wantConditionType: string(fleetv1beta1.ResourcePlacementOverriddenConditionType), + wantStatus: string(metav1.ConditionFalse), + wantReason: "OverrideFailed", + wantTimestamp: overriddenTime, + }, + { + name: "cluster-scoped completed uses final expected condition transition timestamp", + placementObj: newPlacement(true, completedConditions(true)...), + wantConditionType: "Completed", + wantStatus: string(metav1.ConditionTrue), + wantReason: "Completed", + wantTimestamp: availableTime, + }, + { + name: "namespace-scoped completed uses final expected condition transition timestamp", + placementObj: newPlacement(false, completedConditions(false)...), + wantConditionType: "Completed", + wantStatus: string(metav1.ConditionTrue), + wantReason: "Completed", + wantTimestamp: availableTime, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.Reset() + emitPlacementStatusMetric(tc.placementObj) + + gauge := hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues( + tc.placementObj.GetNamespace(), + tc.placementObj.GetName(), + strconv.FormatInt(tc.placementObj.GetGeneration(), 10), + tc.wantConditionType, + tc.wantStatus, + tc.wantReason, + ) + wantTimestamp := float64(tc.wantTimestamp.UnixNano()) / float64(time.Second) + firstTimestamp := testutil.ToFloat64(gauge) + if firstTimestamp != wantTimestamp { + t.Fatalf("emitPlacementStatusMetric() timestamp = %v, want %v", firstTimestamp, wantTimestamp) + } + if gotMetricCount := testutil.CollectAndCount(hubmetrics.FleetPlacementStatusLastTimeStampSeconds); gotMetricCount != 1 { + t.Fatalf("emitPlacementStatusMetric() metric count = %d, want 1", gotMetricCount) + } + + emitPlacementStatusMetric(tc.placementObj) + secondTimestamp := testutil.ToFloat64(gauge) + if secondTimestamp != firstTimestamp { + t.Errorf("emitPlacementStatusMetric() repeated timestamp = %v, want %v", secondTimestamp, firstTimestamp) + } + }) + } +} + func TestGetOrCreateClusterSchedulingPolicySnapshot(t *testing.T) { testPolicy := placementPolicyForTest() testPolicyHash := testPolicy.DeepCopy() diff --git a/pkg/metrics/hub/metrics.go b/pkg/metrics/hub/metrics.go index f9edef3d5..dd18cdc8b 100644 --- a/pkg/metrics/hub/metrics.go +++ b/pkg/metrics/hub/metrics.go @@ -40,10 +40,10 @@ const ( ) var ( - // FleetPlacementStatusLastTimeStampSeconds is a prometheus metric which keeps track of the last placement status. + // FleetPlacementStatusLastTimeStampSeconds is a prometheus metric which keeps track of the last placement status transition. FleetPlacementStatusLastTimeStampSeconds = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Name: "fleet_workload_placement_status_last_timestamp_seconds", - Help: "Last update timestamp of placement status in seconds", + Help: "Last placement status timestamp in seconds; this is the condition transition timestamp when available, or the placement creation timestamp when the selected condition is missing", }, []string{"namespace", "name", "generation", "conditionType", "status", "reason"}) // FleetEvictionStatus is prometheus metrics which holds the From 546bfef4c814b43c458854e3442fd1e3fda1b3fa Mon Sep 17 00:00:00 2001 From: Copilot <198982749+Copilot@users.noreply.github.com> Date: Thu, 8 Oct 2026 15:07:13 -0700 Subject: [PATCH 14/20] feat: support before-stage tasks on the staged update delete stage (#813) * feat: gate delete stage with configured tasks Co-authored-by: ytimocin <5220939+ytimocin@users.noreply.github.com> * fix: avoid reevaluating delete gates after start Co-authored-by: ytimocin <5220939+ytimocin@users.noreply.github.com> * fix: stop safely while delete stage is gated Co-authored-by: ytimocin <5220939+ytimocin@users.noreply.github.com> * chore: complete delete stage task breadcrumb Co-authored-by: ytimocin <5220939+ytimocin@users.noreply.github.com> * feat: support before-stage tasks on the staged update delete stage Let an update strategy hold the delete stage of a staged update run until a wait time passes, an approval request is approved, or both: deleteStage: beforeStageTasks: - type: TimedWait waitTime: 30m - type: Approval The delete stage is configured as a stage, so that its tasks, their statuses and the approval request use the same names as the ones of the update stages. The approval request is named -before-delete-stage. The tasks run concurrently and all of them must complete before any binding is deleted. They only gate the start of the deletion, and they are skipped when there is nothing to delete. An update run that is stopped while it waits for the tasks stops without deleting a binding. A TimedWait task that has elapsed stays elapsed, in the update stages too; stopping and resuming an update run no longer restarts the wait. Signed-off-by: Yetkin Timocin * refactor: align the delete stage tasks with the update stages - Limit the before-stage tasks of the delete stage to a single Approval task, the same as the update stages. A wait before the removal is already possible with a TimedWait task in the after-stage tasks of the last update stage, as the delete stage only starts once those complete. The delete stage now reuses checkBeforeStageTasksStatus, and the TimedWait code path of the delete stage is removed. - Keep a TimedWait after-stage task elapsed once its wait time has passed, as a stop and a resume of the update run move the latest transition of the stage that the wait time is measured from. - Share the code that checks the clusters of the delete stage against the bindings between the execution and the stopping of the delete stage. All the bindings are now checked before one is deleted. - Return early in handleStageApprovalTask instead of nesting. - Build the statuses of the stage tasks with one helper. - Read the conditions of the delete stage with the stage condition types. - Say BeforeStageTaskType, not AfterStageTaskType, in the validation message of an Approval task with a wait time in the before-stage tasks of an update stage. - Fix the kubebuilder marker typo on ResourceSnapshotIndexUsed, and typos in the comments, the messages and the test names. Signed-off-by: Yetkin Timocin --------- Signed-off-by: Yetkin Timocin Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ytimocin <5220939+ytimocin@users.noreply.github.com> Co-authored-by: Yetkin Timocin --- .../2026-08-12-2353-delete-stage-tasks.md | 42 + apis/placement/v1beta1/commons.go | 5 + apis/placement/v1beta1/stageupdate_types.go | 30 +- .../v1beta1/zz_generated.deepcopy.go | 27 + ...etes-fleet.io_clusterstagedupdateruns.yaml | 65 +- ...leet.io_clusterstagedupdatestrategies.yaml | 46 +- ....kubernetes-fleet.io_stagedupdateruns.yaml | 65 +- ...netes-fleet.io_stagedupdatestrategies.yaml | 46 +- pkg/controllers/updaterun/controller.go | 33 +- pkg/controllers/updaterun/controller_test.go | 29 + pkg/controllers/updaterun/execution.go | 318 +++++--- .../updaterun/execution_integration_test.go | 197 ++++- pkg/controllers/updaterun/execution_test.go | 771 +++++++++++++++++- pkg/controllers/updaterun/initialization.go | 45 +- .../initialization_integration_test.go | 22 +- .../updaterun/initialization_test.go | 36 + pkg/controllers/updaterun/stop.go | 58 +- .../updaterun/stop_integration_test.go | 151 ++++ pkg/controllers/updaterun/stop_test.go | 167 +++- pkg/controllers/updaterun/validation.go | 4 +- .../api_validation_integration_test.go | 93 +++ test/e2e/cluster_staged_updaterun_test.go | 138 ++++ test/e2e/staged_updaterun_test.go | 120 +++ test/utils/updaterun/updaterun_status.go | 86 +- 24 files changed, 2302 insertions(+), 292 deletions(-) create mode 100644 .github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md diff --git a/.github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md b/.github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md new file mode 100644 index 000000000..4b624e9b1 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md @@ -0,0 +1,42 @@ +# Support Before-Stage Tasks on the Delete Stage + +## Overview + +Allow staged update strategies to gate the delete stage with an Approval task, the same before-stage task the +update stages support, so that removing the resources from the clusters that left a placement can be held until +a sign-off is given. + +## Plan + +1. Add an optional `deleteStage` to the shared update strategy API, with `beforeStageTasks`. +2. Snapshot the tasks and initialize their statuses as the `beforeStageTaskStatus` of the delete stage. +3. Evaluate the tasks with the same code that evaluates the before-stage tasks of the update stages. +4. Add API validation, unit, integration and E2E coverage for both cluster-scoped and namespaced runs. +5. Regenerate API code and CRDs, then run the tests and the repository quality checks. + +## Decisions + +- The delete stage is configured as a stage (`deleteStage.beforeStageTasks`) rather than with a delete stage + specific task list, so that the API, the status and the approval requests use the same vocabulary as the + update stages, and so that the delete stage can take more configurations later. +- The before-stage tasks of the delete stage follow the update stages: at most one task, and it must be an + Approval. A soak time before the removal is already possible with a TimedWait in the afterStageTasks of the + last update stage, as the delete stage only starts once the last update stage, including its after-stage tasks, + completes. +- The approval request is named `-before-delete-stage` with the generic before-stage name format. + `delete-stage` also is the stage label value of the approval request, as the name of the delete stage, + `kubernetes-fleet.io/deleteStage`, is not a valid label value. An update stage cannot have this name. +- The tasks only gate the start of the deletion and they are skipped if there is no cluster to delete, the same + as an update stage with no clusters. +- An update run that is stopped while it waits for the tasks stops without deleting any binding. + +## Success Criteria + +- [x] No delete stage configuration preserves immediate deletion. +- [x] The approval must be given before any binding is deleted. +- [x] Cluster-scoped and namespaced runs keep the bindings until the tasks pass. +- [x] Generated API and CRD artifacts are current. + +## Follow-ups + +- Document the delete stage configuration in the staged update docs (website repository). diff --git a/apis/placement/v1beta1/commons.go b/apis/placement/v1beta1/commons.go index 6631cb7f0..f4781025f 100644 --- a/apis/placement/v1beta1/commons.go +++ b/apis/placement/v1beta1/commons.go @@ -186,6 +186,11 @@ const ( // UpdateRunDeleteStageName is the name of delete stage in the staged update run. UpdateRunDeleteStageName = FleetPrefix + "deleteStage" + // UpdateRunDeleteStageTaskName refers to the delete stage in the names and the labels of the objects created + // for its tasks, where UpdateRunDeleteStageName is not a valid value. It cannot collide with the name of an + // update stage, as that cannot have a hyphen. + UpdateRunDeleteStageTaskName = "delete-stage" + // IsLatestUpdateRunApprovalLabel indicates if the approval is the latest approval on a staged run. IsLatestUpdateRunApprovalLabel = FleetPrefix + "isLatestUpdateRunApproval" diff --git a/apis/placement/v1beta1/stageupdate_types.go b/apis/placement/v1beta1/stageupdate_types.go index 0054bba0e..576ebcf0b 100644 --- a/apis/placement/v1beta1/stageupdate_types.go +++ b/apis/placement/v1beta1/stageupdate_types.go @@ -262,10 +262,29 @@ func (c *ClusterStagedUpdateStrategy) SetUpdateStrategySpec(spec UpdateStrategyS // UpdateStrategySpec defines the desired state of the StagedUpdateStrategy. type UpdateStrategySpec struct { - // Stage specifies the configuration for each update stage. + // Stages specifies the configuration for each update stage. // +kubebuilder:validation:MaxItems=31 // +kubebuilder:validation:Required Stages []StageConfig `json:"stages"` + + // DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + // complete and removes the resources from the clusters that are no longer selected by the placement. + // If not specified, the delete stage starts as soon as the last update stage completes. + // +kubebuilder:validation:Optional + DeleteStage *DeleteStageConfig `json:"deleteStage,omitempty"` +} + +// DeleteStageConfig describes the delete stage. +type DeleteStageConfig struct { + // The collection of tasks that needs to be completed successfully before the delete stage starts removing + // the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + // before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + // The tasks are skipped if there is no cluster to remove the resources from. + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=1 + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="BeforeStageTaskType is Approval, waitTime is not allowed" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait')",message="BeforeStageTaskType cannot be TimedWait" + BeforeStageTasks []StageTask `json:"beforeStageTasks,omitempty"` } // ClusterStagedUpdateStrategyList contains a list of StagedUpdateStrategy. @@ -330,11 +349,11 @@ type StageConfig struct { // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait' && !has(e.waitTime))",message="AfterStageTaskType is TimedWait, waitTime is required" AfterStageTasks []StageTask `json:"afterStageTasks,omitempty"` - // The collection of tasks that needs to completed successfully by each stage before starting the stage. + // The collection of tasks that needs to be completed successfully by each stage before starting the stage. // Each task is executed in parallel and there cannot be more than one task of the same type. // +kubebuilder:validation:Optional // +kubebuilder:validation:MaxItems=1 - // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="AfterStageTaskType is Approval, waitTime is not allowed" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="BeforeStageTaskType is Approval, waitTime is not allowed" // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait')",message="BeforeStageTaskType cannot be TimedWait" BeforeStageTasks []StageTask `json:"beforeStageTasks,omitempty"` } @@ -373,7 +392,7 @@ type UpdateRunStatus struct { // ResourceSnapshotIndexUsed records the resource snapshot index that the update run is based on. // The index represents the same resource snapshots as specified in the spec field, or the latest. - // +kubbebuilder:validation:Optional + // +kubebuilder:validation:Optional ResourceSnapshotIndexUsed string `json:"resourceSnapshotIndexUsed,omitempty"` // ApplyStrategy is the apply strategy that the stagedUpdateRun is using. @@ -548,6 +567,7 @@ const ( ClusterUpdatingConditionSucceeded ClusterUpdatingStatusConditionType = "Succeeded" ) +// StageTaskStatus defines the status of a pre or post update task of a stage. type StageTaskStatus struct { // The type of the pre or post update task. // +kubebuilder:validation:Enum=TimedWait;Approval @@ -555,7 +575,7 @@ type StageTaskStatus struct { Type StageTaskType `json:"type"` // The name of the approval request object that is created for this stage. - // Only valid if the AfterStageTaskType is Approval. + // Only valid if the type of the task is Approval. // +kubebuilder:validation:Optional ApprovalRequestName string `json:"approvalRequestName,omitempty"` diff --git a/apis/placement/v1beta1/zz_generated.deepcopy.go b/apis/placement/v1beta1/zz_generated.deepcopy.go index b9ff2e710..ae74b5326 100644 --- a/apis/placement/v1beta1/zz_generated.deepcopy.go +++ b/apis/placement/v1beta1/zz_generated.deepcopy.go @@ -1266,6 +1266,28 @@ func (in *ClusterUpdatingStatus) DeepCopy() *ClusterUpdatingStatus { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *DeleteStageConfig) DeepCopyInto(out *DeleteStageConfig) { + *out = *in + if in.BeforeStageTasks != nil { + in, out := &in.BeforeStageTasks, &out.BeforeStageTasks + *out = make([]StageTask, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DeleteStageConfig. +func (in *DeleteStageConfig) DeepCopy() *DeleteStageConfig { + if in == nil { + return nil + } + out := new(DeleteStageConfig) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DeleteStrategy) DeepCopyInto(out *DeleteStrategy) { *out = *in @@ -3044,6 +3066,11 @@ func (in *UpdateStrategySpec) DeepCopyInto(out *UpdateStrategySpec) { (*in)[i].DeepCopyInto(&(*out)[i]) } } + if in.DeleteStage != nil { + in, out := &in.DeleteStage, &out.DeleteStage + *out = new(DeleteStageConfig) + (*in).DeepCopyInto(*out) + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategySpec. diff --git a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml index 27dc987da..22369d2f3 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml @@ -2859,11 +2859,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -2944,11 +2946,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -3242,8 +3246,49 @@ spec: The update run fails to initialize if the strategy fails to produce a valid list of stages where each selected cluster is included in exactly one stage. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that + needs to be completed before starting or moving to the + next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not + allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update + description: Stages specifies the configuration for each update stage. items: description: |- @@ -3287,7 +3332,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -3313,8 +3358,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not - allowed + - message: BeforeStageTaskType is Approval, waitTime is + not allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' @@ -3418,11 +3463,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -3503,11 +3550,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- diff --git a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml index b134a75b5..a67d7ba7d 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml @@ -375,8 +375,47 @@ spec: spec: description: The desired state of ClusterStagedUpdateStrategy. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that needs + to be completed before starting or moving to the next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update stage. + description: Stages specifies the configuration for each update stage. items: description: |- StageConfig describes a single update stage. @@ -416,7 +455,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -441,7 +480,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not allowed + - message: BeforeStageTaskType is Approval, waitTime is not + allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' diff --git a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml index 81b506f4a..4622f51d1 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml @@ -1779,11 +1779,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -1864,11 +1866,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -2162,8 +2166,49 @@ spec: The update run fails to initialize if the strategy fails to produce a valid list of stages where each selected cluster is included in exactly one stage. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that + needs to be completed before starting or moving to the + next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not + allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update + description: Stages specifies the configuration for each update stage. items: description: |- @@ -2207,7 +2252,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -2233,8 +2278,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not - allowed + - message: BeforeStageTaskType is Approval, waitTime is + not allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' @@ -2338,11 +2383,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -2423,11 +2470,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- diff --git a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml index ccfb44e85..f8e8dabd1 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml @@ -235,8 +235,47 @@ spec: spec: description: The desired state of StagedUpdateStrategy. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that needs + to be completed before starting or moving to the next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update stage. + description: Stages specifies the configuration for each update stage. items: description: |- StageConfig describes a single update stage. @@ -276,7 +315,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -301,7 +340,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not allowed + - message: BeforeStageTaskType is Approval, waitTime is not + allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' diff --git a/pkg/controllers/updaterun/controller.go b/pkg/controllers/updaterun/controller.go index a55169bf1..b571951e9 100644 --- a/pkg/controllers/updaterun/controller.go +++ b/pkg/controllers/updaterun/controller.go @@ -492,21 +492,26 @@ func handleApprovalRequestDelete(obj client.Object, q workqueue.TypedRateLimitin }) } +// removeWaitTimeFromUpdateRunStatus removes waitTime from the Approval tasks in the updateRun status. func removeWaitTimeFromUpdateRunStatus(updateRun placementv1beta1.UpdateRunObj) { - // Remove waitTime from the updateRun status for BeforeStageTask and AfterStageTask for type Approval. - updateRunStatus := updateRun.GetUpdateRunStatus() - if updateRunStatus.UpdateStrategySnapshot != nil { - for i := range updateRunStatus.UpdateStrategySnapshot.Stages { - for j := range updateRunStatus.UpdateStrategySnapshot.Stages[i].BeforeStageTasks { - if updateRunStatus.UpdateStrategySnapshot.Stages[i].BeforeStageTasks[j].Type == placementv1beta1.StageTaskTypeApproval { - updateRunStatus.UpdateStrategySnapshot.Stages[i].BeforeStageTasks[j].WaitTime = nil - } - } - for j := range updateRunStatus.UpdateStrategySnapshot.Stages[i].AfterStageTasks { - if updateRunStatus.UpdateStrategySnapshot.Stages[i].AfterStageTasks[j].Type == placementv1beta1.StageTaskTypeApproval { - updateRunStatus.UpdateStrategySnapshot.Stages[i].AfterStageTasks[j].WaitTime = nil - } - } + strategy := updateRun.GetUpdateRunStatus().UpdateStrategySnapshot + if strategy == nil { + return + } + for i := range strategy.Stages { + removeWaitTimeFromApprovalTasks(strategy.Stages[i].BeforeStageTasks) + removeWaitTimeFromApprovalTasks(strategy.Stages[i].AfterStageTasks) + } + if strategy.DeleteStage != nil { + removeWaitTimeFromApprovalTasks(strategy.DeleteStage.BeforeStageTasks) + } +} + +// removeWaitTimeFromApprovalTasks removes waitTime from the tasks of type Approval. +func removeWaitTimeFromApprovalTasks(tasks []placementv1beta1.StageTask) { + for i := range tasks { + if tasks[i].Type == placementv1beta1.StageTaskTypeApproval { + tasks[i].WaitTime = nil } } } diff --git a/pkg/controllers/updaterun/controller_test.go b/pkg/controllers/updaterun/controller_test.go index 5bae2f70a..c2841f59c 100644 --- a/pkg/controllers/updaterun/controller_test.go +++ b/pkg/controllers/updaterun/controller_test.go @@ -986,6 +986,35 @@ func TestRemoveWaitTimeFromUpdateRunStatus(t *testing.T) { }, }, }, + "should remove waitTime from the Approval task of the delete stage BeforeStageTasks": { + inputUpdateRun: &placementv1beta1.ClusterStagedUpdateRun{ + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + WaitTime: &waitTime, + }, + }, + }, + }, + }, + }, + wantUpdateRun: &placementv1beta1.ClusterStagedUpdateRun{ + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + }, + }, + }, + }, + }, "should handle multiple stages": { inputUpdateRun: &placementv1beta1.ClusterStagedUpdateRun{ Status: placementv1beta1.UpdateRunStatus{ diff --git a/pkg/controllers/updaterun/execution.go b/pkg/controllers/updaterun/execution.go index b8c392806..996213c92 100644 --- a/pkg/controllers/updaterun/execution.go +++ b/pkg/controllers/updaterun/execution.go @@ -83,7 +83,8 @@ func (r *Reconciler) execute( // No need to wait to get to the next stage. return false, 0, nil } - approved, err := r.checkBeforeStageTasksStatus(ctx, updatingStageIndex, updateRun) + beforeStageTasks := updateRunStatus.UpdateStrategySnapshot.Stages[updatingStageIndex].BeforeStageTasks + approved, err := r.checkBeforeStageTasksStatus(ctx, updatingStageStatus, beforeStageTasks, updateRun) if err != nil { return false, 0, err } @@ -101,26 +102,28 @@ func (r *Reconciler) execute( return false, waitTime, err } // All the stages have finished, now start the delete stage. - finished, err = r.executeDeleteStage(ctx, updateRun, toBeDeletedBindings) - return finished, clusterUpdatingWaitTime, err + return r.executeDeleteStage(ctx, updateRun, toBeDeletedBindings) } -// checkBeforeStageTasksStatus checks if the before stage tasks have finished. +// checkBeforeStageTasksStatus checks if the before stage tasks of a stage, an update stage or the delete stage, +// have finished. // It returns if the before stage tasks have finished or error if the before stage tasks failed. -func (r *Reconciler) checkBeforeStageTasksStatus(ctx context.Context, updatingStageIndex int, updateRun placementv1beta1.UpdateRunObj) (bool, error) { +func (r *Reconciler) checkBeforeStageTasksStatus( + ctx context.Context, + stageStatus *placementv1beta1.StageUpdatingStatus, + tasks []placementv1beta1.StageTask, + updateRun placementv1beta1.UpdateRunObj, +) (bool, error) { updateRunRef := klog.KObj(updateRun) - updateRunStatus := updateRun.GetUpdateRunStatus() - updatingStage := &updateRunStatus.UpdateStrategySnapshot.Stages[updatingStageIndex] - if updatingStage.BeforeStageTasks == nil { - klog.V(2).InfoS("There is no before stage task for this stage", "stage", updatingStage.Name, "updateRun", updateRunRef) + if len(tasks) == 0 { + klog.V(2).InfoS("There is no before stage task for this stage", "stage", stageStatus.StageName, "updateRun", updateRunRef) return true, nil } - updatingStageStatus := &updateRunStatus.StagesStatus[updatingStageIndex] - for i, task := range updatingStage.BeforeStageTasks { + for i, task := range tasks { switch task.Type { case placementv1beta1.StageTaskTypeApproval: - approved, err := r.handleStageApprovalTask(ctx, &updatingStageStatus.BeforeStageTaskStatus[i], updatingStage, updateRun, placementv1beta1.BeforeStageTaskLabelValue) + approved, err := r.handleStageApprovalTask(ctx, &stageStatus.BeforeStageTaskStatus[i], stageStatus.StageName, updateRun, placementv1beta1.BeforeStageTaskLabelValue) if err != nil { return false, err } @@ -128,7 +131,7 @@ func (r *Reconciler) checkBeforeStageTasksStatus(ctx context.Context, updatingSt default: // Approval is the only supported before stage task. unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("found unsupported task type in before stage tasks: %s", task.Type)) - klog.ErrorS(unexpectedErr, "Task type is not supported in before stage tasks", "stage", updatingStage.Name, "updateRun", updateRunRef, "taskType", task.Type) + klog.ErrorS(unexpectedErr, "Task type is not supported in before stage tasks", "stage", stageStatus.StageName, "updateRun", updateRunRef, "taskType", task.Type) return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) } } @@ -272,7 +275,7 @@ func (r *Reconciler) executeUpdatingStage( // If cluster update has been running for more than "updateRunStuckThreshold", mark the update run as stuck. timeElapsed := time.Since(clusterStartedCond.LastTransitionTime.Time) if timeElapsed > updateRunStuckThreshold { - klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "time elapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) + klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "timeElapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) stuckClusterNames = append(stuckClusterNames, clusterStatus.ClusterName) } } @@ -335,80 +338,141 @@ func (r *Reconciler) handleStageCompletion( return waitTime, nil } -// executeDeleteStage executes the delete stage by deleting the bindings. +// executeDeleteStage executes the delete stage by deleting the bindings, once the before stage tasks have completed. +// It returns a boolean indicating if the delete stage is completed, +// the time to wait before rechecking the delete stage, and any error encountered. func (r *Reconciler) executeDeleteStage( ctx context.Context, updateRun placementv1beta1.UpdateRunObj, toBeDeletedBindings []placementv1beta1.BindingObj, -) (bool, error) { +) (bool, time.Duration, error) { updateRunRef := klog.KObj(updateRun) updateRunStatus := updateRun.GetUpdateRunStatus() existingDeleteStageStatus := updateRunStatus.DeletionStageStatus - existingDeleteStageClusterMap := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(existingDeleteStageStatus.Clusters)) - for i := range existingDeleteStageStatus.Clusters { - existingDeleteStageClusterMap[existingDeleteStageStatus.Clusters[i].ClusterName] = &existingDeleteStageStatus.Clusters[i] + approved, err := r.checkBeforeDeleteStageTasksStatus(ctx, updateRun, toBeDeletedBindings) + if err != nil { + return false, 0, err + } + if !approved { + return false, stageUpdatingWaitTime, nil } // Mark the delete stage as started in case it's not. - markStageUpdatingProgressStarted(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) + markStageUpdatingProgressStarted(existingDeleteStageStatus, updateRun.GetGeneration()) + toBeStartedClusters, err := syncDeleteStageClusterStatuses(updateRun, toBeDeletedBindings) + if err != nil { + return false, 0, err + } + for _, binding := range toBeDeletedBindings { + curCluster, toBeStarted := toBeStartedClusters[binding.GetBindingSpec().TargetCluster] + if !toBeStarted { + continue + } + if err := r.Client.Delete(ctx, binding); err != nil { + klog.ErrorS(err, "Failed to delete a binding in the update run", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) + return false, 0, controller.NewAPIServerError(false, err) + } + klog.V(2).InfoS("Deleted a binding pointing to a to be deleted cluster", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) + markClusterUpdatingStarted(curCluster, updateRun.GetGeneration()) + } + klog.InfoS("The delete stage is progressing", "numberOfDeletingClusters", len(toBeDeletedBindings), "updateRun", updateRunRef) + if len(toBeDeletedBindings) == 0 { + markStageUpdatingSucceeded(existingDeleteStageStatus, updateRun.GetGeneration()) + } + return len(toBeDeletedBindings) == 0, clusterUpdatingWaitTime, nil +} + +// syncDeleteStageClusterStatuses checks the clusters in the delete stage against the bindings that are still to be +// deleted and marks the clusters with no binding left as succeeded in memory. +// It returns the clusters with a binding whose deletion has not started, by their names. +func syncDeleteStageClusterStatuses( + updateRun placementv1beta1.UpdateRunObj, + toBeDeletedBindings []placementv1beta1.BindingObj, +) (map[string]*placementv1beta1.ClusterUpdatingStatus, error) { + updateRunRef := klog.KObj(updateRun) + deleteStageStatus := updateRun.GetUpdateRunStatus().DeletionStageStatus + deletedClusters := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(deleteStageStatus.Clusters)) + for i := range deleteStageStatus.Clusters { + deletedClusters[deleteStageStatus.Clusters[i].ClusterName] = &deleteStageStatus.Clusters[i] + } + toBeStartedClusters := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(toBeDeletedBindings)) for _, binding := range toBeDeletedBindings { bindingSpec := binding.GetBindingSpec() - curCluster, exist := existingDeleteStageClusterMap[bindingSpec.TargetCluster] + curCluster, exist := deletedClusters[bindingSpec.TargetCluster] if !exist { // This is unexpected because we already checked in validation. - missingErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the to be deleted cluster `%s` is not in the deleting stage during execution", bindingSpec.TargetCluster)) + missingErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the to be deleted cluster `%s` is not in the deleting stage", bindingSpec.TargetCluster)) klog.ErrorS(missingErr, "The cluster in the deleting stage does not include all the to be deleted binding", "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, missingErr.Error()) + return nil, fmt.Errorf("%w: %s", errStagedUpdatedAborted, missingErr.Error()) } - // In validation, we already check the binding must exist in the status. - delete(existingDeleteStageClusterMap, bindingSpec.TargetCluster) + // The binding is still there, so the cluster is not deleted. + delete(deletedClusters, bindingSpec.TargetCluster) // Make sure the cluster is not marked as deleted as the binding is still there. if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionSucceeded)), updateRun.GetGeneration()) { unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the deleted cluster `%s` in the deleting stage still has a binding", bindingSpec.TargetCluster)) klog.ErrorS(unexpectedErr, "The cluster in the deleting stage is not removed yet but marked as deleted", "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) + return nil, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) } - if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { - // The cluster status is marked as being deleted. - if binding.GetDeletionTimestamp().IsZero() { - // The cluster is marked as deleting but the binding is not deleting. - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the cluster `%s` in the deleting stage is marked as deleting but its corresponding binding is not deleting", curCluster.ClusterName)) - klog.ErrorS(unexpectedErr, "The binding should be deleting before we mark a cluster deleting", "clusterStatus", curCluster, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } + if !condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { + // The cluster status is not deleting yet. + toBeStartedClusters[bindingSpec.TargetCluster] = curCluster continue } - // The cluster status is not deleting yet - if err := r.Client.Delete(ctx, binding); err != nil { - klog.ErrorS(err, "Failed to delete a binding in the update run", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - return false, controller.NewAPIServerError(false, err) + // The cluster status is marked as being deleted. + if binding.GetDeletionTimestamp().IsZero() { + // The cluster is marked as deleting but the binding is not deleting. + unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the cluster `%s` in the deleting stage is marked as deleting but its corresponding binding is not deleting", curCluster.ClusterName)) + klog.ErrorS(unexpectedErr, "The binding should be deleting before we mark a cluster deleting", "clusterStatus", curCluster, "updateRun", updateRunRef) + return nil, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) } - klog.V(2).InfoS("Deleted a binding pointing to a to be deleted cluster", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - markClusterUpdatingStarted(curCluster, updateRun.GetGeneration()) } // The rest of the clusters in the stage are not in the toBeDeletedBindings so it should be marked as delete succeeded. - for _, clusterStatus := range existingDeleteStageClusterMap { + for _, clusterStatus := range deletedClusters { // Make sure the cluster is marked as deleted. if !condition.IsConditionStatusTrue(meta.FindStatusCondition(clusterStatus.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { markClusterUpdatingStarted(clusterStatus, updateRun.GetGeneration()) } markClusterUpdatingSucceeded(clusterStatus, updateRun.GetGeneration()) } - klog.InfoS("The delete stage is progressing", "numberOfDeletingClusters", len(toBeDeletedBindings), "updateRun", updateRunRef) - if len(toBeDeletedBindings) == 0 { - markStageUpdatingSucceeded(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) + return toBeStartedClusters, nil +} + +// checkBeforeDeleteStageTasksStatus checks if the before stage tasks of the delete stage have finished. +// It returns if the delete stage can start deleting the bindings or error if the before stage tasks failed. +func (r *Reconciler) checkBeforeDeleteStageTasksStatus( + ctx context.Context, + updateRun placementv1beta1.UpdateRunObj, + toBeDeletedBindings []placementv1beta1.BindingObj, +) (bool, error) { + updateRunStatus := updateRun.GetUpdateRunStatus() + deleteStage := updateRunStatus.UpdateStrategySnapshot.DeleteStage + deleteStageStatus := updateRunStatus.DeletionStageStatus + // The tasks only gate the start of the deletion. Same as the update stages with no clusters, the delete stage + // skips its tasks if there is nothing to delete. + if deleteStage == nil || len(deleteStage.BeforeStageTasks) == 0 || len(toBeDeletedBindings) == 0 || deleteStageStatus.StartTime != nil { + return true, nil + } + approved, err := r.checkBeforeStageTasksStatus(ctx, deleteStageStatus, deleteStage.BeforeStageTasks, updateRun) + if err != nil { + return false, err + } + if !approved { + markStageUpdatingWaiting(deleteStageStatus, updateRun.GetGeneration(), "Not all before-stage tasks are completed, waiting for approval") + markUpdateRunWaiting(updateRun, fmt.Sprintf(condition.UpdateRunWaitingMessageFmt, "before-stage", deleteStageStatus.StageName)) + return false, nil } - return len(toBeDeletedBindings) == 0, nil + markUpdateRunProgressing(updateRun) + return true, nil } // checkAfterStageTasksStatus checks if the after stage tasks have finished. // It returns if the after stage tasks have finished or error if the after stage tasks failed. -// It also returns the time to wait before rechecking the wait type of task. It turns -1 if the task is not a wait type. +// It also returns the time to wait before rechecking the wait type of task. It returns -1 if the task is not a wait type. func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingStageIndex int, updateRun placementv1beta1.UpdateRunObj) (bool, time.Duration, error) { updateRunRef := klog.KObj(updateRun) updateRunStatus := updateRun.GetUpdateRunStatus() updatingStageStatus := &updateRunStatus.StagesStatus[updatingStageIndex] updatingStage := &updateRunStatus.UpdateStrategySnapshot.Stages[updatingStageIndex] - if updatingStage.AfterStageTasks == nil { + if len(updatingStage.AfterStageTasks) == 0 { klog.V(2).InfoS("There is no after stage task for this stage", "stage", updatingStage.Name, "updateRun", updateRunRef) return true, 0, nil } @@ -417,6 +481,11 @@ func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingSta for i, task := range updatingStage.AfterStageTasks { switch task.Type { case placementv1beta1.StageTaskTypeTimedWait: + // The wait time is measured from the latest transition of the stage, which a stop and a resume of the + // update run move; a wait that has elapsed once stays elapsed. + if meta.IsStatusConditionTrue(updatingStageStatus.AfterStageTaskStatus[i].Conditions, string(placementv1beta1.StageTaskConditionWaitTimeElapsed)) { + continue + } waitStartTime := meta.FindStatusCondition(updatingStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionProgressing)).LastTransitionTime.Time // Check if the wait time has passed. waitTime := time.Until(waitStartTime.Add(task.WaitTime.Duration)) @@ -429,7 +498,7 @@ func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingSta klog.V(2).InfoS("The after stage wait task has completed", "stage", updatingStage.Name, "updateRun", updateRunRef) } case placementv1beta1.StageTaskTypeApproval: - approved, err := r.handleStageApprovalTask(ctx, &updatingStageStatus.AfterStageTaskStatus[i], updatingStage, updateRun, placementv1beta1.AfterStageTaskLabelValue) + approved, err := r.handleStageApprovalTask(ctx, &updatingStageStatus.AfterStageTaskStatus[i], updatingStage.Name, updateRun, placementv1beta1.AfterStageTaskLabelValue) if err != nil { return false, -1, err } @@ -449,7 +518,7 @@ func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingSta func (r *Reconciler) handleStageApprovalTask( ctx context.Context, stageTaskStatus *placementv1beta1.StageTaskStatus, - updatingStage *placementv1beta1.StageConfig, + stageName string, updateRun placementv1beta1.UpdateRunObj, stageTaskType string, ) (bool, error) { @@ -462,54 +531,54 @@ func (r *Reconciler) handleStageApprovalTask( } // Check if the approval request has been created. - approvalRequest := buildApprovalRequestObject(types.NamespacedName{Name: stageTaskStatus.ApprovalRequestName, Namespace: updateRun.GetNamespace()}, updatingStage.Name, updateRun.GetName(), stageTaskType) + approvalRequest := buildApprovalRequestObject(types.NamespacedName{Name: stageTaskStatus.ApprovalRequestName, Namespace: updateRun.GetNamespace()}, stageName, updateRun.GetName(), stageTaskType) requestRef := klog.KObj(approvalRequest) - if err := r.Client.Create(ctx, approvalRequest); err != nil { - if apierrors.IsAlreadyExists(err) { - // The approval task already exists. - markStageTaskRequestCreated(stageTaskStatus, updateRun.GetGeneration()) - if err = r.Client.Get(ctx, client.ObjectKeyFromObject(approvalRequest), approvalRequest); err != nil { - klog.ErrorS(err, "Failed to get the already existing approval request", "approvalRequest", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, controller.NewAPIServerError(true, err) - } - approvalRequestSpec := approvalRequest.GetApprovalRequestSpec() - if approvalRequestSpec.TargetStage != updatingStage.Name || approvalRequestSpec.TargetUpdateRun != updateRun.GetName() { - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the approval request task `%s/%s` is targeting update run `%s/%s` and stage `%s`, want target update run `%s/%s and stage `%s`", approvalRequest.GetNamespace(), approvalRequest.GetName(), approvalRequest.GetNamespace(), approvalRequestSpec.TargetUpdateRun, approvalRequestSpec.TargetStage, approvalRequest.GetNamespace(), updateRun.GetName(), updatingStage.Name)) - klog.ErrorS(unexpectedErr, "Found an approval request targeting wrong stage", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } - approvalRequestStatus := approvalRequest.GetApprovalRequestStatus() - approvalAccepted := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApprovalAccepted)), approvalRequest.GetGeneration()) - approved := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApproved)), approvalRequest.GetGeneration()) - if !approvalAccepted && !approved { - klog.V(2).InfoS("The approval request has not been approved yet", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, nil - } - if approved { - klog.V(2).InfoS("The approval request has been approved", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - if !approvalAccepted { - if err = r.updateApprovalRequestAccepted(ctx, approvalRequest); err != nil { - klog.ErrorS(err, "Failed to accept the approved approval request", "approvalRequest", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - // retriable err - return false, err - } - } - } else { - // Approved state should not change once the approval is accepted. - klog.V(2).InfoS("The approval request has been approval-accepted, ignoring changing back to unapproved", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - } - markStageTaskRequestApproved(stageTaskStatus, updateRun, stageTaskType) - } else { - // retriable error - klog.ErrorS(err, "Failed to create the approval request", "approvalRequest", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, controller.NewAPIServerError(false, err) - } - } else { + err := r.Client.Create(ctx, approvalRequest) + if err == nil { // The approval request has been created for the first time. - klog.V(2).InfoS("The approval request has been created", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) + klog.V(2).InfoS("The approval request has been created", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) markStageTaskRequestCreated(stageTaskStatus, updateRun.GetGeneration()) return false, nil } + if !apierrors.IsAlreadyExists(err) { + // retriable error + klog.ErrorS(err, "Failed to create the approval request", "approvalRequest", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, controller.NewAPIServerError(false, err) + } + + // The approval task already exists. + markStageTaskRequestCreated(stageTaskStatus, updateRun.GetGeneration()) + if err = r.Client.Get(ctx, client.ObjectKeyFromObject(approvalRequest), approvalRequest); err != nil { + klog.ErrorS(err, "Failed to get the already existing approval request", "approvalRequest", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, controller.NewAPIServerError(true, err) + } + approvalRequestSpec := approvalRequest.GetApprovalRequestSpec() + if approvalRequestSpec.TargetStage != stageName || approvalRequestSpec.TargetUpdateRun != updateRun.GetName() { + unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the approval request task `%s/%s` is targeting update run `%s/%s` and stage `%s`, want target update run `%s/%s` and stage `%s`", approvalRequest.GetNamespace(), approvalRequest.GetName(), approvalRequest.GetNamespace(), approvalRequestSpec.TargetUpdateRun, approvalRequestSpec.TargetStage, approvalRequest.GetNamespace(), updateRun.GetName(), stageName)) + klog.ErrorS(unexpectedErr, "Found an approval request targeting wrong stage", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) + } + approvalRequestStatus := approvalRequest.GetApprovalRequestStatus() + approvalAccepted := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApprovalAccepted)), approvalRequest.GetGeneration()) + approved := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApproved)), approvalRequest.GetGeneration()) + switch { + case !approvalAccepted && !approved: + klog.V(2).InfoS("The approval request has not been approved yet", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, nil + case !approved: + // Approved state should not change once the approval is accepted. + klog.V(2).InfoS("The approval request has been approval-accepted, ignoring changing back to unapproved", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + case !approvalAccepted: + klog.V(2).InfoS("The approval request has been approved", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + if err = r.updateApprovalRequestAccepted(ctx, approvalRequest); err != nil { + klog.ErrorS(err, "Failed to accept the approved approval request", "approvalRequest", requestRef, "stage", stageName, "updateRun", updateRunRef) + // retriable err + return false, err + } + default: + klog.V(2).InfoS("The approval request has been approved", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + } + markStageTaskRequestApproved(stageTaskStatus, updateRun, stageTaskType) return true, nil } @@ -633,42 +702,29 @@ func checkClusterUpdateResult( // buildApprovalRequestObject creates an approval request object for before-stage or after-stage tasks. // It returns a ClusterApprovalRequest if namespace is empty, otherwise returns an ApprovalRequest. func buildApprovalRequestObject(namespacedName types.NamespacedName, stageName, updateRunName, stageTaskType string) placementv1beta1.ApprovalRequestObj { - var approvalRequest placementv1beta1.ApprovalRequestObj + stageLabelValue := stageName + if stageName == placementv1beta1.UpdateRunDeleteStageName { + // The name of the delete stage is not a valid label value; the spec still targets the stage by its name. + stageLabelValue = placementv1beta1.UpdateRunDeleteStageTaskName + } + objectMeta := metav1.ObjectMeta{ + Name: namespacedName.Name, + Namespace: namespacedName.Namespace, + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: stageLabelValue, + placementv1beta1.TargetUpdateRunLabel: updateRunName, + placementv1beta1.TaskTypeLabel: stageTaskType, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + } + spec := placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRunName, + TargetStage: stageName, + } if namespacedName.Namespace == "" { - approvalRequest = &placementv1beta1.ClusterApprovalRequest{ - ObjectMeta: metav1.ObjectMeta{ - Name: namespacedName.Name, - Labels: map[string]string{ - placementv1beta1.TargetUpdatingStageNameLabel: stageName, - placementv1beta1.TargetUpdateRunLabel: updateRunName, - placementv1beta1.TaskTypeLabel: stageTaskType, - placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", - }, - }, - Spec: placementv1beta1.ApprovalRequestSpec{ - TargetUpdateRun: updateRunName, - TargetStage: stageName, - }, - } - } else { - approvalRequest = &placementv1beta1.ApprovalRequest{ - ObjectMeta: metav1.ObjectMeta{ - Name: namespacedName.Name, - Namespace: namespacedName.Namespace, - Labels: map[string]string{ - placementv1beta1.TargetUpdatingStageNameLabel: stageName, - placementv1beta1.TargetUpdateRunLabel: updateRunName, - placementv1beta1.TaskTypeLabel: stageTaskType, - placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", - }, - }, - Spec: placementv1beta1.ApprovalRequestSpec{ - TargetUpdateRun: updateRunName, - TargetStage: stageName, - }, - } + return &placementv1beta1.ClusterApprovalRequest{ObjectMeta: objectMeta, Spec: spec} } - return approvalRequest + return &placementv1beta1.ApprovalRequest{ObjectMeta: objectMeta, Spec: spec} } // markUpdateRunProgressing marks the update run as progressing in memory. @@ -707,19 +763,13 @@ func markUpdateRunStuck(updateRun placementv1beta1.UpdateRunObj, stageName strin }) } +// generateStuckClustersString lists up to 3 of the stuck clusters. func generateStuckClustersString(stuckClusterNames []string) string { - if len(stuckClusterNames) == 0 { - return "" - } if len(stuckClusterNames) <= 3 { return strings.Join(stuckClusterNames, ", ") } // Print first 3 if more than 3 stuck clusters. - clustersToShow := stuckClusterNames - if len(stuckClusterNames) > 3 { - clustersToShow = stuckClusterNames[:3] - } - return strings.Join(clustersToShow, ", ") + "..." + return strings.Join(stuckClusterNames[:3], ", ") + "..." } // markUpdateRunWaiting marks the updateRun as waiting in memory. diff --git a/pkg/controllers/updaterun/execution_integration_test.go b/pkg/controllers/updaterun/execution_integration_test.go index 12ef4bbb1..87332447a 100644 --- a/pkg/controllers/updaterun/execution_integration_test.go +++ b/pkg/controllers/updaterun/execution_integration_test.go @@ -880,6 +880,199 @@ var _ = Describe("UpdateRun execution tests - double stages", func() { }) }) + + Context("Cluster staged update run should wait for the before-stage tasks of the delete stage", Ordered, func() { + var wantApprovalRequest *placementv1beta1.ClusterApprovalRequest + + // Wait for a few rounds of reconciliation, which is every stageUpdatingWaitTime while waiting for the tasks. + validateToBeDeletedBindingsExistConsistently := func() { + Consistently(func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding); err != nil { + return fmt.Errorf("failed to get binding %s: %w", resourceBindings[i].Name, err) + } + if !binding.DeletionTimestamp.IsZero() { + return fmt.Errorf("binding %s is being deleted", binding.Name) + } + } + return nil + }, 3*stageUpdatingWaitTime, interval).Should(Succeed(), "failed to validate the to-be-deleted bindings are kept") + } + + BeforeAll(func() { + By("Reassigning the strategy to have 1 stage selecting all clusters and the delete stage with a before-stage approval task") + sortingKey := "index" + updateStrategy.Spec.Stages = []placementv1beta1.StageConfig{ + { + Name: "stage1", + LabelSelector: &metav1.LabelSelector{}, // Empty label selector selects all clusters. + SortingLabelKey: &sortingKey, + }, + } + updateStrategy.Spec.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + } + Expect(k8sClient.Update(ctx, updateStrategy)).To(Succeed()) + + By("Creating a new clusterStagedUpdateRun") + Expect(k8sClient.Create(ctx, updateRun)).To(Succeed()) + + By("Validating the initialization succeeded and the execution has started") + initialized := generateInitializedStatus( + crp, updateRun, testResourceSnapshotIndex, policySnapshot, updateStrategy, 10, + generateTenClusterSingleStageStatus(clusterResourceOverride)[:1], generateTenClusterDeletionStageStatus(), + ) + wantStatus = generateExecutionStartedStatus(updateRun, initialized) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + // Build wantStatus based on the current updateRun status after initialization. + wantStatus = updateRun.Status.DeepCopy() + }) + + It("Should complete the 1st stage and wait for the before-stage tasks of the delete stage", func() { + By("Marking all bindings in the 1st stage as available and validating all clusters have completed") + for i := 0; i < numTargetClusters; i++ { + binding := resourceBindings[numTargetClusters-1-i] + validateBindingState(ctx, binding, resourceSnapshot.Name, updateRun, &updateRun.Status.StagesStatus[0].Clusters[i]) + meta.SetStatusCondition(&binding.Status.Conditions, generateTrueCondition(binding, placementv1beta1.ResourceBindingAvailable)) + Expect(k8sClient.Status().Update(ctx, binding)).Should(Succeed(), "failed to update the binding status") + + meta.SetStatusCondition(&wantStatus.StagesStatus[0].Clusters[i].Conditions, generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionStarted)) + meta.SetStatusCondition(&wantStatus.StagesStatus[0].Clusters[i].Conditions, generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionSucceeded)) + } + + By("Validating the 1st stage has completed and the delete stage is waiting for the approval") + wantStatus.StagesStatus[0].Conditions[0] = generateFalseConditionWithReason(updateRun, placementv1beta1.StageUpdatingConditionProgressing, condition.StageUpdatingSucceededReason) + wantStatus.StagesStatus[0].Conditions = append(wantStatus.StagesStatus[0].Conditions, generateTrueCondition(updateRun, placementv1beta1.StageUpdatingConditionSucceeded)) + wantStatus.DeletionStageStatus.Conditions = append(wantStatus.DeletionStageStatus.Conditions, generateFalseCondition(updateRun, placementv1beta1.StageUpdatingConditionProgressing)) + wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions = append(wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions, + generateTrueCondition(updateRun, placementv1beta1.StageTaskConditionApprovalRequestCreated)) + meta.SetStatusCondition(&wantStatus.Conditions, generateFalseCondition(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing)) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + By("Validating the delete stage has not started") + Expect(updateRun.Status.DeletionStageStatus.StartTime).Should(BeNil()) + + By("Validating the approvalRequest has been created") + wantApprovalRequest = &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.Name, placementv1beta1.UpdateRunDeleteStageTaskName), + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: placementv1beta1.UpdateRunDeleteStageTaskName, + placementv1beta1.TargetUpdateRunLabel: updateRun.Name, + placementv1beta1.TaskTypeLabel: placementv1beta1.BeforeStageTaskLabelValue, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + }, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRun.Name, + TargetStage: updateRun.Status.DeletionStageStatus.StageName, + }, + } + Expect(updateRun.Status.DeletionStageStatus.BeforeStageTaskStatus[0].ApprovalRequestName).Should(Equal(wantApprovalRequest.Name)) + validateApprovalRequestCreated(wantApprovalRequest) + + By("Validating the to-be-deleted bindings are not deleted") + validateToBeDeletedBindingsExistConsistently() + + By("Checking update run status metrics are emitted") + validateUpdateRunMetricsEmitted(generateProgressingMetric(placementv1beta1.StateRun, updateRun), generateWaitingMetric(placementv1beta1.StateRun, updateRun)) + }) + + It("Should recreate the approvalRequest of the delete stage if it is deleted", func() { + By("Deleting the approvalRequest") + approvalRequest := &placementv1beta1.ClusterApprovalRequest{} + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantApprovalRequest.Name}, approvalRequest)).Should(Succeed()) + oldUID := approvalRequest.UID + Expect(k8sClient.Delete(ctx, approvalRequest)).Should(Succeed()) + + By("Validating the approvalRequest has been recreated") + Eventually(func() error { + if err := k8sClient.Get(ctx, types.NamespacedName{Name: wantApprovalRequest.Name}, approvalRequest); err != nil { + return err + } + if approvalRequest.UID == oldUID { + return fmt.Errorf("approvalRequest %s is not deleted yet", approvalRequest.Name) + } + return nil + }, timeout, interval).Should(Succeed(), "failed to validate the approvalRequest is recreated") + validateApprovalRequestCreated(wantApprovalRequest) + + By("Validating the to-be-deleted bindings are not deleted") + validateToBeDeletedBindingsExistConsistently() + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + }) + + It("Should stop the update run without deleting the bindings when state is Stop", func() { + By("Updating updateRun state to Stop") + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateStop) + // Update the test's want status to match the new generation. + updateAllStatusConditionsGeneration(wantStatus, updateRun.Generation) + + By("Validating the update run has stopped") + meta.SetStatusCondition(&wantStatus.DeletionStageStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StageUpdatingConditionProgressing, condition.StageUpdatingStoppedReason)) + meta.SetStatusCondition(&wantStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStoppedReason)) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + By("Approving the approvalRequest") + approveClusterApprovalRequest(ctx, wantApprovalRequest.Name) + + By("Validating the to-be-deleted bindings are not deleted") + validateToBeDeletedBindingsExistConsistently() + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + }) + + It("Should delete all the clusterResourceBindings and complete the update run when state is Run", func() { + By("Updating updateRun state to Run") + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateRun) + // Update the test's want status to match the new generation. + updateAllStatusConditionsGeneration(wantStatus, updateRun.Generation) + + By("Validating the approvalRequest has ApprovalAccepted status") + validateApprovalRequestAccepted(ctx, wantApprovalRequest.Name) + + By("Validating the to-be-deleted bindings are all deleted") + Eventually(func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding) + if err == nil { + return fmt.Errorf("binding %s is not deleted", binding.Name) + } + if !apierrors.IsNotFound(err) { + return fmt.Errorf("get binding %s does not return a not-found error: %w", binding.Name, err) + } + } + return nil + }, timeout, interval).Should(Succeed(), "failed to validate the deletion of the to-be-deleted bindings") + + By("Validating the delete stage and the clusterStagedUpdateRun has completed") + wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions = append(wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions, + generateTrueCondition(updateRun, placementv1beta1.StageTaskConditionApprovalRequestApproved)) + for i := range wantStatus.DeletionStageStatus.Clusters { + wantStatus.DeletionStageStatus.Clusters[i].Conditions = append(wantStatus.DeletionStageStatus.Clusters[i].Conditions, + generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionStarted), + generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionSucceeded)) + } + meta.SetStatusCondition(&wantStatus.DeletionStageStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StageUpdatingConditionProgressing, condition.StageUpdatingSucceededReason)) + wantStatus.DeletionStageStatus.Conditions = append(wantStatus.DeletionStageStatus.Conditions, generateTrueCondition(updateRun, placementv1beta1.StageUpdatingConditionSucceeded)) + meta.SetStatusCondition(&wantStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunSucceededReason)) + wantStatus.Conditions = append(wantStatus.Conditions, generateTrueCondition(updateRun, placementv1beta1.StagedUpdateRunConditionSucceeded)) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + By("Validating the delete stage has startTime and endTime set") + Expect(updateRun.Status.DeletionStageStatus.StartTime).ShouldNot(BeNil()) + Expect(updateRun.Status.DeletionStageStatus.EndTime).ShouldNot(BeNil()) + + By("Checking update run metrics are emitted") + validateUpdateRunApprovalStageTaskMetric(generateApprovalStageTaskMetric(updateRun, placementv1beta1.BeforeStageTaskLabelValue, 1)) + }) + }) }) var _ = Describe("UpdateRun execution tests - single stage", func() { @@ -1173,7 +1366,7 @@ var _ = Describe("UpdateRun execution tests - single stage", func() { validateUpdateRunMetricsEmitted(generateWaitingMetric(placementv1beta1.StateRun, updateRun), generateProgressingMetric(placementv1beta1.StateRun, updateRun)) }) - It("Should Should complete the 1st stage and complete the updateRun after the 3rd cluster succeeds", func() { + It("Should complete the 1st stage and complete the updateRun after the 3rd cluster succeeds", func() { By("Validating the 3rd clusterResourceBinding is updated to Bound") binding := resourceBindings[2] // cluster-2 validateBindingState(ctx, binding, resourceSnapshot.Name, updateRun, &updateRun.Status.StagesStatus[0].Clusters[2]) @@ -1959,7 +2152,7 @@ var _ = Describe("UpdateRun execution tests - single stage", func() { By("Validating the 1st stage has endTime set") Expect(updateRun.Status.StagesStatus[0].EndTime).ShouldNot(BeNil()) - By("Checking update run metrics are emitted") + By("Checking update run metrics are emitted") validateUpdateRunMetricsEmitted(generateInitializationSucceededMetric(placementv1beta1.StateInitialize, updateRun), generateProgressingMetric(placementv1beta1.StateRun, updateRun), generateSucceededMetric(placementv1beta1.StateRun, updateRun)) validateUpdateRunStageMetricsEmitted(generateStageClusterUpdatingMetric(updateRun)) }) diff --git a/pkg/controllers/updaterun/execution_test.go b/pkg/controllers/updaterun/execution_test.go index c65d8fb1f..52664c4cc 100644 --- a/pkg/controllers/updaterun/execution_test.go +++ b/pkg/controllers/updaterun/execution_test.go @@ -26,6 +26,7 @@ import ( "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -37,6 +38,7 @@ import ( placementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" "github.com/kubefleet-dev/kubefleet/pkg/utils/condition" + "github.com/kubefleet-dev/kubefleet/pkg/utils/controller" ) func TestIsBindingSyncedWithClusterStatus(t *testing.T) { @@ -363,6 +365,12 @@ func TestCheckClusterUpdateResult(t *testing.T) { } func TestBuildApprovalRequestObject(t *testing.T) { + // Pin the values that show up on the approval requests of the delete stage, which users see and select with. + const ( + deleteStageName = "kubernetes-fleet.io/deleteStage" + deleteStageLabelValue = "delete-stage" + deleteStageApprovalRequestName = "test-update-run-before-delete-stage" + ) tests := []struct { name string namespacedName types.NamespacedName @@ -422,6 +430,56 @@ func TestBuildApprovalRequestObject(t *testing.T) { }, }, }, + { + name: "should create ClusterApprovalRequest with a valid stage label for the delete stage", + namespacedName: types.NamespacedName{ + Name: deleteStageApprovalRequestName, + }, + stageName: deleteStageName, + updateRunName: "test-update-run", + stageTaskType: placementv1beta1.BeforeStageTaskLabelValue, + want: &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: deleteStageApprovalRequestName, + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: deleteStageLabelValue, + placementv1beta1.TargetUpdateRunLabel: "test-update-run", + placementv1beta1.TaskTypeLabel: placementv1beta1.BeforeStageTaskLabelValue, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + }, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: "test-update-run", + TargetStage: deleteStageName, + }, + }, + }, + { + name: "should create namespaced ApprovalRequest with a valid stage label for the delete stage", + namespacedName: types.NamespacedName{ + Name: deleteStageApprovalRequestName, + Namespace: testNamespaceName, + }, + stageName: deleteStageName, + updateRunName: "test-update-run", + stageTaskType: placementv1beta1.BeforeStageTaskLabelValue, + want: &placementv1beta1.ApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: deleteStageApprovalRequestName, + Namespace: testNamespaceName, + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: deleteStageLabelValue, + placementv1beta1.TargetUpdateRunLabel: "test-update-run", + placementv1beta1.TaskTypeLabel: placementv1beta1.BeforeStageTaskLabelValue, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + }, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: "test-update-run", + TargetStage: deleteStageName, + }, + }, + }, } for _, test := range tests { @@ -1251,7 +1309,9 @@ func TestCheckBeforeStageTasksStatus_NegativeCases(t *testing.T) { Client: fakeClient, } ctx := context.Background() - _, gotErr := r.checkBeforeStageTasksStatus(ctx, tt.stageIndex, tt.updateRun) + status := tt.updateRun.GetUpdateRunStatus() + tasks := status.UpdateStrategySnapshot.Stages[tt.stageIndex].BeforeStageTasks + _, gotErr := r.checkBeforeStageTasksStatus(ctx, &status.StagesStatus[tt.stageIndex], tasks, tt.updateRun) if gotErr == nil { t.Fatalf("checkBeforeStageTasksStatus() want error but got nil") } @@ -1265,6 +1325,318 @@ func TestCheckBeforeStageTasksStatus_NegativeCases(t *testing.T) { } } +func TestExecuteDeleteStage(t *testing.T) { + const ( + updateRunName = "test-update-run" + clusterName = "cluster-1" + ) + approvalRequestName := fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunName, placementv1beta1.UpdateRunDeleteStageTaskName) + now := metav1.Now() + + approvalTask := placementv1beta1.StageTask{Type: placementv1beta1.StageTaskTypeApproval} + approvalTaskStatus := placementv1beta1.StageTaskStatus{Type: placementv1beta1.StageTaskTypeApproval, ApprovalRequestName: approvalRequestName} + newCondition := func(condType any, status metav1.ConditionStatus, reason string) metav1.Condition { + return metav1.Condition{Type: fmt.Sprint(condType), Status: status, Reason: reason, ObservedGeneration: 1} + } + stageWaitingCond := newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionFalse, condition.StageUpdatingWaitingReason) + stageStartedCond := newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionTrue, condition.StageUpdatingStartedReason) + runWaitingCond := newCondition(placementv1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionFalse, condition.UpdateRunWaitingReason) + runProgressingCond := newCondition(placementv1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionTrue, condition.UpdateRunProgressingReason) + requestCreatedCond := newCondition(placementv1beta1.StageTaskConditionApprovalRequestCreated, metav1.ConditionTrue, condition.StageTaskApprovalRequestCreatedReason) + requestApprovedCond := newCondition(placementv1beta1.StageTaskConditionApprovalRequestApproved, metav1.ConditionTrue, condition.StageTaskApprovalRequestApprovedReason) + clusterStartedCond := newCondition(placementv1beta1.ClusterUpdatingConditionStarted, metav1.ConditionTrue, condition.ClusterUpdatingStartedReason) + approvedRequest := &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{Name: approvalRequestName, Generation: 1}, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRunName, + TargetStage: placementv1beta1.UpdateRunDeleteStageName, + }, + Status: placementv1beta1.ApprovalRequestStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.ApprovalRequestConditionApproved, metav1.ConditionTrue, "Approved"), + }, + }, + } + noClusters := []placementv1beta1.ClusterUpdatingStatus{} + + tests := []struct { + name string + // tasks are the before stage tasks of the delete stage. + tasks []placementv1beta1.StageTask + // deleteStageStatus is the status of the delete stage, whose name and clusters are defaulted. + deleteStageStatus placementv1beta1.StageUpdatingStatus + approvalRequest *placementv1beta1.ClusterApprovalRequest + noBinding bool + wantFinished bool + wantWaitTime time.Duration + wantErr error + wantBindingKept bool + // wantDeleteStageStatus is the wanted status of the delete stage, whose name and clusters are defaulted. + wantDeleteStageStatus placementv1beta1.StageUpdatingStatus + wantRunConditions []metav1.Condition + wantApprovalRequest bool + }{ + { + name: "no delete stage configuration should delete the binding", + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + }, + { + name: "pending approval task should create the approval request and keep the binding", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + wantWaitTime: stageUpdatingWaitTime, + wantBindingKept: true, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond}, + }}, + Conditions: []metav1.Condition{stageWaitingCond}, + }, + wantRunConditions: []metav1.Condition{runWaitingCond}, + wantApprovalRequest: true, + }, + { + name: "approved approval task should delete the binding", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{stageWaitingCond}, + }, + approvalRequest: approvedRequest, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond, requestApprovedCond}, + }}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + wantRunConditions: []metav1.Condition{runProgressingCond}, + wantApprovalRequest: true, + }, + { + name: "accepted approval request that is unapproved afterwards should still delete the binding", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + approvalRequest: &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: approvedRequest.ObjectMeta, + Spec: approvedRequest.Spec, + Status: placementv1beta1.ApprovalRequestStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.ApprovalRequestConditionApproved, metav1.ConditionFalse, "Unapproved"), + newCondition(placementv1beta1.ApprovalRequestConditionApprovalAccepted, metav1.ConditionTrue, condition.ApprovalRequestApprovalAcceptedReason), + }, + }, + }, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond, requestApprovedCond}, + }}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + wantRunConditions: []metav1.Condition{runProgressingCond}, + wantApprovalRequest: true, + }, + { + // A delete stage that has started is not gated again, so no approval request is created. + name: "tasks should not be checked again after the delete stage has started", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + StartTime: &now, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + }, + { + name: "tasks should be skipped when there is no cluster to delete", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: noClusters, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + noBinding: true, + wantFinished: true, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: noClusters, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionFalse, condition.StageUpdatingSucceededReason), + newCondition(placementv1beta1.StageUpdatingConditionSucceeded, metav1.ConditionTrue, condition.StageUpdatingSucceededReason), + }, + }, + }, + { + name: "tasks should be skipped when the bindings are already deleted", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + noBinding: true, + wantFinished: true, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ + ClusterName: clusterName, + Conditions: []metav1.Condition{ + clusterStartedCond, + newCondition(placementv1beta1.ClusterUpdatingConditionSucceeded, metav1.ConditionTrue, condition.ClusterUpdatingSucceededReason), + }, + }}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionFalse, condition.StageUpdatingSucceededReason), + newCondition(placementv1beta1.StageUpdatingConditionSucceeded, metav1.ConditionTrue, condition.StageUpdatingSucceededReason), + }, + }, + }, + { + name: "approval request targeting another update run should abort the update run", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + approvalRequest: &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{Name: approvalRequestName, Generation: 1}, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: "another-update-run", + TargetStage: placementv1beta1.UpdateRunDeleteStageName, + }, + }, + wantErr: errStagedUpdatedAborted, + wantBindingKept: true, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond}, + }}, + }, + wantApprovalRequest: true, + }, + { + // The API rejects a TimedWait task on the delete stage, so this only happens with a corrupted snapshot. + name: "unsupported timed wait task should abort the update run", + tasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: time.Hour}}}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{Type: placementv1beta1.StageTaskTypeTimedWait}}, + }, + wantErr: errStagedUpdatedAborted, + wantBindingKept: true, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{Type: placementv1beta1.StageTaskTypeTimedWait}}, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + binding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "test-binding"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: clusterName}, + } + clientBuilder := fake.NewClientBuilder().WithScheme(scheme).WithStatusSubresource(&placementv1beta1.ClusterApprovalRequest{}) + var toBeDeletedBindings []placementv1beta1.BindingObj + if !tt.noBinding { + clientBuilder = clientBuilder.WithObjects(binding) + toBeDeletedBindings = append(toBeDeletedBindings, binding) + } + if tt.approvalRequest != nil { + clientBuilder = clientBuilder.WithObjects(tt.approvalRequest.DeepCopy()) + } + fakeClient := clientBuilder.Build() + r := &Reconciler{Client: fakeClient} + + deleteStageStatus := tt.deleteStageStatus.DeepCopy() + deleteStageStatus.StageName = placementv1beta1.UpdateRunDeleteStageName + if deleteStageStatus.Clusters == nil { + deleteStageStatus.Clusters = []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName}} + } + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: updateRunName, Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: deleteStageStatus, + }, + } + if tt.tasks != nil { + updateRun.Status.UpdateStrategySnapshot.DeleteStage = &placementv1beta1.DeleteStageConfig{BeforeStageTasks: tt.tasks} + } + // The metrics are global; an approved approval request records one. + t.Cleanup(func() { deleteUpdateRunMetrics(updateRun) }) + + gotFinished, gotWaitTime, gotErr := r.executeDeleteStage(ctx, updateRun, toBeDeletedBindings) + if !errors.Is(gotErr, tt.wantErr) { + t.Fatalf("executeDeleteStage() error = %v, want %v", gotErr, tt.wantErr) + } + if gotFinished != tt.wantFinished { + t.Errorf("executeDeleteStage() finished = %v, want %v", gotFinished, tt.wantFinished) + } + if gotWaitTime != tt.wantWaitTime { + t.Errorf("executeDeleteStage() waitTime = %v, want %v", gotWaitTime, tt.wantWaitTime) + } + + if !tt.noBinding { + err := fakeClient.Get(ctx, client.ObjectKeyFromObject(binding), &placementv1beta1.ClusterResourceBinding{}) + if gotBindingKept := err == nil; gotBindingKept != tt.wantBindingKept || (err != nil && !apierrors.IsNotFound(err)) { + t.Errorf("executeDeleteStage() binding kept = %v (get error: %v), want %v", gotBindingKept, err, tt.wantBindingKept) + } + } + err := fakeClient.Get(ctx, client.ObjectKey{Name: approvalRequestName}, &placementv1beta1.ClusterApprovalRequest{}) + if gotApprovalRequest := err == nil; gotApprovalRequest != tt.wantApprovalRequest || (err != nil && !apierrors.IsNotFound(err)) { + t.Errorf("executeDeleteStage() approval request exists = %v (get error: %v), want %v", gotApprovalRequest, err, tt.wantApprovalRequest) + } + + wantDeleteStageStatus := tt.wantDeleteStageStatus.DeepCopy() + wantDeleteStageStatus.StageName = placementv1beta1.UpdateRunDeleteStageName + if wantDeleteStageStatus.Clusters == nil { + wantDeleteStageStatus.Clusters = []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName}} + } + cmpOpts := []cmp.Option{ + cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message"), + cmpopts.IgnoreFields(placementv1beta1.StageUpdatingStatus{}, "StartTime", "EndTime"), + } + if diff := cmp.Diff(wantDeleteStageStatus, updateRun.Status.DeletionStageStatus, cmpOpts...); diff != "" { + t.Errorf("executeDeleteStage() delete stage status mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(tt.wantRunConditions, updateRun.Status.Conditions, cmpOpts...); diff != "" { + t.Errorf("executeDeleteStage() update run conditions mismatch (-want +got):\n%s", diff) + } + // The delete stage is only considered started once the tasks are completed. + if gotStarted, wantStarted := updateRun.Status.DeletionStageStatus.StartTime != nil, !tt.wantBindingKept; gotStarted != wantStarted { + t.Errorf("executeDeleteStage() delete stage started = %v, want %v", gotStarted, wantStarted) + } + }) + } +} + func TestGenerateStuckClustersString(t *testing.T) { tests := []struct { name string @@ -1464,3 +1836,400 @@ func TestExecute_ZeroClustersSkipsEntireStage(t *testing.T) { }) } } + +// TestExecuteDeleteStage_MultipleClusters tests that executeDeleteStage only deletes the bindings whose deletion has +// not started yet, and marks the clusters with no binding left as deleted. +func TestExecuteDeleteStage_MultipleClusters(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + newCondition := func(condType any, reason string) metav1.Condition { + return metav1.Condition{Type: fmt.Sprint(condType), Status: metav1.ConditionTrue, Reason: reason, ObservedGeneration: 1} + } + clusterStartedCond := newCondition(placementv1beta1.ClusterUpdatingConditionStarted, condition.ClusterUpdatingStartedReason) + clusterSucceededCond := newCondition(placementv1beta1.ClusterUpdatingConditionSucceeded, condition.ClusterUpdatingSucceededReason) + deletionTime := metav1.Now() + + // The deletion of the binding on cluster-1 has started already; the fake client keeps it as it has a finalizer. + deletingBinding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-1", DeletionTimestamp: &deletionTime, Finalizers: []string{"test-finalizer"}}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + } + binding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-2"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-2"}, + } + var deletedBindings []string + fakeClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(deletingBinding, binding).WithInterceptorFuncs(interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + deletedBindings = append(deletedBindings, obj.GetName()) + return c.Delete(ctx, obj, opts...) + }, + }).Build() + r := &Reconciler{Client: fakeClient} + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + {ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond}}, + {ClusterName: "cluster-2"}, + // The binding on cluster-3 is gone, so the cluster is deleted. + {ClusterName: "cluster-3", Conditions: []metav1.Condition{clusterStartedCond}}, + }, + }, + }, + } + + gotFinished, gotWaitTime, err := r.executeDeleteStage(ctx, updateRun, []placementv1beta1.BindingObj{deletingBinding, binding}) + if err != nil { + t.Fatalf("executeDeleteStage() error = %v, want no error", err) + } + if gotFinished || gotWaitTime != clusterUpdatingWaitTime { + t.Errorf("executeDeleteStage() = (%v, %v), want (false, %v)", gotFinished, gotWaitTime, clusterUpdatingWaitTime) + } + if diff := cmp.Diff([]string{"binding-2"}, deletedBindings); diff != "" { + t.Errorf("executeDeleteStage() deleted bindings mismatch (-want +got):\n%s", diff) + } + wantClusters := []placementv1beta1.ClusterUpdatingStatus{ + {ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond}}, + {ClusterName: "cluster-2", Conditions: []metav1.Condition{clusterStartedCond}}, + {ClusterName: "cluster-3", Conditions: []metav1.Condition{clusterStartedCond, clusterSucceededCond}}, + } + if diff := cmp.Diff(wantClusters, updateRun.Status.DeletionStageStatus.Clusters, cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message")); diff != "" { + t.Errorf("executeDeleteStage() delete stage clusters mismatch (-want +got):\n%s", diff) + } +} + +// TestExecuteDeleteStage_Errors tests that executeDeleteStage reports the API server errors as retriable and aborts +// the update run, without deleting any binding, if the delete stage status does not match the bindings. +func TestExecuteDeleteStage_Errors(t *testing.T) { + const updateRunName = "test-update-run" + approvalRequestName := fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunName, placementv1beta1.UpdateRunDeleteStageTaskName) + errAPIServer := apierrors.NewServiceUnavailable("api server error") + newCondition := func(condType any, reason string) metav1.Condition { + return metav1.Condition{Type: fmt.Sprint(condType), Status: metav1.ConditionTrue, Reason: reason, ObservedGeneration: 1} + } + clusterStartedCond := newCondition(placementv1beta1.ClusterUpdatingConditionStarted, condition.ClusterUpdatingStartedReason) + clusterSucceededCond := newCondition(placementv1beta1.ClusterUpdatingConditionSucceeded, condition.ClusterUpdatingSucceededReason) + approvalRequest := func(conds ...metav1.Condition) *placementv1beta1.ClusterApprovalRequest { + return &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{Name: approvalRequestName, Generation: 1}, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRunName, + TargetStage: placementv1beta1.UpdateRunDeleteStageName, + }, + Status: placementv1beta1.ApprovalRequestStatus{Conditions: conds}, + } + } + isApprovalRequest := func(obj client.Object) bool { + _, ok := obj.(*placementv1beta1.ClusterApprovalRequest) + return ok + } + + tests := []struct { + name string + withApproval bool + approvalRequest *placementv1beta1.ClusterApprovalRequest + // clusters are the clusters in the delete stage; cluster-2 is always added without any condition. + clusters []placementv1beta1.ClusterUpdatingStatus + interceptors interceptor.Funcs + wantErr error + }{ + { + name: "failing to create the approval request should be retriable", + withApproval: true, + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if isApprovalRequest(obj) { + return errAPIServer + } + return c.Create(ctx, obj, opts...) + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "failing to get the existing approval request should be retriable", + withApproval: true, + approvalRequest: approvalRequest(), + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if isApprovalRequest(obj) { + return errAPIServer + } + return c.Get(ctx, key, obj, opts...) + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "failing to accept the approval should be retriable", + withApproval: true, + approvalRequest: approvalRequest(newCondition(placementv1beta1.ApprovalRequestConditionApproved, "Approved")), + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, c client.Client, subResourceName string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + return errAPIServer + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "failing to delete a binding should be retriable", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + return errAPIServer + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "binding on a cluster missing from the delete stage should abort the update run", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "another-cluster"}}, + wantErr: errStagedUpdatedAborted, + }, + { + name: "deleted cluster that still has a binding should abort the update run", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond, clusterSucceededCond}}}, + wantErr: errStagedUpdatedAborted, + }, + { + name: "deleting cluster whose binding is not deleting should abort the update run", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond}}}, + wantErr: errStagedUpdatedAborted, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + // The binding on cluster-2 comes after the binding on cluster-1, so that an abort caused by the + // cluster-1 shows that no binding is deleted before all the clusters are checked. + bindings := []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-1"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + }, + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-2"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-2"}, + }, + } + clientBuilder := fake.NewClientBuilder().WithScheme(scheme). + WithStatusSubresource(&placementv1beta1.ClusterApprovalRequest{}). + WithObjects(bindings[0], bindings[1]) + if tt.approvalRequest != nil { + clientBuilder = clientBuilder.WithObjects(tt.approvalRequest) + } + var deletedBindings []string + interceptors := tt.interceptors + if interceptors.Delete == nil { + interceptors.Delete = func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + deletedBindings = append(deletedBindings, obj.GetName()) + return c.Delete(ctx, obj, opts...) + } + } + r := &Reconciler{Client: clientBuilder.WithInterceptorFuncs(interceptors).Build()} + + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: updateRunName, Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: append(tt.clusters, placementv1beta1.ClusterUpdatingStatus{ClusterName: "cluster-2"}), + }, + }, + } + if tt.withApproval { + updateRun.Status.UpdateStrategySnapshot.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeApproval}}, + } + updateRun.Status.DeletionStageStatus.BeforeStageTaskStatus = []placementv1beta1.StageTaskStatus{ + {Type: placementv1beta1.StageTaskTypeApproval, ApprovalRequestName: approvalRequestName}, + } + } + t.Cleanup(func() { deleteUpdateRunMetrics(updateRun) }) + + gotFinished, _, gotErr := r.executeDeleteStage(ctx, updateRun, bindings) + if !errors.Is(gotErr, tt.wantErr) { + t.Fatalf("executeDeleteStage() error = %v, want %v", gotErr, tt.wantErr) + } + if tt.wantErr == controller.ErrAPIServerError && errors.Is(gotErr, errStagedUpdatedAborted) { + t.Errorf("executeDeleteStage() error = %v, want a retriable error that does not abort the update run", gotErr) + } + if gotFinished { + t.Errorf("executeDeleteStage() finished = true, want false") + } + if len(deletedBindings) != 0 { + t.Errorf("executeDeleteStage() deleted bindings %v, want none", deletedBindings) + } + }) + } +} + +// TestExecuteDeleteStage_Namespaced tests that the approval request of the delete stage of a namespaced update run +// is created in the namespace of the update run. +func TestExecuteDeleteStage_Namespaced(t *testing.T) { + const ( + namespace = "test-namespace" + updateRunName = "test-update-run" + ) + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + approvalRequestName := fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunName, placementv1beta1.UpdateRunDeleteStageTaskName) + binding := &placementv1beta1.ResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "test-binding", Namespace: namespace}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + } + fakeClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(binding).Build() + r := &Reconciler{Client: fakeClient} + updateRun := &placementv1beta1.StagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: updateRunName, Namespace: namespace, Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeApproval}}, + }, + }, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{ + {Type: placementv1beta1.StageTaskTypeApproval, ApprovalRequestName: approvalRequestName}, + }, + }, + }, + } + + gotFinished, gotWaitTime, err := r.executeDeleteStage(ctx, updateRun, []placementv1beta1.BindingObj{binding}) + if err != nil { + t.Fatalf("executeDeleteStage() error = %v, want no error", err) + } + if gotFinished || gotWaitTime != stageUpdatingWaitTime { + t.Errorf("executeDeleteStage() = (%v, %v), want (false, %v)", gotFinished, gotWaitTime, stageUpdatingWaitTime) + } + gotApprovalRequest := &placementv1beta1.ApprovalRequest{} + if err := fakeClient.Get(ctx, client.ObjectKey{Name: approvalRequestName, Namespace: namespace}, gotApprovalRequest); err != nil { + t.Fatalf("Get() approval request = %v, want no error", err) + } + wantApprovalRequest := buildApprovalRequestObject(types.NamespacedName{Name: approvalRequestName, Namespace: namespace}, + placementv1beta1.UpdateRunDeleteStageName, updateRunName, placementv1beta1.BeforeStageTaskLabelValue) + if diff := cmp.Diff(wantApprovalRequest, gotApprovalRequest, cmpopts.IgnoreFields(metav1.ObjectMeta{}, "ResourceVersion")); diff != "" { + t.Errorf("executeDeleteStage() approval request mismatch (-want +got):\n%s", diff) + } + if got := gotApprovalRequest.Labels[placementv1beta1.TargetUpdatingStageNameLabel]; got != placementv1beta1.UpdateRunDeleteStageTaskName { + t.Errorf("executeDeleteStage() approval request stage label = %q, want %q", got, placementv1beta1.UpdateRunDeleteStageTaskName) + } + if err := fakeClient.Get(ctx, client.ObjectKeyFromObject(binding), &placementv1beta1.ResourceBinding{}); err != nil { + t.Errorf("Get() binding = %v, want the binding to be kept", err) + } +} + +// TestExecute_DeleteStageAbort tests that an abort in the delete stage marks the delete stage as failed. +func TestExecute_DeleteStageAbort(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + binding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "test-binding"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + } + r := &Reconciler{Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(binding).Build()} + // The delete stage does not include the cluster of the binding. + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "another-cluster"}}, + }, + }, + } + + _, _, err := r.execute(ctx, updateRun, 0, nil, []placementv1beta1.BindingObj{binding}) + if !errors.Is(err, errStagedUpdatedAborted) { + t.Fatalf("execute() error = %v, want %v", err, errStagedUpdatedAborted) + } + wantConds := []metav1.Condition{ + { + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: condition.StageUpdatingFailedReason, + }, + { + Type: string(placementv1beta1.StageUpdatingConditionSucceeded), + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: condition.StageUpdatingFailedReason, + }, + } + if diff := cmp.Diff(wantConds, updateRun.Status.DeletionStageStatus.Conditions, cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message")); diff != "" { + t.Errorf("execute() delete stage conditions mismatch (-want +got):\n%s", diff) + } +} + +// TestCheckAfterStageTasksStatus_ElapsedTimedWait tests that a timed wait task that has elapsed stays elapsed after +// the stage transitions again, e.g., when the update run is stopped and resumed. +func TestCheckAfterStageTasksStatus_ElapsedTimedWait(t *testing.T) { + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 2}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{{ + Name: "stage-1", + AfterStageTasks: []placementv1beta1.StageTask{ + {Type: placementv1beta1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: time.Hour}}, + }, + }}, + }, + StagesStatus: []placementv1beta1.StageUpdatingStatus{{ + StageName: "stage-1", + Conditions: []metav1.Condition{{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionFalse, + ObservedGeneration: 2, + Reason: condition.StageUpdatingStoppedReason, + LastTransitionTime: metav1.Now(), + }}, + AfterStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeTimedWait, + Conditions: []metav1.Condition{{ + Type: string(placementv1beta1.StageTaskConditionWaitTimeElapsed), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.AfterStageTaskWaitTimeElapsedReason, + }}, + }}, + }}, + }, + } + r := &Reconciler{} + + gotPassed, gotWaitTime, err := r.checkAfterStageTasksStatus(context.Background(), 0, updateRun) + if err != nil { + t.Fatalf("checkAfterStageTasksStatus() error = %v, want no error", err) + } + if !gotPassed || gotWaitTime != 0 { + t.Errorf("checkAfterStageTasksStatus() = (%v, %v), want (true, 0)", gotPassed, gotWaitTime) + } +} diff --git a/pkg/controllers/updaterun/initialization.go b/pkg/controllers/updaterun/initialization.go index 7a3217462..9f721eb18 100644 --- a/pkg/controllers/updaterun/initialization.go +++ b/pkg/controllers/updaterun/initialization.go @@ -285,6 +285,17 @@ func (r *Reconciler) generateStagesByStrategy( // Remove waitTime from the updateRun status for BeforeStageTask and AfterStageTask for type Approval. removeWaitTimeFromUpdateRunStatus(updateRun) + var deleteStageTasks []placementv1beta1.StageTask + if updateStrategySpec.DeleteStage != nil { + deleteStageTasks = updateStrategySpec.DeleteStage.BeforeStageTasks + } + if err := validateBeforeStageTask(deleteStageTasks); err != nil { + klog.ErrorS(err, "Failed to validate the before stage tasks of the delete stage", "updateStrategy", strategyKey, "updateRun", updateRunRef) + // no more retries here. + invalidBeforeStageErr := controller.NewUserError(fmt.Errorf("the before stage tasks are invalid, updateStrategy: `%s`, stage: %s, err: %s", strategyKey, placementv1beta1.UpdateRunDeleteStageName, err.Error())) + return fmt.Errorf("%w: %s", errValidationFailed, invalidBeforeStageErr.Error()) + } + // Compute the update stages. if err := r.computeRunStageStatus(ctx, scheduledBindings, updateRun); err != nil { return err @@ -305,6 +316,9 @@ func (r *Reconciler) generateStagesByStrategy( StageName: placementv1beta1.UpdateRunDeleteStageName, Clusters: toBeDeletedClusters, } + // Create the before stage tasks of the delete stage. + updateRunStatus.DeletionStageStatus.BeforeStageTaskStatus = buildStageTaskStatuses(deleteStageTasks, + fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), placementv1beta1.UpdateRunDeleteStageTaskName)) return nil } @@ -415,21 +429,11 @@ func (r *Reconciler) computeRunStageStatus( } // Create the before stage tasks. - curStageUpdatingStatus.BeforeStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.BeforeStageTasks)) - for i, task := range stage.BeforeStageTasks { - curStageUpdatingStatus.BeforeStageTaskStatus[i].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - curStageUpdatingStatus.BeforeStageTaskStatus[i].ApprovalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } + curStageUpdatingStatus.BeforeStageTaskStatus = buildStageTaskStatuses(stage.BeforeStageTasks, + fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name)) // Create the after stage tasks. - curStageUpdatingStatus.AfterStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.AfterStageTasks)) - for i, task := range stage.AfterStageTasks { - curStageUpdatingStatus.AfterStageTaskStatus[i].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - curStageUpdatingStatus.AfterStageTaskStatus[i].ApprovalRequestName = fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } + curStageUpdatingStatus.AfterStageTaskStatus = buildStageTaskStatuses(stage.AfterStageTasks, + fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name)) stagesStatus = append(stagesStatus, curStageUpdatingStatus) } updateRunStatus.StagesStatus = stagesStatus @@ -452,6 +456,19 @@ func (r *Reconciler) computeRunStageStatus( return nil } +// buildStageTaskStatuses builds the statuses of the given tasks of a stage, where approvalRequestName is the name of +// the approval request of the Approval task. +func buildStageTaskStatuses(tasks []placementv1beta1.StageTask, approvalRequestName string) []placementv1beta1.StageTaskStatus { + taskStatuses := make([]placementv1beta1.StageTaskStatus, len(tasks)) + for i, task := range tasks { + taskStatuses[i].Type = task.Type + if task.Type == placementv1beta1.StageTaskTypeApproval { + taskStatuses[i].ApprovalRequestName = approvalRequestName + } + } + return taskStatuses +} + // validateBeforeStageTask validates the beforeStageTasks in the stage defined in the UpdateStrategy. // The error returned from this function is not retriable. func validateBeforeStageTask(tasks []placementv1beta1.StageTask) error { diff --git a/pkg/controllers/updaterun/initialization_integration_test.go b/pkg/controllers/updaterun/initialization_integration_test.go index f208fc67a..be0faa6bb 100644 --- a/pkg/controllers/updaterun/initialization_integration_test.go +++ b/pkg/controllers/updaterun/initialization_integration_test.go @@ -697,7 +697,12 @@ var _ = Describe("Updaterun initialization tests", func() { }) It("Should generate the cluster update stage in the status as expected", func() { - By("Creating a clusterStagedUpdateStrategy") + By("Creating a clusterStagedUpdateStrategy with a before stage approval task for the delete stage") + updateStrategy.Spec.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + {Type: placementv1beta1.StageTaskTypeApproval}, + }, + } Expect(k8sClient.Create(ctx, updateStrategy)).To(Succeed()) By("Creating a new clusterStagedUpdateRun") @@ -1102,12 +1107,21 @@ func validateFailedInitCondition(ctx context.Context, updateRun *placementv1beta } // populateStageTaskStatuses populates the BeforeStageTaskStatus and AfterStageTaskStatus -// for all stages in the status based on the strategy configuration. +// for all stages, including the delete stage, in the status based on the strategy configuration. func populateStageTaskStatuses( status *placementv1beta1.UpdateRunStatus, updateRunName string, - stages []placementv1beta1.StageConfig, + strategy *placementv1beta1.UpdateStrategySpec, ) { + stages := strategy.Stages + if strategy.DeleteStage != nil && status.DeletionStageStatus != nil { + status.DeletionStageStatus.BeforeStageTaskStatus = buildTaskStatuses( + strategy.DeleteStage.BeforeStageTasks, + placementv1beta1.BeforeStageApprovalTaskNameFmt, + updateRunName, + placementv1beta1.UpdateRunDeleteStageTaskName, + ) + } for i := range status.StagesStatus { status.StagesStatus[i].BeforeStageTaskStatus = buildTaskStatuses( stages[i].BeforeStageTasks, @@ -1170,7 +1184,7 @@ func generateInitializedStatus( generateTrueCondition(updateRun, placementv1beta1.StagedUpdateRunConditionInitialized), }, } - populateStageTaskStatuses(status, updateRun.Name, updateStrategy.Spec.Stages) + populateStageTaskStatuses(status, updateRun.Name, &updateStrategy.Spec) return status } diff --git a/pkg/controllers/updaterun/initialization_test.go b/pkg/controllers/updaterun/initialization_test.go index 6ce3689a8..29f8e5654 100644 --- a/pkg/controllers/updaterun/initialization_test.go +++ b/pkg/controllers/updaterun/initialization_test.go @@ -17,12 +17,17 @@ limitations under the License. package updaterun import ( + "context" + "errors" "fmt" + "strings" "testing" "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client/fake" placementv1beta1 "github.com/kubefleet-dev/kubefleet/apis/placement/v1beta1" ) @@ -169,3 +174,34 @@ func TestValidateAfterStageTask(t *testing.T) { }) } } + +// TestGenerateStagesByStrategy_InvalidDeleteStageTasks tests that an update run whose strategy has invalid before +// stage tasks in its delete stage fails the validation. +func TestGenerateStagesByStrategy_InvalidDeleteStageTasks(t *testing.T) { + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + strategy := &placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{Name: "test-strategy"}, + Spec: placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeTimedWait}}, + }, + }, + } + r := &Reconciler{Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(strategy).Build()} + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 1}, + Spec: placementv1beta1.UpdateRunSpec{StagedUpdateStrategyName: strategy.Name}, + } + + err := r.generateStagesByStrategy(context.Background(), nil, nil, updateRun) + if !errors.Is(err, errValidationFailed) { + t.Fatalf("generateStagesByStrategy() error = %v, want %v", err, errValidationFailed) + } + wantErrMsg := fmt.Sprintf("the before stage tasks are invalid, updateStrategy: `/test-strategy`, stage: %s, err: task 0 of type TimedWait is not allowed in beforeStageTasks, allowed type: Approval", placementv1beta1.UpdateRunDeleteStageName) + if !strings.Contains(err.Error(), wantErrMsg) { + t.Errorf("generateStagesByStrategy() error = %v, want an error containing %q", err, wantErrMsg) + } +} diff --git a/pkg/controllers/updaterun/stop.go b/pkg/controllers/updaterun/stop.go index 17c878dc3..005493c9a 100644 --- a/pkg/controllers/updaterun/stop.go +++ b/pkg/controllers/updaterun/stop.go @@ -109,7 +109,7 @@ func (r *Reconciler) stopUpdatingStage( // If cluster update has been running for more than "updateRunStuckThreshold", mark the update run as stuck. timeElapsed := time.Since(clusterStartedCond.LastTransitionTime.Time) if timeElapsed > updateRunStuckThreshold { - klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "time elapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) + klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "timeElapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) stuckClusterNames = append(stuckClusterNames, clusterStatus.ClusterName) } } @@ -142,51 +142,11 @@ func (r *Reconciler) stopDeleteStage( ) (bool, error) { updateRunRef := klog.KObj(updateRun) updateRunStatus := updateRun.GetUpdateRunStatus() - existingDeleteStageStatus := updateRunStatus.DeletionStageStatus - existingDeleteStageClusterMap := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(existingDeleteStageStatus.Clusters)) - for i := range existingDeleteStageStatus.Clusters { - existingDeleteStageClusterMap[existingDeleteStageStatus.Clusters[i].ClusterName] = &existingDeleteStageStatus.Clusters[i] - } // Mark the delete stage as stopping in case it's not. - markStageUpdatingStopping(existingDeleteStageStatus, updateRun.GetGeneration()) - - for _, binding := range toBeDeletedBindings { - bindingSpec := binding.GetBindingSpec() - curCluster, exist := existingDeleteStageClusterMap[bindingSpec.TargetCluster] - if !exist { - // This is unexpected because we already checked in validation. - missingErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the to be deleted cluster `%s` is not in the deleting stage during stopping", bindingSpec.TargetCluster)) - klog.ErrorS(missingErr, "The cluster in the deleting stage does not include all the to be deleted binding", "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, missingErr.Error()) - } - // In validation, we already check the binding must exist in the status. - delete(existingDeleteStageClusterMap, bindingSpec.TargetCluster) - // Make sure the cluster is not marked as deleted as the binding is still there. - if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionSucceeded)), updateRun.GetGeneration()) { - // The cluster status is marked as deleted. - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the deleted cluster `%s` in the deleting stage still has a binding", bindingSpec.TargetCluster)) - klog.ErrorS(unexpectedErr, "The cluster in the deleting stage is not removed yet but marked as deleted", "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } - if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { - // The cluster status is marked as being deleted. - if binding.GetDeletionTimestamp().IsZero() { - // The cluster is marked as deleting but the binding is not deleting. - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the cluster `%s` in the deleting stage is marked as deleting but its corresponding binding is not deleting", curCluster.ClusterName)) - klog.ErrorS(unexpectedErr, "The binding should be deleting before we mark a cluster deleting", "clusterStatus", curCluster, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } - continue - } - } - - // The rest of the clusters in the stage are not in the toBeDeletedBindings so it should be marked as delete succeeded. - for _, clusterStatus := range existingDeleteStageClusterMap { - // Make sure the cluster is marked as deleted. - if !condition.IsConditionStatusTrue(meta.FindStatusCondition(clusterStatus.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { - markClusterUpdatingStarted(clusterStatus, updateRun.GetGeneration()) - } - markClusterUpdatingSucceeded(clusterStatus, updateRun.GetGeneration()) + markStageUpdatingStopping(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) + // The clusters whose deletion has not started are left as they are. + if _, err := syncDeleteStageClusterStatuses(updateRun, toBeDeletedBindings); err != nil { + return false, err } klog.V(2).InfoS("The delete stage is stopping", "numberOfDeletingClusters", len(toBeDeletedBindings), "updateRun", updateRunRef) @@ -204,7 +164,7 @@ func (r *Reconciler) stopDeleteStage( if allDeletingClustersDeleted { markStageUpdatingStopped(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) } - return len(toBeDeletedBindings) == 0, nil + return allDeletingClustersDeleted, nil } // markUpdateRunStopping marks the update run as stopping in memory. @@ -216,11 +176,11 @@ func markUpdateRunStopping(updateRun placementv1beta1.UpdateRunObj) { Status: metav1.ConditionUnknown, ObservedGeneration: updateRun.GetGeneration(), Reason: condition.UpdateRunStoppingReason, - Message: "The update run is the process of stopping, waiting for all the updating/deleting clusters to finish updating before completing the stop process", + Message: "The update run is in the process of stopping, waiting for all the updating/deleting clusters to finish updating before completing the stop process", }) } -// markStageUpdatingStopping marks the stage updating status as pausing in memory. +// markStageUpdatingStopping marks the stage updating status as stopping in memory. func markStageUpdatingStopping(stageUpdatingStatus *placementv1beta1.StageUpdatingStatus, generation int64) { meta.SetStatusCondition(&stageUpdatingStatus.Conditions, metav1.Condition{ Type: string(placementv1beta1.StageUpdatingConditionProgressing), @@ -242,6 +202,8 @@ func markStageUpdatingStopped(stageUpdatingStatus *placementv1beta1.StageUpdatin }) } +// checkIfErrorStagedUpdateAborted marks the updating stage, or the delete stage if there is no updating stage, +// as failed in memory if the error aborts the update run. func checkIfErrorStagedUpdateAborted(err error, updateRun placementv1beta1.UpdateRunObj, updatingStageStatus *placementv1beta1.StageUpdatingStatus) { if errors.Is(err, errStagedUpdatedAborted) { if updatingStageStatus != nil { diff --git a/pkg/controllers/updaterun/stop_integration_test.go b/pkg/controllers/updaterun/stop_integration_test.go index a0fdfa1c1..f43b741f7 100644 --- a/pkg/controllers/updaterun/stop_integration_test.go +++ b/pkg/controllers/updaterun/stop_integration_test.go @@ -625,6 +625,157 @@ var _ = Describe("UpdateRun stop tests", func() { validateUpdateRunStageMetricsEmitted(generateStageClusterUpdatingMetric(updateRun)) }) }) + + Context("Cluster staged update run should stop and resume at the before-stage tasks of the delete stage", Ordered, func() { + var approvalRequestName string + + getUpdateRun := func() (*placementv1beta1.ClusterStagedUpdateRun, error) { + gotUpdateRun := &placementv1beta1.ClusterStagedUpdateRun{} + if err := k8sClient.Get(ctx, updateRunNamespacedName, gotUpdateRun); err != nil { + return nil, fmt.Errorf("failed to get the update run: %w", err) + } + return gotUpdateRun, nil + } + + toBeDeletedBindingsKeptActual := func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding); err != nil { + return fmt.Errorf("failed to get binding %s: %w", resourceBindings[i].Name, err) + } + if !binding.DeletionTimestamp.IsZero() { + return fmt.Errorf("binding %s is being deleted", binding.Name) + } + } + return nil + } + + BeforeAll(func() { + By("Reassigning the strategy to have no stage tasks and the delete stage with a before-stage approval task") + updateStrategy.Spec.Stages[0].BeforeStageTasks = nil + updateStrategy.Spec.Stages[0].AfterStageTasks = nil + updateStrategy.Spec.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + } + Expect(k8sClient.Update(ctx, updateStrategy)).To(Succeed()) + + By("Creating a new clusterStagedUpdateRun") + updateRun.Spec.State = placementv1beta1.StateRun + Expect(k8sClient.Create(ctx, updateRun)).To(Succeed()) + approvalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.Name, placementv1beta1.UpdateRunDeleteStageTaskName) + }) + + It("Should complete the 1st stage once all bindings are available", func() { + Eventually(func() error { + for i := 0; i < numTargetClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding); err != nil { + return fmt.Errorf("failed to get binding %s: %w", resourceBindings[i].Name, err) + } + if binding.Spec.State != placementv1beta1.BindingStateBound { + return fmt.Errorf("binding %s is not bound yet", binding.Name) + } + if meta.IsStatusConditionTrue(binding.Status.Conditions, string(placementv1beta1.ResourceBindingAvailable)) { + continue + } + meta.SetStatusCondition(&binding.Status.Conditions, generateTrueCondition(binding, placementv1beta1.ResourceBindingAvailable)) + if err := k8sClient.Status().Update(ctx, binding); err != nil { + return fmt.Errorf("failed to mark binding %s as available: %w", binding.Name, err) + } + } + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + if !meta.IsStatusConditionTrue(gotUpdateRun.Status.StagesStatus[0].Conditions, string(placementv1beta1.StageUpdatingConditionSucceeded)) { + return fmt.Errorf("the 1st stage has not succeeded yet") + } + return nil + }, 3*timeout, interval).Should(Succeed(), "failed to complete the 1st stage") + }) + + It("Should wait at the delete stage once the approval request is created", func() { + Eventually(func() error { + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + deleteStageStatus := gotUpdateRun.Status.DeletionStageStatus + if !meta.IsStatusConditionTrue(deleteStageStatus.BeforeStageTaskStatus[0].Conditions, string(placementv1beta1.StageTaskConditionApprovalRequestCreated)) { + return fmt.Errorf("the approval request of the delete stage has not been created yet") + } + if deleteStageStatus.StartTime != nil { + return fmt.Errorf("the delete stage has started before its tasks completed") + } + return nil + }, timeout, interval).Should(Succeed(), "failed to wait at the delete stage") + + Expect(toBeDeletedBindingsKeptActual()).To(Succeed(), "the to-be-deleted bindings should be kept while waiting") + }) + + It("Should stop the update run at the delete stage and keep the bindings when state is Stop", func() { + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateStop) + + By("Validating the delete stage and the update run have stopped") + Eventually(func() error { + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + stageCond := meta.FindStatusCondition(gotUpdateRun.Status.DeletionStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionProgressing)) + if stageCond == nil || stageCond.Reason != condition.StageUpdatingStoppedReason || stageCond.ObservedGeneration != gotUpdateRun.Generation { + return fmt.Errorf("the delete stage has not stopped yet: %v", stageCond) + } + runCond := meta.FindStatusCondition(gotUpdateRun.Status.Conditions, string(placementv1beta1.StagedUpdateRunConditionProgressing)) + if runCond == nil || runCond.Reason != condition.UpdateRunStoppedReason || runCond.ObservedGeneration != gotUpdateRun.Generation { + return fmt.Errorf("the update run has not stopped yet: %v", runCond) + } + return nil + }, timeout, interval).Should(Succeed(), "failed to stop the update run") + + By("Approving the approval request and validating the bindings are kept while stopped") + approveClusterApprovalRequest(ctx, approvalRequestName) + Consistently(toBeDeletedBindingsKeptActual, 2*stageUpdatingWaitTime, interval).Should(Succeed(), "the to-be-deleted bindings should be kept while stopped") + }) + + It("Should delete the bindings and complete the update run when state is Run", func() { + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateRun) + + By("Validating the to-be-deleted bindings are deleted") + Eventually(func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding) + if err == nil { + return fmt.Errorf("binding %s is not deleted", binding.Name) + } + if !apierrors.IsNotFound(err) { + return fmt.Errorf("get binding %s does not return a not-found error: %w", binding.Name, err) + } + } + return nil + }, timeout, interval).Should(Succeed(), "failed to delete the to-be-deleted bindings") + + By("Validating the update run has succeeded") + Eventually(func() error { + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + if !meta.IsStatusConditionTrue(gotUpdateRun.Status.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions, string(placementv1beta1.StageTaskConditionApprovalRequestApproved)) { + return fmt.Errorf("the approval task of the delete stage is not approved") + } + if !meta.IsStatusConditionTrue(gotUpdateRun.Status.Conditions, string(placementv1beta1.StagedUpdateRunConditionSucceeded)) { + return fmt.Errorf("the update run has not succeeded yet") + } + return nil + }, timeout, interval).Should(Succeed(), "failed to complete the update run") + }) + }) }) func updateClusterStagedUpdateRunState(updateRunName string, state placementv1beta1.State) *placementv1beta1.ClusterStagedUpdateRun { diff --git a/pkg/controllers/updaterun/stop_test.go b/pkg/controllers/updaterun/stop_test.go index 499e1111d..31a460779 100644 --- a/pkg/controllers/updaterun/stop_test.go +++ b/pkg/controllers/updaterun/stop_test.go @@ -613,7 +613,61 @@ func TestStopDeleteStage(t *testing.T) { }, }, { - name: "cluster not marked as deleting and binding not deleting", + name: "deleted cluster and cluster not marked as deleting should stop without deleting the remaining binding", + updateRun: &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-updaterun", + Generation: 1, + }, + Status: placementv1beta1.UpdateRunStatus{ + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: "deletion", + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + { + ClusterName: "cluster-1", + Conditions: []metav1.Condition{ + { + Type: string(placementv1beta1.ClusterUpdatingConditionStarted), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingStartedReason, + }, + { + Type: string(placementv1beta1.ClusterUpdatingConditionSucceeded), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingSucceededReason, + }, + }, + }, + { + ClusterName: "cluster-2", + }, + }, + }, + }, + }, + toBeDeletedBindings: []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{ + // No DeletionTimestamp set + }, + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-2", + }, + }, + }, + wantFinished: true, + wantError: nil, + wantProgressCond: metav1.Condition{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: condition.StageUpdatingStoppedReason, + }, + }, + { + name: "cluster waiting on delete stage tasks should stop without deleting its binding", updateRun: &placementv1beta1.ClusterStagedUpdateRun{ ObjectMeta: metav1.ObjectMeta{ Name: "test-updaterun", @@ -640,7 +694,7 @@ func TestStopDeleteStage(t *testing.T) { }, }, }, - wantFinished: false, + wantFinished: true, wantError: nil, wantProgressCond: metav1.Condition{ Type: string(placementv1beta1.StageUpdatingConditionProgressing), @@ -649,6 +703,111 @@ func TestStopDeleteStage(t *testing.T) { Reason: condition.StageUpdatingStoppedReason, }, }, + { + name: "deleting cluster and cluster not marked as deleting should not finish", + updateRun: &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-updaterun", + Generation: 1, + }, + Status: placementv1beta1.UpdateRunStatus{ + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: "deletion", + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + { + ClusterName: "cluster-1", + Conditions: []metav1.Condition{ + { + Type: string(placementv1beta1.ClusterUpdatingConditionStarted), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + LastTransitionTime: now, + Reason: condition.ClusterUpdatingStartedReason, + }, + }, + }, + { + ClusterName: "cluster-2", + }, + }, + }, + }, + }, + toBeDeletedBindings: []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{ + DeletionTimestamp: &deletionTime, + }, + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-1", + }, + }, + &placementv1beta1.ClusterResourceBinding{ + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-2", + }, + }, + }, + wantFinished: false, + wantError: nil, + wantProgressCond: metav1.Condition{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionUnknown, + ObservedGeneration: 1, + Reason: condition.StageUpdatingStoppingReason, + }, + }, + { + name: "deleted cluster that still has a binding - should abort", + updateRun: &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-updaterun", + Generation: 1, + }, + Status: placementv1beta1.UpdateRunStatus{ + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: "deletion", + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + { + ClusterName: "cluster-1", + Conditions: []metav1.Condition{ + { + Type: string(placementv1beta1.ClusterUpdatingConditionStarted), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingStartedReason, + }, + { + Type: string(placementv1beta1.ClusterUpdatingConditionSucceeded), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingSucceededReason, + }, + }, + }, + }, + }, + }, + }, + toBeDeletedBindings: []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{ + DeletionTimestamp: &deletionTime, + }, + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-1", + }, + }, + }, + wantFinished: false, + wantError: errors.New("the deleted cluster `cluster-1` in the deleting stage still has a binding"), + wantProgressCond: metav1.Condition{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionUnknown, + ObservedGeneration: 1, + Reason: condition.StageUpdatingStoppingReason, + }, + }, { name: "multiple clusters with mixed states", updateRun: &placementv1beta1.ClusterStagedUpdateRun{ @@ -729,13 +888,13 @@ func TestStopDeleteStage(t *testing.T) { // Verify error expectation. if (tt.wantError != nil) != (gotErr != nil) { - t.Fatalf("stopUpdatingStage() want error: %v, got error: %v", tt.wantError, gotErr) + t.Fatalf("stopDeleteStage() want error: %v, got error: %v", tt.wantError, gotErr) } // Verify error message contains expected substring. if tt.wantError != nil && gotErr != nil { if !strings.Contains(gotErr.Error(), tt.wantError.Error()) { - t.Fatalf("stopUpdatingStage() want error: %v, got error: %v", tt.wantError, gotErr) + t.Fatalf("stopDeleteStage() want error: %v, got error: %v", tt.wantError, gotErr) } } diff --git a/pkg/controllers/updaterun/validation.go b/pkg/controllers/updaterun/validation.go index a102637b3..ad5ff084c 100644 --- a/pkg/controllers/updaterun/validation.go +++ b/pkg/controllers/updaterun/validation.go @@ -289,8 +289,8 @@ func validateDeleteStageStatus( } } - deleteStageFinishedCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionSucceeded)) - deleteStageProgressingCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionProgressing)) + deleteStageFinishedCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionSucceeded)) + deleteStageProgressingCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionProgressing)) // Check if there is any active updating stage if updatingStageIndex != -1 || lastFinishedStageIndex < totalStages-1 { // There are still stages updating before the delete stage, make sure the delete stage is not active/finished. diff --git a/test/apis/placement/v1beta1/api_validation_integration_test.go b/test/apis/placement/v1beta1/api_validation_integration_test.go index 8f084fb7e..ead8c7bf4 100644 --- a/test/apis/placement/v1beta1/api_validation_integration_test.go +++ b/test/apis/placement/v1beta1/api_validation_integration_test.go @@ -2901,6 +2901,31 @@ var _ = Describe("Test placement v1beta1 API validation", func() { Expect(statusErr.ErrStatus.Message).Should(MatchRegexp("Too many: 3: must have at most 2 items")) }) + It("Should deny creation of ClusterStagedUpdateStrategy with BeforeStageTask of type Approval with waitTime specified", func() { + strategy := placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(updateRunStrategyNameTemplate, GinkgoParallelProcess()), + }, + Spec: placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{ + { + Name: fmt.Sprintf(updateRunStageNameTemplate, GinkgoParallelProcess(), 1), + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + WaitTime: &metav1.Duration{Duration: time.Minute * 10}, + }, + }, + }, + }, + }, + } + err := hubClient.Create(ctx, &strategy) + var statusErr *k8sErrors.StatusError + Expect(errors.As(err, &statusErr)).To(BeTrue(), fmt.Sprintf("Create updateRunStrategy call produced error %s. Error type wanted is %s.", reflect.TypeOf(err), reflect.TypeOf(&k8sErrors.StatusError{}))) + Expect(statusErr.ErrStatus.Message).Should(MatchRegexp("BeforeStageTaskType is Approval, waitTime is not allowed")) + }) + It("Should deny creation of ClusterStagedUpdateStrategy with AfterStageTask of type Approval with waitTime specified", func() { strategy := placementv1beta1.ClusterStagedUpdateStrategy{ ObjectMeta: metav1.ObjectMeta{ @@ -3380,6 +3405,74 @@ var _ = Describe("Test placement v1beta1 API validation", func() { }) }) + // The delete stage tasks have the same rules in both scopes, as the cluster-scoped and the namespaced + // strategies share their spec; each CRD is generated separately, so both are checked. + Context("Test the delete stage of a StagedUpdateStrategy and a ClusterStagedUpdateStrategy", func() { + approval := placementv1beta1.StageTask{Type: placementv1beta1.StageTaskTypeApproval} + + deleteStageEntries := func(namespaced bool) []TableEntry { + scope := "ClusterStagedUpdateStrategy" + if namespaced { + scope = "StagedUpdateStrategy" + } + return []TableEntry{ + Entry(scope+": a delete stage without tasks is accepted", + namespaced, &placementv1beta1.DeleteStageConfig{}, ""), + Entry(scope+": a single Approval task is accepted", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{approval}}, ""), + Entry(scope+": more than 1 task is rejected", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{approval, approval}}, + "Too many: 2: must have at most 1 item"), + Entry(scope+": a TimedWait task is rejected", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: time.Minute}}}}, + "BeforeStageTaskType cannot be TimedWait"), + Entry(scope+": an Approval task with a waitTime is rejected", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeApproval, WaitTime: &metav1.Duration{Duration: 10 * time.Minute}}}}, + "BeforeStageTaskType is Approval, waitTime is not allowed"), + } + } + + DescribeTable("the before-stage tasks of the delete stage", + func(namespaced bool, deleteStage *placementv1beta1.DeleteStageConfig, wantMessage string) { + spec := placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{ + { + Name: fmt.Sprintf(updateRunStageNameTemplate, GinkgoParallelProcess(), 1), + }, + }, + DeleteStage: deleteStage, + } + var strategy client.Object = &placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(updateRunStrategyNameTemplate, GinkgoParallelProcess()), + }, + Spec: spec, + } + if namespaced { + strategy = &placementv1beta1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(updateRunStrategyNameTemplate, GinkgoParallelProcess()), + Namespace: testNamespace, + }, + Spec: spec, + } + } + + err := hubClient.Create(ctx, strategy) + if wantMessage == "" { + Expect(err).To(Succeed(), "Expected the strategy to be accepted") + Expect(hubClient.Delete(ctx, strategy)).To(Succeed()) + return + } + var statusErr *k8sErrors.StatusError + Expect(errors.As(err, &statusErr)).To(BeTrue(), fmt.Sprintf("Create updateRunStrategy call produced error %s. Error type wanted is %s.", reflect.TypeOf(err), reflect.TypeOf(&k8sErrors.StatusError{}))) + Expect(statusErr.ErrStatus.Message).Should(ContainSubstring(wantMessage)) + }, + deleteStageEntries(false), + deleteStageEntries(true), + ) + }) + Context("Test ClusterStagedUpdateRun State API validation - valid Initialize state transitions", func() { var updateRun *placementv1beta1.ClusterStagedUpdateRun updateRunName := fmt.Sprintf(validupdateRunNameTemplate, GinkgoParallelProcess()) diff --git a/test/e2e/cluster_staged_updaterun_test.go b/test/e2e/cluster_staged_updaterun_test.go index 3215362ea..914275c9b 100644 --- a/test/e2e/cluster_staged_updaterun_test.go +++ b/test/e2e/cluster_staged_updaterun_test.go @@ -691,6 +691,123 @@ var _ = Describe("test CRP rollout with staged update run", func() { }) }) + Context("Test cluster shrink with before-stage tasks for the delete stage", Ordered, func() { + var strategy *placementv1beta1.ClusterStagedUpdateStrategy + updateRunNames := []string{} + + BeforeAll(func() { + // Create a test namespace and a configMap inside it on the hub cluster. + createWorkResources() + + // Create the CRP with external rollout strategy and pick fixed policy. + crp := &placementv1beta1.ClusterResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{ + Name: crpName, + // Add a custom finalizer; this would allow us to better observe + // the behavior of the controllers. + Finalizers: []string{customDeletionBlockerFinalizer}, + }, + Spec: placementv1beta1.PlacementSpec{ + ResourceSelectors: workResourceSelector(), + Policy: &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{allMemberClusterNames[0], allMemberClusterNames[1]}, // member-cluster-1 and member-cluster-2 + }, + Strategy: placementv1beta1.RolloutStrategy{ + Type: placementv1beta1.ExternalRolloutStrategyType, + }, + }, + } + Expect(hubClient.Create(ctx, crp)).To(Succeed(), "Failed to create CRP") + + // Create the clusterStagedUpdateStrategy with a single stage and a gated delete stage. + strategy = &placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: strategyName, + }, + Spec: updateStrategySpecWithDeleteStageTasks(), + } + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create ClusterStagedUpdateStrategy") + + for i := 0; i < 2; i++ { + updateRunNames = append(updateRunNames, fmt.Sprintf(clusterStagedUpdateRunNameWithSubIndexTemplate, GinkgoParallelProcess(), i)) + } + }) + + AfterAll(func() { + // Remove the custom deletion blocker finalizer from the CRP. + ensureCRPAndRelatedResourcesDeleted(crpName, allMemberClusters) + + // Remove all the clusterStagedUpdateRuns. + for _, name := range updateRunNames { + ensureClusterStagedUpdateRunDeletion(name) + } + + // Delete the clusterStagedUpdateStrategy. + ensureClusterUpdateRunStrategyDeletion(strategyName) + }) + + It("Should successfully schedule the crp", func() { + validateLatestClusterSchedulingPolicySnapshot(crpName, policySnapshotIndex1st, 2) + }) + + It("Should create a cluster staged update run successfully", func() { + createClusterStagedUpdateRunWithAutoCreatedSnapshot(updateRunNames[0], crpName, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-1 and member-cluster-2 and complete the cluster staged update run without the delete stage tasks as there is no cluster to delete", func() { + csurSucceededActual := testutilsupdaterun.ClusterStagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[0], resourceSnapshotIndex1st, policySnapshotIndex1st, 2, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[0], allMemberClusterNames[1]}}, nil, nil, nil, true) + Eventually(csurSucceededActual, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s succeeded", updateRunNames[0]) + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + }) + + It("Update the crp to only keep member-cluster-3", func() { + Eventually(func() error { + crp := &placementv1beta1.ClusterResourcePlacement{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: crpName}, crp); err != nil { + return fmt.Errorf("failed to get the crp: %w", err) + } + crp.Spec.Policy.ClusterNames = []string{allMemberClusterNames[2]} + return hubClient.Update(ctx, crp) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the crp to only keep member-cluster-3") + }) + + It("Should successfully schedule the crp", func() { + validateLatestClusterSchedulingPolicySnapshot(crpName, policySnapshotIndex2nd, 1) + }) + + It("Should create a cluster staged update run successfully", func() { + createClusterStagedUpdateRunSucceed(updateRunNames[1], crpName, resourceSnapshotIndex1st, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-3 and keep them on member-cluster-1 and member-cluster-2 until the delete stage tasks complete", func() { + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[2]}) + + By("Validating the approval request of the delete stage is created") + Eventually(func() error { + appReq := &placementv1beta1.ClusterApprovalRequest{} + return hubClient.Get(ctx, client.ObjectKey{Name: fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunNames[1], placementv1beta1.UpdateRunDeleteStageTaskName)}, appReq) + }, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the approval request of the delete stage") + + By("Validating the resources are kept while the delete stage waits for the approval") + checkIfPlacedWorkResourcesOnMemberClustersConsistently(allMemberClusters) + }) + + It("Should remove resources on member-cluster-1 and member-cluster-2 after approval and complete the cluster staged update run successfully", func() { + validateAndApproveClusterApprovalRequests(updateRunNames[1], placementv1beta1.UpdateRunDeleteStageTaskName, placementv1beta1.BeforeStageApprovalTaskNameFmt, placementv1beta1.BeforeStageTaskLabelValue) + + csurSucceededActual := testutilsupdaterun.ClusterStagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[1], resourceSnapshotIndex1st, policySnapshotIndex2nd, 1, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[2]}}, []string{allMemberClusterNames[0], allMemberClusterNames[1]}, nil, nil, true) + Eventually(csurSucceededActual, 2*updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s succeeded", updateRunNames[1]) + checkIfRemovedWorkResourcesFromMemberClusters([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + checkIfPlacedWorkResourcesOnMemberClustersConsistently([]*framework.Cluster{allMemberClusters[2]}) + }) + + It("Should update crp status as completed with member-cluster-3 only", func() { + crpStatusUpdatedActual := crpStatusWithExternalStrategyActual(workResourceIdentifiers(), resourceSnapshotIndex1st, true, []string{allMemberClusterNames[2]}, []string{resourceSnapshotIndex1st}, []bool{true}, nil, nil) + Eventually(crpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep CRP %s status as expected", crpName) + }) + }) + Context("Test cluster scale out and shrink using pickN policy with staged update run", Ordered, func() { var strategy *placementv1beta1.ClusterStagedUpdateStrategy updateRunNames := []string{} @@ -1996,6 +2113,27 @@ var _ = Describe("Test member cluster join and leave flow with updateRun", Label }) }) +// updateStrategySpecWithDeleteStageTasks returns an update strategy that updates all the clusters in one stage +// and waits for an approval before the delete stage. +func updateStrategySpecWithDeleteStageTasks() placementv1beta1.UpdateStrategySpec { + return placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{ + { + Name: "all", + LabelSelector: &metav1.LabelSelector{}, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }, + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + }, + } +} + func createClusterStagedUpdateStrategySucceed(strategyName string) *placementv1beta1.ClusterStagedUpdateStrategy { strategy := &placementv1beta1.ClusterStagedUpdateStrategy{ ObjectMeta: metav1.ObjectMeta{ diff --git a/test/e2e/staged_updaterun_test.go b/test/e2e/staged_updaterun_test.go index 31fe9a123..63506269d 100644 --- a/test/e2e/staged_updaterun_test.go +++ b/test/e2e/staged_updaterun_test.go @@ -687,6 +687,126 @@ var _ = Describe("test RP rollout with staged update run", Label("resourceplacem }) }) + Context("Test cluster shrink with before-stage tasks for the delete stage", Ordered, func() { + var strategy *placementv1beta1.StagedUpdateStrategy + updateRunNames := []string{} + + It("should wait for namespace collection to sync on all member clusters", func() { + waitForNamespaceCollectionOnClusters(appNamespace().Name, allMemberClusterNames) + }) + + BeforeAll(func() { + // Create the RP with external rollout strategy and pick fixed policy. + rp := &placementv1beta1.ResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{ + Name: rpName, + Namespace: testNamespace, + // Add a custom finalizer; this would allow us to better observe + // the behavior of the controllers. + Finalizers: []string{customDeletionBlockerFinalizer}, + }, + Spec: placementv1beta1.PlacementSpec{ + ResourceSelectors: configMapSelector(), + Policy: &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{allMemberClusterNames[0], allMemberClusterNames[1]}, // member-cluster-1 and member-cluster-2 + }, + Strategy: placementv1beta1.RolloutStrategy{ + Type: placementv1beta1.ExternalRolloutStrategyType, + }, + }, + } + Expect(hubClient.Create(ctx, rp)).To(Succeed(), "Failed to create RP") + + // Create the stagedUpdateStrategy with a single stage and a gated delete stage. + strategy = &placementv1beta1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: strategyName, + Namespace: testNamespace, + }, + Spec: updateStrategySpecWithDeleteStageTasks(), + } + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create StagedUpdateStrategy") + + for i := 0; i < 2; i++ { + updateRunNames = append(updateRunNames, fmt.Sprintf(stagedUpdateRunNameWithSubIndexTemplate, GinkgoParallelProcess(), i)) + } + }) + + AfterAll(func() { + // Remove the custom deletion blocker finalizer from the RP. + ensureRPAndRelatedResourcesDeleted(types.NamespacedName{Name: rpName, Namespace: testNamespace}, allMemberClusters) + + // Remove all the stagedUpdateRuns. + for _, name := range updateRunNames { + ensureStagedUpdateRunDeletion(name, testNamespace) + } + + // Delete the stagedUpdateStrategy. + ensureStagedUpdateRunStrategyDeletion(strategyName, testNamespace) + }) + + It("Should successfully schedule the rp", func() { + validateLatestSchedulingPolicySnapshot(rpName, testNamespace, policySnapshotIndex1st, 2) + }) + + It("Should create a staged update run successfully", func() { + createStagedUpdateRunWithAutoCreatedSnapshot(updateRunNames[0], testNamespace, rpName, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-1 and member-cluster-2 and complete the staged update run without the delete stage tasks as there is no cluster to delete", func() { + surSucceededActual := testutilsupdaterun.StagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[0], testNamespace, resourceSnapshotIndex1st, policySnapshotIndex1st, 2, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[0], allMemberClusterNames[1]}}, nil, nil, nil, true) + Eventually(surSucceededActual, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s/%s succeeded", testNamespace, updateRunNames[0]) + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + }) + + It("Update the rp to only keep member-cluster-3", func() { + Eventually(func() error { + rp := &placementv1beta1.ResourcePlacement{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: rpName, Namespace: testNamespace}, rp); err != nil { + return fmt.Errorf("failed to get the rp: %w", err) + } + rp.Spec.Policy.ClusterNames = []string{allMemberClusterNames[2]} + return hubClient.Update(ctx, rp) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the rp to only keep member-cluster-3") + }) + + It("Should successfully schedule the rp", func() { + validateLatestSchedulingPolicySnapshot(rpName, testNamespace, policySnapshotIndex2nd, 1) + }) + + It("Should create a staged update run successfully", func() { + createStagedUpdateRunSucceed(updateRunNames[1], testNamespace, rpName, resourceSnapshotIndex1st, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-3 and keep them on member-cluster-1 and member-cluster-2 until the delete stage tasks complete", func() { + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[2]}) + + By("Validating the approval request of the delete stage is created") + Eventually(func() error { + appReq := &placementv1beta1.ApprovalRequest{} + return hubClient.Get(ctx, client.ObjectKey{Name: fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunNames[1], placementv1beta1.UpdateRunDeleteStageTaskName), Namespace: testNamespace}, appReq) + }, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the approval request of the delete stage") + + By("Validating the resources are kept while the delete stage waits for the approval") + checkIfPlacedWorkResourcesOnMemberClustersConsistently(allMemberClusters) + }) + + It("Should remove resources on member-cluster-1 and member-cluster-2 after approval and complete the staged update run successfully", func() { + validateAndApproveNamespacedApprovalRequests(updateRunNames[1], testNamespace, placementv1beta1.UpdateRunDeleteStageTaskName, placementv1beta1.BeforeStageApprovalTaskNameFmt, placementv1beta1.BeforeStageTaskLabelValue) + + surSucceededActual := testutilsupdaterun.StagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[1], testNamespace, resourceSnapshotIndex1st, policySnapshotIndex2nd, 1, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[2]}}, []string{allMemberClusterNames[0], allMemberClusterNames[1]}, nil, nil, true) + Eventually(surSucceededActual, 2*updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s/%s succeeded", testNamespace, updateRunNames[1]) + checkIfRemovedConfigMapFromMemberClusters([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + checkIfPlacedWorkResourcesOnMemberClustersConsistently([]*framework.Cluster{allMemberClusters[2]}) + }) + + It("Should update rp status as completed with member-cluster-3 only", func() { + rpStatusUpdatedActual := rpStatusWithExternalStrategyActual(appConfigMapIdentifiers(), resourceSnapshotIndex1st, true, []string{allMemberClusterNames[2]}, []string{resourceSnapshotIndex1st}, []bool{true}, nil, nil) + Eventually(rpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep RP %s/%s status as expected", testNamespace, rpName) + }) + }) + Context("Test cluster scale out and shrink using pickN policy with namespaced staged update run", Ordered, func() { var strategy *placementv1beta1.StagedUpdateStrategy updateRunNames := []string{} diff --git a/test/utils/updaterun/updaterun_status.go b/test/utils/updaterun/updaterun_status.go index 09f749a82..b4fe15ac8 100644 --- a/test/utils/updaterun/updaterun_status.go +++ b/test/utils/updaterun/updaterun_status.go @@ -75,11 +75,11 @@ func ClusterStagedUpdateRunStatusSucceededActual( if execute { wantStatus.StagesStatus = buildStageUpdatingStatuses(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunSucceedConditions(updateRun.Generation) } else { wantStatus.StagesStatus = buildStageUpdatingStatusesForInitialized(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunInitializedConditions(updateRun.Generation) } if diff := cmp.Diff(updateRun.Status, wantStatus, updateRunStatusCmpOption...); diff != "" { @@ -120,11 +120,11 @@ func StagedUpdateRunStatusSucceededActual( if execute { wantStatus.StagesStatus = buildStageUpdatingStatuses(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunSucceedConditions(updateRun.Generation) } else { wantStatus.StagesStatus = buildStageUpdatingStatusesForInitialized(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunInitializedConditions(updateRun.Generation) } if diff := cmp.Diff(updateRun.Status, wantStatus, updateRunStatusCmpOption...); diff != "" { @@ -150,20 +150,8 @@ func buildStageUpdatingStatusesForInitialized( stagesStatus[i].Clusters[j].ClusterResourceOverrideSnapshots = wantCROs[wantSelectedClusters[i][j]] stagesStatus[i].Clusters[j].ResourceOverrideSnapshots = wantROs[wantSelectedClusters[i][j]] } - stagesStatus[i].BeforeStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.BeforeStageTasks)) - for j, task := range stage.BeforeStageTasks { - stagesStatus[i].BeforeStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].BeforeStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } - stagesStatus[i].AfterStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.AfterStageTasks)) - for j, task := range stage.AfterStageTasks { - stagesStatus[i].AfterStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].AfterStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } + stagesStatus[i].BeforeStageTaskStatus = buildStageTaskStatuses(stage.BeforeStageTasks, placementv1beta1.BeforeStageApprovalTaskNameFmt, stage.Name, updateRun, false) + stagesStatus[i].AfterStageTaskStatus = buildStageTaskStatuses(stage.AfterStageTasks, placementv1beta1.AfterStageApprovalTaskNameFmt, stage.Name, updateRun, false) } return stagesStatus } @@ -185,28 +173,10 @@ func buildStageUpdatingStatuses( stagesStatus[i].Clusters[j].ResourceOverrideSnapshots = wantROs[wantSelectedClusters[i][j]] stagesStatus[i].Clusters[j].Conditions = updateRunClusterRolloutSucceedConditions(updateRun.GetGeneration()) } - stagesStatus[i].BeforeStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.BeforeStageTasks)) - for j, task := range stage.BeforeStageTasks { - stagesStatus[i].BeforeStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].BeforeStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - // Skip populating task conditions if the stage has 0 clusters (stage is skipped). - if len(wantSelectedClusters[i]) > 0 { - stagesStatus[i].BeforeStageTaskStatus[j].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), task.Type) - } - } - stagesStatus[i].AfterStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.AfterStageTasks)) - for j, task := range stage.AfterStageTasks { - stagesStatus[i].AfterStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].AfterStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - // Skip populating task conditions if the stage has 0 clusters (stage is skipped). - if len(wantSelectedClusters[i]) > 0 { - stagesStatus[i].AfterStageTaskStatus[j].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), task.Type) - } - } + // Skip populating task conditions if the stage has 0 clusters (stage is skipped). + stageExecuted := len(wantSelectedClusters[i]) > 0 + stagesStatus[i].BeforeStageTaskStatus = buildStageTaskStatuses(stage.BeforeStageTasks, placementv1beta1.BeforeStageApprovalTaskNameFmt, stage.Name, updateRun, stageExecuted) + stagesStatus[i].AfterStageTaskStatus = buildStageTaskStatuses(stage.AfterStageTasks, placementv1beta1.AfterStageApprovalTaskNameFmt, stage.Name, updateRun, stageExecuted) // Use skipped conditions if the stage has 0 clusters. if len(wantSelectedClusters[i]) == 0 { stagesStatus[i].Conditions = updateRunStageSkippedNoClustersConditions(updateRun.GetGeneration()) @@ -217,21 +187,53 @@ func buildStageUpdatingStatuses( return stagesStatus } +// buildStageTaskStatuses builds the statuses of the before or after stage tasks of a stage, which have their +// conditions set if the tasks are completed. +func buildStageTaskStatuses( + tasks []placementv1beta1.StageTask, + approvalRequestNameFmt, stageName string, + updateRun placementv1beta1.UpdateRunObj, + completed bool, +) []placementv1beta1.StageTaskStatus { + taskStatuses := make([]placementv1beta1.StageTaskStatus, len(tasks)) + for i, task := range tasks { + taskStatuses[i].Type = task.Type + if task.Type == placementv1beta1.StageTaskTypeApproval { + taskStatuses[i].ApprovalRequestName = fmt.Sprintf(approvalRequestNameFmt, updateRun.GetName(), stageName) + } + if completed { + taskStatuses[i].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), task.Type) + } + } + return taskStatuses +} + func buildDeletionStageStatus( + wantStrategySpec *placementv1beta1.UpdateStrategySpec, wantUnscheduledClusters []string, updateRun placementv1beta1.UpdateRunObj, ) *placementv1beta1.StageUpdatingStatus { - deleteStageStatus := buildDeletionStatusWithoutConditions(wantUnscheduledClusters, updateRun) + deleteStageStatus := buildDeletionStatusWithoutConditions(wantStrategySpec, wantUnscheduledClusters, updateRun) deleteStageStatus.Conditions = updateRunStageRolloutSucceedConditions(updateRun.GetGeneration()) + // The tasks are skipped if the delete stage has 0 clusters. + if len(wantUnscheduledClusters) > 0 { + for i := range deleteStageStatus.BeforeStageTaskStatus { + deleteStageStatus.BeforeStageTaskStatus[i].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), deleteStageStatus.BeforeStageTaskStatus[i].Type) + } + } return deleteStageStatus } func buildDeletionStatusWithoutConditions( + wantStrategySpec *placementv1beta1.UpdateStrategySpec, wantUnscheduledClusters []string, updateRun placementv1beta1.UpdateRunObj, ) *placementv1beta1.StageUpdatingStatus { deleteStageStatus := &placementv1beta1.StageUpdatingStatus{ - StageName: "kubernetes-fleet.io/deleteStage", + StageName: placementv1beta1.UpdateRunDeleteStageName, + } + if wantStrategySpec.DeleteStage != nil { + deleteStageStatus.BeforeStageTaskStatus = buildStageTaskStatuses(wantStrategySpec.DeleteStage.BeforeStageTasks, placementv1beta1.BeforeStageApprovalTaskNameFmt, placementv1beta1.UpdateRunDeleteStageTaskName, updateRun, false) } deleteStageStatus.Clusters = make([]placementv1beta1.ClusterUpdatingStatus, len(wantUnscheduledClusters)) for i := range deleteStageStatus.Clusters { From 0d770fe5c4476f0b10153c6c5b6bb89231a46714 Mon Sep 17 00:00:00 2001 From: Copilot <198982749+Copilot@users.noreply.github.com> Date: Thu, 8 Oct 2026 15:45:28 -0700 Subject: [PATCH 15/20] fix: update distroless base for OpenSSL CVEs (#963) * Initial plan * fix: update distroless base image for OpenSSL CVEs Co-authored-by: britaniar <145056127+britaniar@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: britaniar <145056127+britaniar@users.noreply.github.com> --- docker/hub-agent.Dockerfile | 2 +- docker/member-agent.Dockerfile | 2 +- docker/refresh-token.Dockerfile | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docker/hub-agent.Dockerfile b/docker/hub-agent.Dockerfile index 62b97b822..3441e1d1b 100644 --- a/docker/hub-agent.Dockerfile +++ b/docker/hub-agent.Dockerfile @@ -62,7 +62,7 @@ RUN echo "Building hubagent with GOOS=${TARGETOS} GOARCH=${TARGETARCH} CC=$(read # The pinned digest must reference a multi-arch image index so BuildKit can # resolve the matching base layer for each target architecture. # Refer to https://github.com/GoogleContainerTools/distroless for more details -FROM gcr.io/distroless/base:nonroot@sha256:0896741ba5bafd3ac87ea025a5f578952f2d238ddc3614cb368acc983a687aa2 +FROM gcr.io/distroless/base:nonroot@sha256:a0d70d6a97cd697d9362bc2aae4a6560dd65817e365d0043b07325a97975dc91 WORKDIR / COPY --link --from=builder /workspace/hubagent . USER 65532:65532 diff --git a/docker/member-agent.Dockerfile b/docker/member-agent.Dockerfile index 16ee9044e..00c7807a1 100644 --- a/docker/member-agent.Dockerfile +++ b/docker/member-agent.Dockerfile @@ -62,7 +62,7 @@ RUN echo "Building memberagent with GOOS=${TARGETOS} GOARCH=${TARGETARCH} CC=$(r # The pinned digest must reference a multi-arch image index so BuildKit can # resolve the matching base layer for each target architecture. # Refer to https://github.com/GoogleContainerTools/distroless for more details -FROM gcr.io/distroless/base:nonroot@sha256:0896741ba5bafd3ac87ea025a5f578952f2d238ddc3614cb368acc983a687aa2 +FROM gcr.io/distroless/base:nonroot@sha256:a0d70d6a97cd697d9362bc2aae4a6560dd65817e365d0043b07325a97975dc91 WORKDIR / COPY --link --from=builder /workspace/memberagent . USER 65532:65532 diff --git a/docker/refresh-token.Dockerfile b/docker/refresh-token.Dockerfile index 500ba8048..74e76ecaa 100644 --- a/docker/refresh-token.Dockerfile +++ b/docker/refresh-token.Dockerfile @@ -63,7 +63,7 @@ RUN echo "Building refreshtoken with GOOS=${TARGETOS} GOARCH=${TARGETARCH} CC=$( # The pinned digest must reference a multi-arch image index so BuildKit can # resolve the matching base layer for each target architecture. # Refer to https://github.com/GoogleContainerTools/distroless for more details -FROM gcr.io/distroless/base:nonroot@sha256:0896741ba5bafd3ac87ea025a5f578952f2d238ddc3614cb368acc983a687aa2 +FROM gcr.io/distroless/base:nonroot@sha256:a0d70d6a97cd697d9362bc2aae4a6560dd65817e365d0043b07325a97975dc91 WORKDIR / COPY --link --from=builder /workspace/refreshtoken . USER 65532:65532 From 58812a261108853da1e43fdb8e5c953be33ec726 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 10 Oct 2026 00:20:30 +1100 Subject: [PATCH 16/20] chore: bump step-security/harden-runner from 2.21.1 to 2.22.0 (#958) --- .github/workflows/codespell.yml | 2 +- .github/workflows/release.yml | 2 +- .github/workflows/workflow-lint.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml index 7991f3002..92243e482 100644 --- a/.github/workflows/codespell.yml +++ b/.github/workflows/codespell.yml @@ -12,7 +12,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 + uses: step-security/harden-runner@351661ca32ac09a36dc5ee2d536e3128f2a3c8ed # v2.22.0 with: egress-policy: audit diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 12176fce5..1112ae14e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -252,7 +252,7 @@ jobs: # follow-up issue). Auditing egress at least records what it reaches out # to until that action is replaced. - name: Harden Runner - uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 + uses: step-security/harden-runner@351661ca32ac09a36dc5ee2d536e3128f2a3c8ed # v2.22.0 with: egress-policy: audit diff --git a/.github/workflows/workflow-lint.yml b/.github/workflows/workflow-lint.yml index 903630348..b4ef2b78d 100644 --- a/.github/workflows/workflow-lint.yml +++ b/.github/workflows/workflow-lint.yml @@ -24,7 +24,7 @@ jobs: timeout-minutes: 10 steps: - name: Harden Runner - uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 + uses: step-security/harden-runner@351661ca32ac09a36dc5ee2d536e3128f2a3c8ed # v2.22.0 with: egress-policy: audit From d059a7c8d4cf23c77e74b559ba0ea338f233c260 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:55:31 +1100 Subject: [PATCH 17/20] chore: bump thehanimo/pr-title-checker from 1.4.3 to 2.0.0 (#956) --- .github/workflows/pr-title-lint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pr-title-lint.yml b/.github/workflows/pr-title-lint.yml index 1178161c0..542872ded 100644 --- a/.github/workflows/pr-title-lint.yml +++ b/.github/workflows/pr-title-lint.yml @@ -17,7 +17,7 @@ jobs: check: runs-on: ubuntu-latest steps: - - uses: thehanimo/pr-title-checker@7fbfe05602bdd86f926d3fb3bccb6f3aed43bc70 # v1.4.3 + - uses: thehanimo/pr-title-checker@b69bc3a841cf4e3cfef2d822625d8c90a588e9da # v2.0.0 with: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} pass_on_octokit_error: true From 74427b81c0e757235a24551b9ebf67307dac4f1b Mon Sep 17 00:00:00 2001 From: Chen Yu Date: Mon, 12 Oct 2026 21:16:19 +0800 Subject: [PATCH 18/20] test: [FEP-0001] integration tests for the new work generator (with minor adjustments) (#943) --- .../controllers/workgenerator/controller.go | 30 +- .../controller_integration_test.go | 4197 +++++++++++++++++ pkg/v1/controllers/workgenerator/status.go | 13 + .../controllers/workgenerator/suite_test.go | 144 + .../controllers/workgenerator/utils_test.go | 859 ++++ pkg/v1/controllers/workgenerator/works.go | 36 +- 6 files changed, 5253 insertions(+), 26 deletions(-) create mode 100644 pkg/v1/controllers/workgenerator/controller_integration_test.go create mode 100644 pkg/v1/controllers/workgenerator/suite_test.go create mode 100644 pkg/v1/controllers/workgenerator/utils_test.go diff --git a/pkg/v1/controllers/workgenerator/controller.go b/pkg/v1/controllers/workgenerator/controller.go index 1c27336a6..7c30bfa40 100644 --- a/pkg/v1/controllers/workgenerator/controller.go +++ b/pkg/v1/controllers/workgenerator/controller.go @@ -112,7 +112,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{}, wrappedErr } - // Retrieve the Work objects owned by the placement binding. + // Retrieve the work objects owned by the placement binding. works, err := r.listWorksByOwnerBinding(ctx, placementBindingSpec.ClusterName, placementBinding.GetNamespace(), placementBinding.GetName()) if err != nil { wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), @@ -121,8 +121,8 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{}, wrappedErr } - // Check if the Work objects are consistent with the assigned primary and secondary placement resource snapshots. - // If so, no need to update the spec of the Work objects; just sync the status back to the placement binding + // Check if the work objects are consistent with the assigned primary and secondary placement resource snapshots. + // If so, no need to update the spec of the work objects; just sync the status back to the placement binding // instead. // // Note (chenyu1): this check is intended as a shortcut to avoid constant re-generation and validation of @@ -152,8 +152,8 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{}, nil } - // The Work objects are absent or not up-to-date. Retrieve the placement resource snapshots and create/update - // the Work objects accordingly. + // The work objects are absent or not up-to-date. Retrieve the placement resource snapshots and create/update + // the work objects accordingly. // Retrieve the assigned primary and secondary placement resource snapshots referenced by the placement binding. placementResourceSnapshots, err := r.retrievePrimaryAndSecondaryPlacementResourceSnapshots(ctx, placementBinding) @@ -167,7 +167,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu } // Create or update the work objects. - createdOrUpdatedWorks, writtenToStorage, err := r.refreshWorks(ctx, placementBinding, placementResourceSnapshots, works) + createdOrUpdatedWorks, workSpecsChanged, err := r.refreshWorks(ctx, placementBinding, placementResourceSnapshots, works) if err != nil { wrappedErr := errors.Wraps(err, "failed to refresh work objects", "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, @@ -189,16 +189,18 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu // The work objects have been refreshed. Normally the controller needs only to wait for the work objects // to be processed by the KubeFleet member agent, then refresh the placement binding status upon receiving - // create/update events from the work objects, and there is no need to requeue manually. However, there exists - // a corner case in which a rollout attempt does not involve any change in the work objects; in this case there - // will not be any change events from the work objects and the work generator needs to requeue manually to - // have the placement binding status refreshed. - if !writtenToStorage { - // The work objects have not been created or updated; requeue manually. + // create/status update events from the work objects, and there is no need to requeue manually. However, there + // exists a corner case in which a rollout attempt does not involve any spec change in the work objects (e.g., + // the new placement resource snapshot has the same content as the old one, and the work objects are only + // re-linked via metadata changes); in this case the member agent will not re-process the work objects, there + // will not be any status change events, and the work generator needs to requeue manually to have the placement + // binding status refreshed. + if !workSpecsChanged { + // No work object has been created or has had its spec changed; requeue manually. return ctrl.Result{RequeueAfter: 1 * time.Second}, nil } - // The work objects have been created or updated; wait for change events from the work objects to refresh - // the placement binding status. + // Some work objects have been created or have had their specs changed; wait for change events from the work + // objects to refresh the placement binding status. return ctrl.Result{}, nil } diff --git a/pkg/v1/controllers/workgenerator/controller_integration_test.go b/pkg/v1/controllers/workgenerator/controller_integration_test.go new file mode 100644 index 000000000..d82529dfb --- /dev/null +++ b/pkg/v1/controllers/workgenerator/controller_integration_test.go @@ -0,0 +1,4197 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package workgenerator contains the controller logic for reconciling placement binding objects. +package workgenerator + +import ( + "fmt" + "strconv" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils" +) + +var _ = Describe("reconciling placement bindings (single placement resource snapshot)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(appNamespaceName, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + nsIdentifier := placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + } + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + oldManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + newManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + oldManifestIdentifiers[0], + { + Ordinal: 1, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + oldSnapshot = createPlacementResourceSnapshot(appNamespaceName, oldSnapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, oldManifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new placement resource snapshot", func() { + // Prepare a NS object (unchanged). + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a ConfigMap object (new); the Deployment object is dropped. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new placement resource snapshot with the NS and the new ConfigMap objects. + newSnapshot = createPlacementResourceSnapshot(appNamespaceName, newSnapshotName, placementPolicyName, 1, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Point the placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, newManifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new placement resource snapshot") + }) + }) + + Context("refresh binding status upon work status changes", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(appNamespaceName, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the deployment (and consequently the work object) as unavailable", func() { + markWorkManifestAsUnavailable(workNS, workName, 1) + }) + + It("should refresh the placement binding status", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(2)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + var resources []placementv1alpha1.SnapshottedResource + var oldBindingGeneration int64 + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + resources = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + } + oldSnapshot = createPlacementResourceSnapshot(appNamespaceName, oldSnapshotName, placementPolicyName, 0, resources...) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new placement resource snapshot with the same content", func() { + // Record the current generation of the placement binding. + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create a new placement resource snapshot with exactly the same resources. + newSnapshot = createPlacementResourceSnapshot(appNamespaceName, newSnapshotName, placementPolicyName, 1, resources...) + + // Point the placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary placement resource snapshot changes. + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the placement binding + // should be reported as synchronized and available, with the conditions observing the new generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new placement resource snapshot") + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + // The diff details reported for the ConfigMap object, as if the member agent has failed to take over an + // existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamp is truncated to the second, as this is the precision kept by the API server. + diffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a placement resource snapshot with the NS, the Deployment, and the ConfigMap objects. + snapshot = createPlacementResourceSnapshot(appNamespaceName, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the deployment as unavailable, and the config map as failed to be applied (with diff details)", func() { + markWorkManifestsAsFailed(workNS, workName, manifestIdentifiers, + map[int]*placementv1alpha1.DiffDetails{2: diffDetails}, + sets.New(1), + ) + }) + + It("should refresh the placement binding status with the failure info", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: configMapIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: diffDetails, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + }) + }) +}) + +var _ = Describe("reconciling placement bindings (multiple placement resource snapshots)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of the same index. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot (and consequently each work object) features its own set of differently named NS, + // Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + } + + BeforeAll(func() { + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(snapshotNames)) + for subIdx := range snapshotNames { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsNames[subIdx] + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsNames[subIdx] + regularDeploy.Name = deployNames[subIdx] + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsNames[subIdx] + regularConfigMap.Name = configMapNames[subIdx] + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + }, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + } + } + + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the primary work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should not refresh the placement binding status until all work objects have been processed", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Consistently(statusUpdatedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has been refreshed prematurely") + }) + + It("can mark the secondary work objects as applied and available", func() { + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually. + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for subIdx := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotNames[subIdx]) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of + // index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + // After the first rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of + // index 1. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // After the second rollout: the primary snapshot (sub-index 0) and three secondary snapshots (sub-indices 1, + // 2, and 3) of index 2. + latestSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 3), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from; work objects derived + // from snapshots of the same sub-index share the same name across snapshot indices. + workNames := make([]string, len(latestSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + latestSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(latestSnapshotNames)) + + // The UIDs of the work objects that should be kept (updated rather than re-created) across the rollout. + var primaryWorkUID, secondaryWorkUID types.UID + var ownerRef *metav1.OwnerReference + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(latestSnapshotNames)) + deployNames := make([]string, len(latestSnapshotNames)) + configMapNames := make([]string, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + } + + // Before the rollouts, each snapshot features its own set of NS, Deployment, and ConfigMap objects. + oldManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + oldManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // After the first rollout, the snapshot of sub-index 2 is gone; its Deployment and ConfigMap objects are moved to the + // snapshots of sub-index 0 and 1 respectively, and its NS object is dropped. + newManifestIdentifiersPerWork := [][]placementv1alpha1.ManifestIdentifier{ + { + nsManifestIdentifier(0, nsNames[0]), + deployManifestIdentifier(1, nsNames[0], deployNames[0]), + configMapManifestIdentifier(2, nsNames[0], configMapNames[0]), + deployManifestIdentifier(3, nsNames[2], deployNames[2]), + }, + { + nsManifestIdentifier(0, nsNames[1]), + deployManifestIdentifier(1, nsNames[1], deployNames[1]), + configMapManifestIdentifier(2, nsNames[1], configMapNames[1]), + configMapManifestIdentifier(3, nsNames[2], configMapNames[2]), + }, + } + + // After the second rollout, each snapshot again features its own set of NS, Deployment, and ConfigMap + // objects. + latestManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + latestManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range latestSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the secondary work object that should be kept across the rollout. + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + secondaryWorkUID = secondaryWork.UID + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], oldManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new set of placement resource snapshots", func() { + // Create the new placement resource snapshots; the Deployment and ConfigMap objects from the old snapshot + // of sub-index 2 are moved to the new snapshots of sub-index 0 and 1 respectively. + newSnapshots[1] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, newSnapshotNames[1], placementPolicyName, + 1, 1, len(newSnapshotNames), + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + configMapSnapshottedResource(nsNames[2], configMapNames[2]), + ) + newSnapshots[0] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, newSnapshotNames[0], placementPolicyName, + 1, 0, len(newSnapshotNames), + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + deploySnapshottedResource(nsNames[2], deployNames[2]), + ) + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The work object derived from the snapshot of sub-index 2 should be gone; the other two should be + // updated in place, with the secondary one still owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should delete the work object that is no longer needed", func() { + workRemovedActual := workRemovedActual(workNS, workNames[2]) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to delete the work object that is no longer needed") + }) + + It("should update the other work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range newSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], newManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out another new set of placement resource snapshots", func() { + // Create the latest placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(latestSnapshotNames) - 1; subIdx >= 0; subIdx-- { + latestSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, latestSnapshotNames[subIdx], placementPolicyName, + 2, subIdx, len(latestSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Point the placement binding to the latest primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, latestSnapshotNames[0]) + }) + + It("should update and create the work objects", func() { + // The work objects derived from the snapshots of sub-index 0 and 1 should be updated; the ones derived from + // the snapshots of sub-index 2 and 3 should be created, and owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + latestSnapshotNames[0], subIdx, len(latestSnapshotNames), ownerRef, latestSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the existing work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range latestSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], latestManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append(append([]string{}, oldSnapshotNames...), newSnapshotNames...), latestSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("handle inconsistent resource snapshots", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // Index 0: a single primary snapshot. + snapshotNameIdx0 := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + // Index 1: a primary snapshot that expects one secondary snapshot. + snapshotNamesIdx1 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // Index 2: a primary snapshot that expects two secondary snapshots. + snapshotNamesIdx2 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + } + contentsHashIdx1 := "hash-1" + mismatchedContentsHashIdx1 := "hash-1-mismatched" + contentsHashIdx2 := "hash-2" + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(snapshotNamesIdx2)) + var snapshotIdx0 placementv1alpha1.PlacementResourceSnapshotAccessor + snapshotsIdx2 := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNamesIdx2)) + + var primaryWorkUID types.UID + // The expected state of the work objects and the placement binding status before any rollout + // attempt; neither should change as long as the snapshots in use are inconsistent. + var wantWorksBeforeRollouts []placementv1alpha1.Work + var wantStatusBeforeRollouts *placementv1alpha1.PlacementBindingStatus + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(snapshotNamesIdx2)) + deployNames := make([]string, len(snapshotNamesIdx2)) + configMapNames := make([]string, len(snapshotNamesIdx2)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + bindingAndWorksUnchangedActual := func() error { + if err := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorksBeforeRollouts)(); err != nil { + return err + } + return placementBindingStatusMatchesActual(appNamespaceName, bindingName, wantStatusBeforeRollouts)() + } + + BeforeAll(func() { + // Create a single placement resource snapshot of index 0. + snapshotIdx0 = createPlacementResourceSnapshot(appNamespaceName, snapshotNameIdx0, placementPolicyName, 0, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotNameIdx0) + for subIdx := range snapshotNamesIdx2 { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorksBeforeRollouts = []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workNames[0], appNamespaceName, bindingName, placementPolicyName, snapshotIdx0), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorksBeforeRollouts) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the primary work object, which should be kept across rollouts. + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + + // Record the status (with the current observed generation), which should be kept as long as the + // snapshots in use are inconsistent. + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + wantStatusBeforeRollouts = wantStatus.DeepCopy() + for idx := range wantStatusBeforeRollouts.Conditions { + wantStatusBeforeRollouts.Conditions[idx].ObservedGeneration = binding.GetGeneration() + } + }) + + It("can roll out a new primary snapshot (index 1) without its secondary snapshot", func() { + // The new primary snapshot features new ConfigMap data, and expects one secondary snapshot, which is + // absent for now. + createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx1[0], placementPolicyName, + 1, 0, len(snapshotNamesIdx1), contentsHashIdx1, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue1}), + ) + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, snapshotNamesIdx1[0]) + }) + + It("should not update the placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add a secondary snapshot (index 1) with a mismatched contents hash", func() { + createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx1[1], placementPolicyName, + 1, 1, len(snapshotNamesIdx1), mismatchedContentsHashIdx1, + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + ) + }) + + It("should not update the placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status or the work objects have been updated unexpectedly") + }) + + It("can roll out a new primary snapshot (index 2) without its secondary snapshots", func() { + // The new primary snapshot features new ConfigMap data, and expects two secondary snapshots, which are + // absent for now. + snapshotsIdx2[0] = createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx2[0], placementPolicyName, + 2, 0, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue2}), + ) + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, snapshotNamesIdx2[0]) + }) + + It("should not update the placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add the missing secondary snapshots (index 2) with the expected contents hash", func() { + for subIdx := len(snapshotNamesIdx2) - 1; subIdx >= 1; subIdx-- { + snapshotsIdx2[subIdx] = createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx2[subIdx], placementPolicyName, + 2, subIdx, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + }) + + It("should update and create the work objects", func() { + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: workNames[0], + UID: primaryWorkUID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + wantWorks := make([]placementv1alpha1.Work, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + snapshotNamesIdx2[0], subIdx, len(snapshotNamesIdx2), ownerRef, snapshotsIdx2[subIdx]) + } + + // The work generator does not watch placement resource snapshots; it picks up the newly added secondary + // snapshots only when it retries the failed reconciliation, which is subject to exponential backoff. + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDurationForBackoffRetries, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the primary work object in place (not re-create it)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range snapshotNamesIdx2 { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{snapshotNameIdx0}, snapshotNamesIdx1...), snapshotNamesIdx2...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + // After the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 1, + // with exactly the same content as their counterparts of index 0. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(oldSnapshotNames)) + workUIDs := make([]types.UID, len(oldSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(oldSnapshotNames)) + var ownerRef *metav1.OwnerReference + var oldBindingGeneration int64 + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(oldSnapshotNames)) + deployNames := make([]string, len(oldSnapshotNames)) + configMapNames := make([]string, len(oldSnapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + for subIdx := range oldSnapshotNames { + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + } + } + + // Create the placement resource snapshots of index 0. The secondary snapshot is created first, as the + // placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range oldSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UIDs of the work objects, which should be kept across the rollout. + for subIdx := range oldSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + workUIDs[subIdx] = work.UID + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new set of placement resource snapshots with the same content", func() { + // Record the current generation of the placement binding. + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create the new placement resource snapshots with exactly the same resources as their counterparts. + for subIdx := len(newSnapshotNames) - 1; subIdx >= 0; subIdx-- { + newSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, newSnapshotNames[subIdx], placementPolicyName, + 1, subIdx, len(newSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary placement resource snapshot changes. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should update the work objects in place (not re-create them)", func() { + for subIdx := range newSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + Expect(work.UID).To(Equal(workUIDs[subIdx]), "The work object has been re-created") + } + }) + + It("should refresh the placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the placement binding + // should be reported as synchronized and available, with the conditions observing the new generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{}, oldSnapshotNames...), newSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // The diff details reported for the ConfigMap object in the primary work object, as if the member agent has + // failed to take over an existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamps are truncated to the second, as this is the precision kept by the API server. + configMapDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + // The diff details reported for the Deployment object in the secondary work object, as if the member agent + // has failed to take over an existing Deployment object with a different replica count in the member cluster. + deployDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshot is created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark some manifests in the primary work object as failed", func() { + // The Deployment object fails to become available; the ConfigMap object fails to be applied (with diff + // details). + markWorkManifestsAsFailed(workNS, workNames[0], manifestIdentifiersPerWork[0], + map[int]*placementv1alpha1.DiffDetails{2: configMapDiffDetails}, + sets.New(1), + ) + }) + + It("can mark some manifests in the secondary work object as failed", func() { + // The Deployment object fails to be applied (with diff details); the ConfigMap object fails to become + // available. + markWorkManifestsAsFailed(workNS, workNames[1], manifestIdentifiersPerWork[1], + map[int]*placementv1alpha1.DiffDetails{1: deployDiffDetails}, + sets.New(2), + ) + }) + + It("should refresh the placement binding status with the failure info from all work objects", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + // In each work object, the NS object is applied and available; the Deployment and ConfigMap objects + // have failed in different ways. + SelectedResources: ptr.To(int32(6)), + SynchronizedResources: ptr.To(int32(4)), + AvailableResources: ptr.To(int32(2)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: configMapDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: deployDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotNames[0]), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for _, snapshotName := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) +}) + +var _ = Describe("reconciling cluster placement bindings (single placement resource snapshot)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(clusterScopedNS, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + nsIdentifier := placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + } + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + oldManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + newManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + oldManifestIdentifiers[0], + { + Ordinal: 1, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + oldSnapshot = createPlacementResourceSnapshot(clusterScopedNS, oldSnapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, oldManifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new cluster placement resource snapshot", func() { + // Prepare a NS object (unchanged). + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a ConfigMap object (new); the Deployment object is dropped. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new cluster placement resource snapshot with the NS and the new ConfigMap objects. + newSnapshot = createPlacementResourceSnapshot(clusterScopedNS, newSnapshotName, placementPolicyName, 1, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Point the cluster placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, newManifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old cluster placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new cluster placement resource snapshot") + }) + }) + + Context("refresh binding status upon work status changes", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(clusterScopedNS, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the deployment (and consequently the work object) as unavailable", func() { + markWorkManifestAsUnavailable(workNS, workName, 1) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(2)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + var resources []placementv1alpha1.SnapshottedResource + var oldBindingGeneration int64 + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + resources = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + } + oldSnapshot = createPlacementResourceSnapshot(clusterScopedNS, oldSnapshotName, placementPolicyName, 0, resources...) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new cluster placement resource snapshot with the same content", func() { + // Record the current generation of the cluster placement binding. + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create a new cluster placement resource snapshot with exactly the same resources. + newSnapshot = createPlacementResourceSnapshot(clusterScopedNS, newSnapshotName, placementPolicyName, 1, resources...) + + // Point the cluster placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary cluster placement resource snapshot changes. + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the cluster placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the cluster placement binding + // should be reported as synchronized and available, with the conditions observing the new generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The cluster placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old cluster placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new cluster placement resource snapshot") + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + // The diff details reported for the ConfigMap object, as if the member agent has failed to take over an + // existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamp is truncated to the second, as this is the precision kept by the API server. + diffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a cluster placement resource snapshot with the NS, the Deployment, and the ConfigMap objects. + snapshot = createPlacementResourceSnapshot(clusterScopedNS, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the deployment as unavailable, and the config map as failed to be applied (with diff details)", func() { + markWorkManifestsAsFailed(workNS, workName, manifestIdentifiers, + map[int]*placementv1alpha1.DiffDetails{2: diffDetails}, + sets.New(1), + ) + }) + + It("should refresh the cluster placement binding status with the failure info", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: configMapIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: diffDetails, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + }) + }) +}) + +var _ = Describe("reconciling cluster placement bindings (multiple placement resource snapshots)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of the same index. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot (and consequently each work object) features its own set of differently named NS, + // Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + } + + BeforeAll(func() { + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(snapshotNames)) + for subIdx := range snapshotNames { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsNames[subIdx] + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsNames[subIdx] + regularDeploy.Name = deployNames[subIdx] + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsNames[subIdx] + regularConfigMap.Name = configMapNames[subIdx] + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + }, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + } + } + + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the primary work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should not refresh the cluster placement binding status until all work objects have been processed", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Consistently(statusUpdatedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status has been refreshed prematurely") + }) + + It("can mark the secondary work objects as applied and available", func() { + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually. + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for subIdx := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotNames[subIdx]) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of + // index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + // After the first rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of + // index 1. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // After the second rollout: the primary snapshot (sub-index 0) and three secondary snapshots (sub-indices 1, + // 2, and 3) of index 2. + latestSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 3), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from; work objects derived + // from snapshots of the same sub-index share the same name across snapshot indices. + workNames := make([]string, len(latestSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + latestSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(latestSnapshotNames)) + + // The UIDs of the work objects that should be kept (updated rather than re-created) across the rollout. + var primaryWorkUID, secondaryWorkUID types.UID + var ownerRef *metav1.OwnerReference + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(latestSnapshotNames)) + deployNames := make([]string, len(latestSnapshotNames)) + configMapNames := make([]string, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + } + + // Before the rollouts, each snapshot features its own set of NS, Deployment, and ConfigMap objects. + oldManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + oldManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // After the first rollout, the snapshot of sub-index 2 is gone; its Deployment and ConfigMap objects are moved to the + // snapshots of sub-index 0 and 1 respectively, and its NS object is dropped. + newManifestIdentifiersPerWork := [][]placementv1alpha1.ManifestIdentifier{ + { + nsManifestIdentifier(0, nsNames[0]), + deployManifestIdentifier(1, nsNames[0], deployNames[0]), + configMapManifestIdentifier(2, nsNames[0], configMapNames[0]), + deployManifestIdentifier(3, nsNames[2], deployNames[2]), + }, + { + nsManifestIdentifier(0, nsNames[1]), + deployManifestIdentifier(1, nsNames[1], deployNames[1]), + configMapManifestIdentifier(2, nsNames[1], configMapNames[1]), + configMapManifestIdentifier(3, nsNames[2], configMapNames[2]), + }, + } + + // After the second rollout, each snapshot again features its own set of NS, Deployment, and ConfigMap + // objects. + latestManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + latestManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range latestSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the secondary work object that should be kept across the rollout. + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + secondaryWorkUID = secondaryWork.UID + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], oldManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new set of cluster placement resource snapshots", func() { + // Create the new placement resource snapshots; the Deployment and ConfigMap objects from the old snapshot + // of sub-index 2 are moved to the new snapshots of sub-index 0 and 1 respectively. + newSnapshots[1] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, newSnapshotNames[1], placementPolicyName, + 1, 1, len(newSnapshotNames), + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + configMapSnapshottedResource(nsNames[2], configMapNames[2]), + ) + newSnapshots[0] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, newSnapshotNames[0], placementPolicyName, + 1, 0, len(newSnapshotNames), + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + deploySnapshottedResource(nsNames[2], deployNames[2]), + ) + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The work object derived from the snapshot of sub-index 2 should be gone; the other two should be + // updated in place, with the secondary one still owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should delete the work object that is no longer needed", func() { + workRemovedActual := workRemovedActual(workNS, workNames[2]) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to delete the work object that is no longer needed") + }) + + It("should update the other work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range newSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], newManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out another new set of cluster placement resource snapshots", func() { + // Create the latest placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(latestSnapshotNames) - 1; subIdx >= 0; subIdx-- { + latestSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, latestSnapshotNames[subIdx], placementPolicyName, + 2, subIdx, len(latestSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Point the cluster placement binding to the latest primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, latestSnapshotNames[0]) + }) + + It("should update and create the work objects", func() { + // The work objects derived from the snapshots of sub-index 0 and 1 should be updated; the ones derived from + // the snapshots of sub-index 2 and 3 should be created, and owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + latestSnapshotNames[0], subIdx, len(latestSnapshotNames), ownerRef, latestSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the existing work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range latestSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], latestManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append(append([]string{}, oldSnapshotNames...), newSnapshotNames...), latestSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("handle inconsistent resource snapshots", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // Index 0: a single primary snapshot. + snapshotNameIdx0 := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + // Index 1: a primary snapshot that expects one secondary snapshot. + snapshotNamesIdx1 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // Index 2: a primary snapshot that expects two secondary snapshots. + snapshotNamesIdx2 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + } + contentsHashIdx1 := "hash-1" + mismatchedContentsHashIdx1 := "hash-1-mismatched" + contentsHashIdx2 := "hash-2" + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(snapshotNamesIdx2)) + var snapshotIdx0 placementv1alpha1.PlacementResourceSnapshotAccessor + snapshotsIdx2 := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNamesIdx2)) + + var primaryWorkUID types.UID + // The expected state of the work objects and the cluster placement binding status before any rollout + // attempt; neither should change as long as the snapshots in use are inconsistent. + var wantWorksBeforeRollouts []placementv1alpha1.Work + var wantStatusBeforeRollouts *placementv1alpha1.PlacementBindingStatus + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(snapshotNamesIdx2)) + deployNames := make([]string, len(snapshotNamesIdx2)) + configMapNames := make([]string, len(snapshotNamesIdx2)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + bindingAndWorksUnchangedActual := func() error { + if err := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorksBeforeRollouts)(); err != nil { + return err + } + return placementBindingStatusMatchesActual(clusterScopedNS, bindingName, wantStatusBeforeRollouts)() + } + + BeforeAll(func() { + // Create a single placement resource snapshot of index 0. + snapshotIdx0 = createPlacementResourceSnapshot(clusterScopedNS, snapshotNameIdx0, placementPolicyName, 0, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotNameIdx0) + for subIdx := range snapshotNamesIdx2 { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorksBeforeRollouts = []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workNames[0], clusterScopedNS, bindingName, placementPolicyName, snapshotIdx0), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorksBeforeRollouts) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the primary work object, which should be kept across rollouts. + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + + // Record the status (with the current observed generation), which should be kept as long as the + // snapshots in use are inconsistent. + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + wantStatusBeforeRollouts = wantStatus.DeepCopy() + for idx := range wantStatusBeforeRollouts.Conditions { + wantStatusBeforeRollouts.Conditions[idx].ObservedGeneration = binding.GetGeneration() + } + }) + + It("can roll out a new primary snapshot (index 1) without its secondary snapshot", func() { + // The new primary snapshot features new ConfigMap data, and expects one secondary snapshot, which is + // absent for now. + createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx1[0], placementPolicyName, + 1, 0, len(snapshotNamesIdx1), contentsHashIdx1, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue1}), + ) + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, snapshotNamesIdx1[0]) + }) + + It("should not update the cluster placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add a secondary snapshot (index 1) with a mismatched contents hash", func() { + createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx1[1], placementPolicyName, + 1, 1, len(snapshotNamesIdx1), mismatchedContentsHashIdx1, + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + ) + }) + + It("should not update the cluster placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status or the work objects have been updated unexpectedly") + }) + + It("can roll out a new primary snapshot (index 2) without its secondary snapshots", func() { + // The new primary snapshot features new ConfigMap data, and expects two secondary snapshots, which are + // absent for now. + snapshotsIdx2[0] = createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx2[0], placementPolicyName, + 2, 0, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue2}), + ) + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, snapshotNamesIdx2[0]) + }) + + It("should not update the cluster placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add the missing secondary snapshots (index 2) with the expected contents hash", func() { + for subIdx := len(snapshotNamesIdx2) - 1; subIdx >= 1; subIdx-- { + snapshotsIdx2[subIdx] = createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx2[subIdx], placementPolicyName, + 2, subIdx, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + }) + + It("should update and create the work objects", func() { + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: workNames[0], + UID: primaryWorkUID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + wantWorks := make([]placementv1alpha1.Work, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + snapshotNamesIdx2[0], subIdx, len(snapshotNamesIdx2), ownerRef, snapshotsIdx2[subIdx]) + } + + // The work generator does not watch placement resource snapshots; it picks up the newly added secondary + // snapshots only when it retries the failed reconciliation, which is subject to exponential backoff. + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDurationForBackoffRetries, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the primary work object in place (not re-create it)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range snapshotNamesIdx2 { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{snapshotNameIdx0}, snapshotNamesIdx1...), snapshotNamesIdx2...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + // After the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 1, + // with exactly the same content as their counterparts of index 0. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(oldSnapshotNames)) + workUIDs := make([]types.UID, len(oldSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(oldSnapshotNames)) + var ownerRef *metav1.OwnerReference + var oldBindingGeneration int64 + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(oldSnapshotNames)) + deployNames := make([]string, len(oldSnapshotNames)) + configMapNames := make([]string, len(oldSnapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + for subIdx := range oldSnapshotNames { + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + } + } + + // Create the placement resource snapshots of index 0. The secondary snapshot is created first, as the + // placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range oldSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UIDs of the work objects, which should be kept across the rollout. + for subIdx := range oldSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + workUIDs[subIdx] = work.UID + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new set of cluster placement resource snapshots with the same content", func() { + // Record the current generation of the cluster placement binding. + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create the new placement resource snapshots with exactly the same resources as their counterparts. + for subIdx := len(newSnapshotNames) - 1; subIdx >= 0; subIdx-- { + newSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, newSnapshotNames[subIdx], placementPolicyName, + 1, subIdx, len(newSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary placement resource snapshot changes. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should update the work objects in place (not re-create them)", func() { + for subIdx := range newSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + Expect(work.UID).To(Equal(workUIDs[subIdx]), "The work object has been re-created") + } + }) + + It("should refresh the cluster placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the cluster placement + // binding should be reported as synchronized and available, with the conditions observing the new + // generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The cluster placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{}, oldSnapshotNames...), newSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // The diff details reported for the ConfigMap object in the primary work object, as if the member agent has + // failed to take over an existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamps are truncated to the second, as this is the precision kept by the API server. + configMapDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + // The diff details reported for the Deployment object in the secondary work object, as if the member agent + // has failed to take over an existing Deployment object with a different replica count in the member cluster. + deployDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshot is created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark some manifests in the primary work object as failed", func() { + // The Deployment object fails to become available; the ConfigMap object fails to be applied (with diff + // details). + markWorkManifestsAsFailed(workNS, workNames[0], manifestIdentifiersPerWork[0], + map[int]*placementv1alpha1.DiffDetails{2: configMapDiffDetails}, + sets.New(1), + ) + }) + + It("can mark some manifests in the secondary work object as failed", func() { + // The Deployment object fails to be applied (with diff details); the ConfigMap object fails to become + // available. + markWorkManifestsAsFailed(workNS, workNames[1], manifestIdentifiersPerWork[1], + map[int]*placementv1alpha1.DiffDetails{1: deployDiffDetails}, + sets.New(2), + ) + }) + + It("should refresh the cluster placement binding status with the failure info from all work objects", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + // In each work object, the NS object is applied and available; the Deployment and ConfigMap objects + // have failed in different ways. + SelectedResources: ptr.To(int32(6)), + SynchronizedResources: ptr.To(int32(4)), + AvailableResources: ptr.To(int32(2)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: configMapDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: deployDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotNames[0]), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for _, snapshotName := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) +}) diff --git a/pkg/v1/controllers/workgenerator/status.go b/pkg/v1/controllers/workgenerator/status.go index 78e29c0dd..094fbb718 100644 --- a/pkg/v1/controllers/workgenerator/status.go +++ b/pkg/v1/controllers/workgenerator/status.go @@ -17,7 +17,9 @@ limitations under the License. package workgenerator import ( + "cmp" "context" + "slices" "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/meta" @@ -246,6 +248,17 @@ func countResourcesInWorksByProcessingResults(works []placementv1alpha1.Work) ( } } } + + // Sort the failed resources to ensure deterministic status reporting. + slices.SortFunc(failed, func(a, b placementv1alpha1.FailedResource) int { + return cmp.Or( + cmp.Compare(a.ObjectRef.APIGroup, b.ObjectRef.APIGroup), + cmp.Compare(a.ObjectRef.APIVersion, b.ObjectRef.APIVersion), + cmp.Compare(a.ObjectRef.Kind, b.ObjectRef.Kind), + cmp.Compare(a.ObjectRef.Namespace, b.ObjectRef.Namespace), + cmp.Compare(a.ObjectRef.Name, b.ObjectRef.Name), + ) + }) return total, synced, available, failed } diff --git a/pkg/v1/controllers/workgenerator/suite_test.go b/pkg/v1/controllers/workgenerator/suite_test.go new file mode 100644 index 000000000..84cc2beab --- /dev/null +++ b/pkg/v1/controllers/workgenerator/suite_test.go @@ -0,0 +1,144 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "context" + "flag" + "fmt" + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/manager" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils" + "github.com/kubefleet-dev/kubefleet/pkg/v1/utils/fieldindexers" +) + +const ( + memberCluster1Name = "member-1" + memberCluster2Name = "member-2" + + // The namespace in which namespace-scoped placement objects (e.g., PlacementBinding and + // PlacementResourceSnapshot objects) are created. + appNamespaceName = "app" + + workerCount = 4 + maxConcurrentReconciles = 4 +) + +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubClient client.Client + mgr manager.Manager + + ctx context.Context + cancel context.CancelFunc + + memberClusterNames = []string{memberCluster1Name, memberCluster2Name} +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Work Generator Integration Test Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + By("Bootstrapping test environment") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).Should(Succeed()) + Expect(hubCfg).NotTo(BeNil()) + + By("Setting up the scheme") + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + By("Starting the controller manager") + mgr, err = ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).Should(Succeed()) + + // Use the same (cache-backed) client as the controller to avoid cache delays when verifying results. + By("Setting the hub client to the one used by the controller manager") + hubClient = mgr.GetClient() + + // The field indexes must be set up before the manager starts. + By("Setting up the field indexes") + Expect(fieldindexers.SetupWithHubControllerManager(ctx, mgr)).Should(Succeed()) + + By("Setting up the work generator") + Expect(New(mgr.GetClient(), workerCount).SetupWithManager(mgr, maxConcurrentReconciles)).Should(Succeed()) + + go func() { + defer GinkgoRecover() + Expect(mgr.Start(ctx)).Should(Succeed(), "Failed to run manager") + }() + Expect(mgr.GetCache().WaitForCacheSync(ctx)).Should(BeTrue(), "Failed to sync the manager cache") + + By("Creating the application namespace") + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: appNamespaceName}, + })).Should(Succeed(), "Failed to create the application namespace") + + By("Creating the reserved namespaces for member clusters") + for _, clusterName := range memberClusterNames { + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf(utils.NamespaceNameFormat, clusterName)}, + })).Should(Succeed(), "Failed to create the reserved namespace for member cluster %s", clusterName) + } +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + + By("Tearing down the test environment") + Expect(hubEnv.Stop()).Should(Succeed()) +}) diff --git a/pkg/v1/controllers/workgenerator/utils_test.go b/pkg/v1/controllers/workgenerator/utils_test.go new file mode 100644 index 000000000..ae4d86fa8 --- /dev/null +++ b/pkg/v1/controllers/workgenerator/utils_test.go @@ -0,0 +1,859 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "fmt" + "strconv" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "github.com/kubefleet-dev/kubefleet/apis/kubefleet.dev/placement/v1alpha1" + "github.com/kubefleet-dev/kubefleet/pkg/utils" + testutilsresource "github.com/kubefleet-dev/kubefleet/test/utils/resource" +) + +const ( + placementBindingNameTemplate = "binding-%s" + placementPolicyNameTemplate = "policy-%s" + clusterPlacementBindingNameTemplate = "cluster-binding-%s" + clusterPlacementPolicyNameTemplate = "cluster-policy-%s" + placementResourceSnapshotNameTemplate = "%s-snapshot-%d" + // The name template for secondary placement resource snapshots (those with a non-zero sub-index). + subIndexedPlacementResourceSnapshotNameTemplate = "%s-snapshot-%d-%d" + nsNameTemplate = "ns-%s" + + deployName = "deploy-1" + configMapName = "configmap-1" + + indexedNSNameTemplate = "ns-%d" + deployNameTemplate = "deploy-%d" + configMapNameTemplate = "configmap-%d" + + workKind = "Work" + nsKind = "Namespace" + deployKind = "Deployment" + configMapKind = "ConfigMap" + nginxName = "nginx" + + appsAPIGroup = "apps" + + configMapDataKey = "foo" + configMapDataValue = "bar" + // Alternative ConfigMap data values, used to simulate changes or drifts. + altConfigMapDataValue1 = "baz" + altConfigMapDataValue2 = "qux" + + dummyContentsHash = "dummy-hash" + + nsResource = "namespaces" + deployResource = "deployments" + configMapResource = "configmaps" + + markedAsAppliedReason = "MarkedAsApplied" + markedAsAvailableReason = "MarkedAsAvailable" + markedAsUnavailableReason = "MarkedAsUnavailable" + markedAsFailedToApplyReason = "MarkedAsFailedToApply" +) + +// The namespace of cluster-scoped objects (i.e., none); the test helpers use it to tell cluster-scoped placement +// bindings and placement resource snapshots apart from their namespace-scoped counterparts. +const clusterScopedNS = "" + +const ( + eventuallyDuration = time.Second * 10 + eventuallyInterval = time.Second * 1 + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Millisecond * 500 + + // The work generator does not watch placement resource snapshots; when it fails to process a placement binding + // due to inconsistent snapshots, it only picks up later snapshot changes when it retries the reconciliation, + // which is subject to exponential backoff. Checks that depend on such retries use a longer timeout. + eventuallyDurationForBackoffRetries = time.Minute +) + +var ( + ignoreFieldObjectMetaAutoGenFields = cmpopts.IgnoreFields(metav1.ObjectMeta{}, "CreationTimestamp", "Generation", "ResourceVersion", "SelfLink", "UID", "ManagedFields") + ignoreFieldWorkTypeMetaAndStatus = cmpopts.IgnoreFields(placementv1alpha1.Work{}, "TypeMeta", "Status") + ignoreFieldConditionLTTMsg = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message") + + lessFuncWork = func(a, b placementv1alpha1.Work) bool { + return a.Name < b.Name + } + // The work generator collects failed resources work object by work object, in the order the work objects are + // listed from the cache, which is not deterministic; failed resources are sorted before comparison. + lessFuncFailedResource = func(a, b placementv1alpha1.FailedResource) bool { + keyA := fmt.Sprintf("%s/%s/%s/%s", a.ObjectRef.APIGroup, a.ObjectRef.Kind, a.ObjectRef.Namespace, a.ObjectRef.Name) + keyB := fmt.Sprintf("%s/%s/%s/%s", b.ObjectRef.APIGroup, b.ObjectRef.Kind, b.ObjectRef.Namespace, b.ObjectRef.Name) + return keyA < keyB + } +) + +var ( + ns = &corev1.Namespace{ + TypeMeta: metav1.TypeMeta{ + Kind: nsKind, + APIVersion: "v1", + }, + } + + deploy = &appsv1.Deployment{ + TypeMeta: metav1.TypeMeta{ + Kind: deployKind, + APIVersion: "apps/v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: deployName, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(1)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "app": nginxName, + }, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": nginxName, + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: nginxName, + Image: nginxName, + Ports: []corev1.ContainerPort{ + { + ContainerPort: 80, + }, + }, + }, + }, + }, + }, + }, + } + + configMap = &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{ + Kind: configMapKind, + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + }, + Data: map[string]string{ + configMapDataKey: configMapDataValue, + }, + } +) + +func marshalK8sObjJSON(obj runtime.Object) []byte { + json, err := testutilsresource.MarshalRuntimeObjToJSONForTest(obj) + Expect(err).To(BeNil(), "Failed to marshal the k8s object to JSON") + return json +} + +// nsSnapshottedResource returns a snapshotted resource for a NS object of the given name. +func nsSnapshottedResource(nsName string) placementv1alpha1.SnapshottedResource { + regularNS := ns.DeepCopy() + regularNS.Name = nsName + return placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: marshalK8sObjJSON(regularNS)}, + } +} + +// deploySnapshottedResource returns a snapshotted resource for a Deployment object of the given namespace and name. +func deploySnapshottedResource(nsName, deployName string) placementv1alpha1.SnapshottedResource { + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + return placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: marshalK8sObjJSON(regularDeploy)}, + } +} + +// configMapSnapshottedResource returns a snapshotted resource for a ConfigMap object of the given namespace and name. +func configMapSnapshottedResource(nsName, configMapName string) placementv1alpha1.SnapshottedResource { + return configMapSnapshottedResourceWithData(nsName, configMapName, nil) +} + +// configMapSnapshottedResourceWithData returns a snapshotted resource for a ConfigMap object of the given namespace +// and name, with the given data (or the data from the template if the given data is nil). +func configMapSnapshottedResourceWithData(nsName, configMapName string, data map[string]string) placementv1alpha1.SnapshottedResource { + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + if data != nil { + regularConfigMap.Data = data + } + return placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + }, + Manifest: runtime.RawExtension{Raw: marshalK8sObjJSON(regularConfigMap)}, + } +} + +// nsManifestIdentifier returns the manifest identifier for a NS object of the given name at the given ordinal. +func nsManifestIdentifier(ordinal int, nsName string) placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: ordinal, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + } +} + +// deployManifestIdentifier returns the manifest identifier for a Deployment object of the given namespace and name +// at the given ordinal. +func deployManifestIdentifier(ordinal int, nsName, deployName string) placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: ordinal, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + } +} + +// configMapManifestIdentifier returns the manifest identifier for a ConfigMap object of the given namespace and name +// at the given ordinal. +func configMapManifestIdentifier(ordinal int, nsName, configMapName string) placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: ordinal, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + } +} + +// objectRefFromManifestIdentifier returns the object reference (as used in the failed resources of placement binding +// status) for the resource identified by the given manifest identifier. +func objectRefFromManifestIdentifier(identifier placementv1alpha1.ManifestIdentifier) placementv1alpha1.ObjectReference { + return placementv1alpha1.ObjectReference{ + Namespace: identifier.Namespace, + Name: identifier.Name, + APIGroup: identifier.APIGroup, + APIVersion: identifier.APIVersion, + Kind: identifier.Kind, + } +} + +// newPlacementBindingObj returns a placement binding object of the appropriate scope with only the namespace and name +// set: a ClusterPlacementBinding object if the namespace is empty, or a PlacementBinding object otherwise. +func newPlacementBindingObj(bindingNS, bindingName string) placementv1alpha1.PlacementBindingAccessor { + if bindingNS == "" { + return &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + }, + } + } + return &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: bindingNS, + Name: bindingName, + }, + } +} + +// newPlacementResourceSnapshotObj returns a placement resource snapshot object of the appropriate scope with only the +// namespace and name set: a ClusterPlacementResourceSnapshot object if the namespace is empty, or a +// PlacementResourceSnapshot object otherwise. +func newPlacementResourceSnapshotObj(snapshotNS, snapshotName string) placementv1alpha1.PlacementResourceSnapshotAccessor { + if snapshotNS == "" { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + }, + } + } + return &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: snapshotNS, + Name: snapshotName, + }, + } +} + +// getPlacementBinding retrieves a placement binding (cluster-scoped if the namespace is empty). +func getPlacementBinding(bindingNS, bindingName string) (placementv1alpha1.PlacementBindingAccessor, error) { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return nil, fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + return binding, nil +} + +// createPlacementResourceSnapshot creates a primary placement resource snapshot (sub-index 0) of the given index +// with the given resources, as the only snapshot of its index; it returns the snapshot as written to the API server. +// +// If the namespace is empty, a ClusterPlacementResourceSnapshot object is created; otherwise, a +// PlacementResourceSnapshot object is created. +func createPlacementResourceSnapshot( + snapshotNS, snapshotName, ownerPlacementPolicyName string, + index int, + resources ...placementv1alpha1.SnapshottedResource, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + return createSubIndexedPlacementResourceSnapshot(snapshotNS, snapshotName, ownerPlacementPolicyName, index, 0, 1, resources...) +} + +// createSubIndexedPlacementResourceSnapshot creates a placement resource snapshot of the given index and sub-index +// with the given resources; it returns the snapshot as written to the API server. +// +// The count of snapshots sharing the same index is set (as a label) only on the primary snapshot (sub-index 0); +// all snapshots of the same index share the same contents hash. If the namespace is empty, a +// ClusterPlacementResourceSnapshot object is created; otherwise, a PlacementResourceSnapshot object is created. +func createSubIndexedPlacementResourceSnapshot( + snapshotNS, snapshotName, ownerPlacementPolicyName string, + index, subIndex, subIndexedSnapshotCount int, + resources ...placementv1alpha1.SnapshottedResource, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + return createSubIndexedPlacementResourceSnapshotWithHash(snapshotNS, snapshotName, ownerPlacementPolicyName, + index, subIndex, subIndexedSnapshotCount, dummyContentsHash, resources...) +} + +// createSubIndexedPlacementResourceSnapshotWithHash is the same as createSubIndexedPlacementResourceSnapshot, except +// that it allows specifying the contents hash annotation of the snapshot. +func createSubIndexedPlacementResourceSnapshotWithHash( + snapshotNS, snapshotName, ownerPlacementPolicyName string, + index, subIndex, subIndexedSnapshotCount int, + contentsHash string, + resources ...placementv1alpha1.SnapshottedResource, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + snapshot := newPlacementResourceSnapshotObj(snapshotNS, snapshotName) + labels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: ownerPlacementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: strconv.Itoa(index), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: strconv.Itoa(subIndex), + } + if subIndex == 0 { + labels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = strconv.Itoa(subIndexedSnapshotCount) + } + snapshot.SetLabels(labels) + snapshot.SetAnnotations(map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: contentsHash, + }) + snapshot.SetSpec(placementv1alpha1.PlacementResourceSnapshotSpec{ + Resources: resources, + }) + Expect(hubClient.Create(ctx, snapshot)).To(Succeed(), "Failed to create the placement resource snapshot") + return snapshot +} + +// createPlacementBinding creates a placement binding that binds a placement resource snapshot to a member cluster; +// it returns the binding as written to the API server. +// +// If the namespace is empty, a ClusterPlacementBinding object is created; otherwise, a PlacementBinding object +// is created. +func createPlacementBinding(bindingNS, bindingName, placementPolicyName, clusterName, snapshotName string) placementv1alpha1.PlacementBindingAccessor { + binding := newPlacementBindingObj(bindingNS, bindingName) + binding.SetSpec(placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: placementPolicyName, + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + }) + Expect(hubClient.Create(ctx, binding)).To(Succeed(), "Failed to create the placement binding") + return binding +} + +// updatePlacementBindingResourceSnapshot points a placement binding to a different placement resource snapshot, +// as if a rollout has been made. +func updatePlacementBindingResourceSnapshot(bindingNS, bindingName, snapshotName string) { + Eventually(func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + binding.GetSpec().ResourceSnapshotName = snapshotName + if err := hubClient.Update(ctx, binding); err != nil { + return fmt.Errorf("failed to update the placement binding: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the placement binding with a new placement resource snapshot") +} + +// wantWorkForPrimaryPlacementResourceSnapshot builds the work object that the work generator is expected to create +// for a placement binding (cluster-scoped if the namespace is empty) from its primary placement resource snapshot, +// when the snapshot is the only one of its index. +func wantWorkForPrimaryPlacementResourceSnapshot( + workNS, workName, bindingNS, bindingName, placementPolicyName string, + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor, +) placementv1alpha1.Work { + return wantWorkForPlacementResourceSnapshot(workNS, workName, bindingNS, bindingName, placementPolicyName, + snapshot.GetName(), 0, 1, nil, snapshot) +} + +// wantWorkForPlacementResourceSnapshot builds the work object that the work generator is expected to create for a +// placement binding (cluster-scoped if the namespace is empty) from a placement resource snapshot of the given +// sub-index. +// +// The linked work count applies only to the primary work object (sub-index 0); the owner reference (which should +// point to the primary work object) applies only to secondary work objects. +func wantWorkForPlacementResourceSnapshot( + workNS, workName, bindingNS, bindingName, placementPolicyName, primarySnapshotName string, + subIndex, linkedWorkCount int, + ownerRef *metav1.OwnerReference, + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor, +) placementv1alpha1.Work { + // The manifests in the work object should match those in the snapshot as written to the API server. + resources := snapshot.GetSpec().Resources + manifests := make([]placementv1alpha1.Manifest, len(resources)) + for idx := range resources { + manifests[idx] = placementv1alpha1.Manifest{RawExtension: resources[idx].Manifest} + } + + work := placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: bindingNS, + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: bindingName, + }, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: primarySnapshotName, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: fmt.Sprintf("placement-resource-snapshot/%d", subIndex), + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: bindingName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: manifests, + }, + } + if subIndex == 0 { + work.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = strconv.Itoa(linkedWorkCount) + } else if ownerRef != nil { + work.OwnerReferences = []metav1.OwnerReference{*ownerRef} + } + return work +} + +// wantPlacementBindingStatusWaitingForSync builds the placement binding status that the work generator is expected +// to report after it has (re-)generated the work objects, but before the work objects have been processed. +func wantPlacementBindingStatusWaitingForSync(selectedResources int32, snapshotName string) *placementv1alpha1.PlacementBindingStatus { + return &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonWaitingForSynchronization, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionUnknown, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonWaitingForAvailabilityCheck, + }, + }, + SelectedResources: ptr.To(selectedResources), + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } +} + +// wantPlacementBindingStatusSyncedAndAvailable builds the placement binding status that the work generator is +// expected to report after all the work objects have been applied and become available. +func wantPlacementBindingStatusSyncedAndAvailable(selectedResources int32, snapshotName string) *placementv1alpha1.PlacementBindingStatus { + return &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonAllResourcesAvailable, + }, + }, + SelectedResources: ptr.To(selectedResources), + SynchronizedResources: ptr.To(selectedResources), + AvailableResources: ptr.To(selectedResources), + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } +} + +func placementBindingFinalizerAddedActual(bindingNS, bindingName string) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + // Check that the cleanup finalizer has been added. + if !controllerutil.ContainsFinalizer(binding, workGeneratorCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer has not been added") + } + return nil + } +} + +func worksGeneratedActual(clusterName, bindingNS, bindingName string, wantWorks []placementv1alpha1.Work) func() error { + return func() error { + // List the work objects owned by the placement binding. + workList := &placementv1alpha1.WorkList{} + listOptions := []client.ListOption{ + client.InNamespace(fmt.Sprintf(utils.NamespaceNameFormat, clusterName)), + client.MatchingLabels{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: bindingNS, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: bindingName, + }, + } + if err := hubClient.List(ctx, workList, listOptions...); err != nil { + return fmt.Errorf("failed to list work objects: %w", err) + } + + // Check that the work objects have been generated as expected. + if diff := cmp.Diff( + workList.Items, wantWorks, + ignoreFieldObjectMetaAutoGenFields, + ignoreFieldWorkTypeMetaAndStatus, + cmpopts.EquateEmpty(), + cmpopts.SortSlices(lessFuncWork), + ); diff != "" { + return fmt.Errorf("works diff (-got +want):\n%s", diff) + } + return nil + } +} + +func placementBindingStatusUpdatedActual(bindingNS, bindingName string, wantStatus *placementv1alpha1.PlacementBindingStatus) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + // Prepare the expected placement binding status; update the conditions with the observed generation. + wantStatusWithObsGen := wantStatus.DeepCopy() + for idx := range wantStatusWithObsGen.Conditions { + wantStatusWithObsGen.Conditions[idx].ObservedGeneration = binding.GetGeneration() + } + + // Check that the placement binding status has been updated as expected. + if diff := cmp.Diff( + *binding.GetStatus(), *wantStatusWithObsGen, + ignoreFieldConditionLTTMsg, + cmpopts.SortSlices(lessFuncFailedResource), + ); diff != "" { + return fmt.Errorf("placement binding status diff (-got +want):\n%s", diff) + } + return nil + } +} + +// placementBindingStatusMatchesActual verifies that the status of a placement binding matches the given one as is, +// i.e., unlike placementBindingStatusUpdatedActual, the observed generations of the conditions are not updated with +// the current generation of the placement binding. This is useful for verifying that a placement binding status has +// not been refreshed after the placement binding spec changes. +func placementBindingStatusMatchesActual(bindingNS, bindingName string, wantStatus *placementv1alpha1.PlacementBindingStatus) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + // Check that the placement binding status matches the expected one. + if diff := cmp.Diff( + *binding.GetStatus(), *wantStatus, + ignoreFieldConditionLTTMsg, + cmpopts.SortSlices(lessFuncFailedResource), + ); diff != "" { + return fmt.Errorf("placement binding status diff (-got +want):\n%s", diff) + } + return nil + } +} + +// markWorkAsAppliedAndAvailable updates the status of a work object to mark it (and all of its manifests) as +// applied and available, as the KubeFleet member agent would do. +func markWorkAsAppliedAndAvailable(workNS, workName string, manifestIdentifiers []placementv1alpha1.ManifestIdentifier) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + // Mark the work object as applied and available. + now := metav1.Now() + manifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestIdentifiers)) + for idx := range manifestIdentifiers { + manifestStatuses[idx] = placementv1alpha1.PerManifestStatus{ + Identifier: manifestIdentifiers[idx], + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAppliedReason, + Message: "Manifest has been marked as applied", + LastTransitionTime: now, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAvailableReason, + Message: "Manifest has been marked as available", + LastTransitionTime: now, + }, + }, + } + } + work.Status = placementv1alpha1.WorkStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAppliedReason, + Message: "Work has been marked as applied", + LastTransitionTime: now, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAvailableReason, + Message: "Work has been marked as available", + LastTransitionTime: now, + }, + }, + Manifests: manifestStatuses, + } + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark the work object as applied and available") +} + +// markWorkManifestAsUnavailable updates the status of a work object to mark one of its manifests (identified by +// its ordinal), and consequently the work object itself, as unavailable, as the KubeFleet member agent would do when +// an applied resource fails its availability check. +func markWorkManifestAsUnavailable(workNS, workName string, ordinal int) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + // Mark the manifest as unavailable. + found := false + for idx := range work.Status.Manifests { + manifestStatus := &work.Status.Manifests[idx] + if manifestStatus.Identifier.Ordinal != ordinal { + continue + } + meta.SetStatusCondition(&manifestStatus.Conditions, metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + Message: "Manifest has been marked as unavailable", + }) + found = true + } + Expect(found).To(BeTrue(), "Failed to find the status of the manifest with ordinal %d", ordinal) + + // Mark the work object as unavailable. + meta.SetStatusCondition(&work.Status.Conditions, metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: work.Generation, + Reason: markedAsUnavailableReason, + Message: "Work has been marked as unavailable", + }) + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark the work object as unavailable") +} + +// markWorkManifestsAsFailed updates the status of a work object to mark some of its manifests (identified by their +// ordinals) as failed to be applied (with the given diff details, if any), and some as applied but failed to become +// available; all the other manifests are marked as applied and available. This simulates what the KubeFleet member +// agent would do when it cannot apply some resources (e.g., when it fails to take over existing resources with diffs) +// and some applied resources fail their availability checks. +func markWorkManifestsAsFailed( + workNS, workName string, + manifestIdentifiers []placementv1alpha1.ManifestIdentifier, + failedToApplyOrdinals map[int]*placementv1alpha1.DiffDetails, + failedToBecomeAvailableOrdinals sets.Set[int], +) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + now := metav1.Now() + appliedCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAppliedReason, + Message: "Manifest has been marked as applied", + LastTransitionTime: now, + } + availableCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAvailableReason, + Message: "Manifest has been marked as available", + LastTransitionTime: now, + } + failedToApplyCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + Message: "Manifest has been marked as failed to be applied", + LastTransitionTime: now, + } + unavailableCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + Message: "Manifest has been marked as unavailable", + LastTransitionTime: now, + } + + manifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestIdentifiers)) + for idx := range manifestIdentifiers { + ordinal := manifestIdentifiers[idx].Ordinal + manifestStatuses[idx] = placementv1alpha1.PerManifestStatus{ + Identifier: manifestIdentifiers[idx], + } + + if diffDetails, failedToApply := failedToApplyOrdinals[ordinal]; failedToApply { + // Mark the manifest as failed to be applied; the availability check is skipped. + manifestStatuses[idx].Conditions = []metav1.Condition{failedToApplyCond} + manifestStatuses[idx].DiffDetails = diffDetails + continue + } + + if failedToBecomeAvailableOrdinals.Has(ordinal) { + // Mark the manifest as applied but unavailable. + manifestStatuses[idx].Conditions = []metav1.Condition{appliedCond, unavailableCond} + continue + } + + // Mark the manifest as applied and available. + manifestStatuses[idx].Conditions = []metav1.Condition{appliedCond, availableCond} + } + + // Mark the work object as failed to be applied (if applicable) and unavailable. + workAppliedCond := metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAppliedReason, + Message: "Work has been marked as applied", + LastTransitionTime: now, + } + if len(failedToApplyOrdinals) > 0 { + workAppliedCond.Status = metav1.ConditionFalse + workAppliedCond.Reason = markedAsFailedToApplyReason + workAppliedCond.Message = "Work has been marked as failed to be applied" + } + work.Status = placementv1alpha1.WorkStatus{ + Conditions: []metav1.Condition{ + workAppliedCond, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: work.Generation, + Reason: markedAsUnavailableReason, + Message: "Work has been marked as unavailable", + LastTransitionTime: now, + }, + }, + Manifests: manifestStatuses, + } + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark some manifests in the work object as failed") +} + +func removePlacementBinding(bindingNS, bindingName string) { + binding := newPlacementBindingObj(bindingNS, bindingName) + Expect(hubClient.Delete(ctx, binding)).To(Succeed(), "Failed to delete the placement binding") +} + +// placementResourceSnapshotRemovedActual deletes a placement resource snapshot, and verifies that it is gone. +func placementResourceSnapshotRemovedActual(snapshotNS, snapshotName string) func() error { + return func() error { + snapshot := newPlacementResourceSnapshotObj(snapshotNS, snapshotName) + if err := hubClient.Delete(ctx, snapshot); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the placement resource snapshot: %w", err) + } + + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: snapshotNS, Name: snapshotName}, snapshot); !apierrors.IsNotFound(err) { + return fmt.Errorf("placement resource snapshot still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func placementBindingRemovedActual(bindingNS, bindingName string) func() error { + return func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); !apierrors.IsNotFound(err) { + return fmt.Errorf("placement binding still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func workRemovedActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work); !apierrors.IsNotFound(err) { + return fmt.Errorf("work object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} diff --git a/pkg/v1/controllers/workgenerator/works.go b/pkg/v1/controllers/workgenerator/works.go index 8d8312c9c..77ab5958a 100644 --- a/pkg/v1/controllers/workgenerator/works.go +++ b/pkg/v1/controllers/workgenerator/works.go @@ -121,12 +121,16 @@ func areWorksUpToDate(placementBinding placementv1alpha1.PlacementBindingAccesso return true, nil } +// refreshWorks creates, updates, and deletes the work objects for a placement binding as needed. Besides the +// created/updated work objects, it also reports whether any work object has been created or has had its spec +// changed; metadata-only changes (e.g., re-linking a work object to a new placement resource snapshot with +// the same content) are not counted. func (r *Reconciler) refreshWorks(ctx context.Context, placementBinding placementv1alpha1.PlacementBindingAccessor, sortedPlacementResourceSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, works []placementv1alpha1.Work, ) ([]*placementv1alpha1.Work, bool, error) { - writtenToStorage := false + workSpecsChanged := false worksToDelete := []*placementv1alpha1.Work{} // Build an index of work objects by their names. @@ -196,14 +200,14 @@ func (r *Reconciler) refreshWorks(ctx context.Context, // Create the primary work object first. This is needed as the controller needs its object UID to set // owner references on the secondary work objects. - createdOrUpdatedWorks, primaryWorkObjWrittenToStorage, err := r.createOrUpdateWorkObjects(ctx, + createdOrUpdatedWorks, primaryWorkSpecChanged, err := r.createOrUpdateWorkObjects(ctx, []*placementv1alpha1.Work{primaryWorkToCreateOrUpdate}, placementBinding) if err != nil { return nil, false, errors.Wraps(err, "failed to create or update work object for primary placement resource snapshot", "primaryPlacementResourceSnapshot", klog.KObj(primaryPlacementResourceSnapshot)) } ownerWorkObjRef := metav1.NewControllerRef(createdOrUpdatedWorks[0], workGVK) - writtenToStorage = primaryWorkObjWrittenToStorage + workSpecsChanged = primaryWorkSpecChanged // Set the owner reference on all secondary work objects. for idx := range additionalWorksToCreateOrUpdate { @@ -212,16 +216,16 @@ func (r *Reconciler) refreshWorks(ctx context.Context, } // Issue the create or update ops for the secondary work objects in parallel. - additionalCreatedOrUpdatedWorks, additionalCreatedOrUpdated, err := r.createOrUpdateWorkObjects(ctx, additionalWorksToCreateOrUpdate, placementBinding) + additionalCreatedOrUpdatedWorks, additionalWorkSpecsChanged, err := r.createOrUpdateWorkObjects(ctx, additionalWorksToCreateOrUpdate, placementBinding) if err != nil { return nil, false, errors.Wraps(err, "failed to create or update additional work objects for secondary placement resource snapshots") } createdOrUpdatedWorks = append(createdOrUpdatedWorks, additionalCreatedOrUpdatedWorks...) - if !writtenToStorage { - writtenToStorage = additionalCreatedOrUpdated + if !workSpecsChanged { + workSpecsChanged = additionalWorkSpecsChanged } - return createdOrUpdatedWorks, writtenToStorage, nil + return createdOrUpdatedWorks, workSpecsChanged, nil } func buildWorkObjectFor( @@ -321,7 +325,7 @@ func (r *Reconciler) createOrUpdateWorkObjects( createdOrUpdatedWorks := make([]*placementv1alpha1.Work, len(worksToCreateOrUpdate)) errFlag := parallelizer.NewErrorFlag() - createdOrUpdated := atomic.Bool{} + specsChanged := atomic.Bool{} r.parallelizer.ParallelizeUntil(childCtx, len(worksToCreateOrUpdate), func(idx int) { work := worksToCreateOrUpdate[idx] @@ -331,7 +335,15 @@ func (r *Reconciler) createOrUpdateWorkObjects( Name: work.GetName(), }, } + specChanged := false resOp, err := controllerutil.CreateOrUpdate(childCtx, r.hubClient, createdOrUpdatedWork, func() error { + // Check if the spec of the work object has changed; this is used to track if the controller + // here performs a metadata-only change, which can happen upon same-content rollouts. + // + // Note that this is different from the comparison automatically performed by + // the CreateOrUpdate method. + specChanged = !equality.Semantic.DeepEqual(createdOrUpdatedWork.Spec, work.Spec) + // Work objects are considered to be fully internal KubeFleet resources; for this reason // here the control loop chooses to overwrite the spec, labels, annotations, and owner references of // the work object with the latest values instead of attempting to do a merge. @@ -350,9 +362,9 @@ func (r *Reconciler) createOrUpdateWorkObjects( } createdOrUpdatedWorks[idx] = createdOrUpdatedWork - if resOp != controllerutil.OperationResultNone { - // The work object has been created or updated. - createdOrUpdated.CompareAndSwap(false, true) + if specChanged { + // The work object has been created or has its spec updated. Metadata-only changes are ignored. + specsChanged.CompareAndSwap(false, true) } klog.V(2).InfoS("Successfully created or updated work object", "work", klog.KObj(createdOrUpdatedWork), "resOp", resOp, @@ -361,7 +373,7 @@ func (r *Reconciler) createOrUpdateWorkObjects( if err := errFlag.Lower(); err != nil { return nil, false, err } - return createdOrUpdatedWorks, createdOrUpdated.Load(), nil + return createdOrUpdatedWorks, specsChanged.Load(), nil } func (r *Reconciler) deleteWorkObjects( From 272e1c1827a35f1dbf67dda792e2cdc2881225e2 Mon Sep 17 00:00:00 2001 From: Chen Yu Date: Mon, 12 Oct 2026 22:22:45 +0800 Subject: [PATCH 19/20] fix: address a faulty method call in the code base caused by earlier merges (#968) --- pkg/v1/controllers/workgenerator/suite_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/v1/controllers/workgenerator/suite_test.go b/pkg/v1/controllers/workgenerator/suite_test.go index 84cc2beab..62d151dc1 100644 --- a/pkg/v1/controllers/workgenerator/suite_test.go +++ b/pkg/v1/controllers/workgenerator/suite_test.go @@ -110,7 +110,7 @@ var _ = BeforeSuite(func() { // The field indexes must be set up before the manager starts. By("Setting up the field indexes") - Expect(fieldindexers.SetupWithHubControllerManager(ctx, mgr)).Should(Succeed()) + Expect(fieldindexers.SetupWithHubAgentControllerManager(ctx, mgr)).Should(Succeed()) By("Setting up the work generator") Expect(New(mgr.GetClient(), workerCount).SetupWithManager(mgr, maxConcurrentReconciles)).Should(Succeed()) From b3199824216dbde3912b5f45c70b2f6815586c26 Mon Sep 17 00:00:00 2001 From: Copilot <198982749+Copilot@users.noreply.github.com> Date: Mon, 12 Oct 2026 12:29:27 -0700 Subject: [PATCH 20/20] fix: remediate Go CVEs from 2026-W42 scan (#969) * Initial plan * fix: remediate Go vulnerabilities Co-authored-by: britaniar <145056127+britaniar@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: britaniar <145056127+britaniar@users.noreply.github.com> --- docker/hub-agent.Dockerfile | 2 +- docker/member-agent.Dockerfile | 2 +- docker/refresh-token.Dockerfile | 2 +- go.mod | 12 ++++++------ go.sum | 24 ++++++++++++------------ 5 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docker/hub-agent.Dockerfile b/docker/hub-agent.Dockerfile index 3441e1d1b..51baf19c6 100644 --- a/docker/hub-agent.Dockerfile +++ b/docker/hub-agent.Dockerfile @@ -8,7 +8,7 @@ # Segmentation fault" / "cgo: gcc produced no output" in runtime/cgo and net) # and, when it did not crash, took over an hour per image. Only the final # distroless stage is per-target, and it runs no commands. -FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.8-1 AS builder +FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.9-1 AS builder WORKDIR /workspace # Copy the Go Modules manifests diff --git a/docker/member-agent.Dockerfile b/docker/member-agent.Dockerfile index 00c7807a1..ee93fdbd4 100644 --- a/docker/member-agent.Dockerfile +++ b/docker/member-agent.Dockerfile @@ -8,7 +8,7 @@ # Segmentation fault" / "cgo: gcc produced no output" in runtime/cgo and net) # and, when it did not crash, took over an hour per image. Only the final # distroless stage is per-target, and it runs no commands. -FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.8-1 AS builder +FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.9-1 AS builder WORKDIR /workspace # Copy the Go Modules manifests diff --git a/docker/refresh-token.Dockerfile b/docker/refresh-token.Dockerfile index 74e76ecaa..cad94f6a0 100644 --- a/docker/refresh-token.Dockerfile +++ b/docker/refresh-token.Dockerfile @@ -8,7 +8,7 @@ # Segmentation fault" / "cgo: gcc produced no output" in runtime/cgo and net) # and, when it did not crash, took over an hour per image. Only the final # distroless stage is per-target, and it runs no commands. -FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.8-1 AS builder +FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.9-1 AS builder WORKDIR /workspace # Copy the Go Modules manifests diff --git a/go.mod b/go.mod index af0c9ae0d..21256b90f 100644 --- a/go.mod +++ b/go.mod @@ -124,14 +124,14 @@ require ( go.uber.org/multierr v1.11.0 // indirect go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect - golang.org/x/crypto v0.56.0 // indirect + golang.org/x/crypto v0.57.0 // indirect golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect - golang.org/x/mod v0.40.0 // indirect - golang.org/x/net v0.58.0 // indirect + golang.org/x/mod v0.41.0 // indirect + golang.org/x/net v0.60.0 // indirect golang.org/x/oauth2 v0.36.0 // indirect - golang.org/x/sys v0.47.0 // indirect - golang.org/x/term v0.45.0 // indirect - golang.org/x/text v0.41.0 // indirect + golang.org/x/sys v0.48.0 // indirect + golang.org/x/term v0.46.0 // indirect + golang.org/x/text v0.42.0 // indirect golang.org/x/tools v0.49.0 // indirect google.golang.org/protobuf v1.36.12 // indirect gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect diff --git a/go.sum b/go.sum index e0d809464..bb758bf60 100644 --- a/go.sum +++ b/go.sum @@ -370,22 +370,22 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= -golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y= -golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I= +golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= +golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA= golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU= golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs= -golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE= +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU= +golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -403,15 +403,15 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= -golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= +golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= -golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0= -golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w= +golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= +golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= @@ -419,8 +419,8 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= -golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= +golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI= +golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E= golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE= golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=