security(longhorn): clear the pre-opt-in C-0211 residual in longhorn-system without detaching volumes unintentionally

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

longhorn-system is opted in to pod-security.devantler.tech/baseline-context, and the mutation
works — but 26 of its 41 pods still carry neither fsGroupChangePolicy nor seLinuxOptions.
Measured against live prod specs 2026-09-20, the split is temporal and completely clean:

• every clean pod was created on or after 2026-09-06T20:46:02Z
• every residual pod was created on or before 2026-08-26T13:14:47Z

#3636 landed the opt-in on 2026-09-06. The rule matches CREATE only, so this is the documented
behaviour working as designed: pods created after the opt-in carry both fields, and pods created
before it keep their original spec until something recreates them.

Nothing in longhorn-system is going to recreate them on its own:

12x DaemonSet/engine-image-ei-*        three engine versions, oldest pod 2026-06-16
 4x DaemonSet/longhorn-csi-plugin
 4x InstanceManager/instance-manager-*
 6x ReplicaSet/csi-{attacher,provisioner,resizer}

Longhorn keeps an engine-image DaemonSet per engine version for as long as any volume references
that version, and an InstanceManager per node for as long as volumes are attached there.

Audience and problem

This is the last chunk of the C-0211 residual in an already opted-in namespace, so it will not
close by itself and it is invisible in the rollout state — the namespace looks done.

It is deliberately not folded into #3239's per-namespace opt-in work, because the risk class is
different. Opting a namespace in is a posture change with no runtime effect on running pods. Clearing
this residual means recreating storage-plane pods, and recreating an InstanceManager requires
detaching the volumes it serves. That is a storage availability decision, and it should not ride
along inside a security sweep.

Expected behaviour

Every pod in longhorn-system carries both fields, reached by a sequenced roll that never detaches a
volume without intent.

Acceptance criteria

• The three workload classes are sequenced by disruption, least first: the CSI sidecar
  ReplicaSets and longhorn-csi-plugin DaemonSet roll normally; the engine-image DaemonSets
  are checked for whether any volume still references each engine version before touching them;
  the InstanceManager pods are handled last and explicitly, against attachment state.
• Any engine-image DaemonSet whose version no longer backs a volume is retired rather than
  rolled — that removes the pods instead of recreating them.
• Re-measured afterwards from live pod specs: 0 of the namespace's pods missing either field.
• No volume left detached or degraded, verified after the roll rather than assumed.

Not in scope

The kube-system opt-in (34 pods missing fsGroupChangePolicy, 26 missing seLinuxOptions, not yet
opted in) is the larger remaining population and belongs to #3239's ordinary per-namespace rollout.

The pod-versus-workload-spec distinction on #3522 is also separate and unaffected: clean pods do not
by themselves move the Kubescape C-0211 score, because Kubescape reads the stored workload spec.

Size

Small in diff, medium in care — the sequencing and the post-roll verification are the work.

Part of #3239

[devantler]

🤖 Generated by the Agentic Engineer

Re-measured scope for this issue (2026-09-20)

While re-measuring C-0211's residual for #3938 / PR #3939, this namespace's share was measured directly against the live PROD stored controller specs — the surface Kubescape reads.

longhorn-system is 8 of 12 scanned workloads, and it is now the largest single contributor to a portfolio-wide residual of 14 of 39 (observability 6; flux-system 0 of 5 and velero 0 of 2 are fully closed). By kind, this namespace's residual is 4 Deployments and 4 DaemonSets — no CronJobs and no StatefulSets.

Two corrections from that measurement change how this work should be checked, both recorded in full on #3239:

1. Check for the seLinuxOptions object, not a populated level. No workload in the population sets seLinuxOptions.level at all, and every object present is empty {}. A level-based acceptance test reports every workload failing and would never register this issue's progress.
2. The CronJob rule family closes nothing here. All 12 CronJobs in the population already carry both fields, so add-baseline-context-optin-controllers* is the family that reaches this namespace's 8.

Also worth carrying into the remediation: this namespace's residual is bounded by pod age, not by a policy defect — the mutation rules are CREATE-only by design, so what remains is workloads not recreated since the 2026-09-06 opt-in. Separation was perfect on that boundary when last checked, so recreation is the mechanism rather than a rule fix.

The per-workload detail is deliberately not reproduced here; it is derivable from the stored specs with the predicates described on #3239.

[devantler]

🤖 Generated by the Agentic Engineer

Independently re-measured against live prod, and the split is confirmed — with two corrections

Measured 2026-09-20T21:20Z against the live PROD stored controller specs (kubectl get deploy,daemonset -n longhorn-system), independently of the 18:24Z measurement. The residual is 8 of 11 controllers, and it lands exactly on the disruption classes this issue defines:

Class	Workloads	Residual
Deployment (CSI sidecars)	csi-attacher, csi-provisioner, csi-resizer, csi-snapshotter	4 / 4
DaemonSet	engine-image-ei-493e04e7, engine-image-ei-75a03ec3, engine-image-ei-a4d05f02, longhorn-csi-plugin	4 / 4
Already clean	longhorn-driver-deployer, longhorn-ui, longhorn-manager	0 / 3

That corroborates "4 Deployments and 4 DaemonSets — no CronJobs and no StatefulSets".

Correction 1 — the body's workload enumeration omits csi-snapshotter

The evidence block lists the sidecar ReplicaSets as 6x ReplicaSet/csi-{attacher,provisioner,resizer}. There are four residual CSI sidecar Deployments, not three: csi-snapshotter is also missing both fields. Anything sequencing off the body's list would leave a workload behind and then fail acceptance criterion 3.

Correction 2 — test seLinuxOptions per CONTAINER, not on the pod

The 18:24Z comment already corrected level-vs-object. There is a second axis, and it fails in the same direction: seLinuxOptions is absent at pod level on all 11 controllers, including the 3 that are fully clean. The baseline supplies it per container. So a pod-level predicate reports 11 of 11 failing and would never register progress either.

The predicate that discriminates:

fsGroupChangePolicy present  AND  every container has securityContext.seLinuxOptions

Under it, 8 fail and 3 pass, matching the Kubescape-side count. Pod-level absence is by design (#3469) and is not a gap.

Decomposition

The three classes carry genuinely different risk, so I have split the least-disruptive one out rather than sequencing all three in one change — the CSI sidecar Deployments roll normally and need no volume-attachment reasoning. Filed as a sub-issue; the engine-image and InstanceManager work stays here.

[devantler]

🤖 Generated by the Agentic Engineer

Progress, measured in prod 2026-09-21 ~02:05Z

• security(longhorn): clear the C-0211 residual on the four CSI sidecar Deployments #3946 / security(longhorn): retrofit C-0211 onto the four CSI sidecar Deployments #3953 verified: all four CSI sidecar Deployments (csi-attacher, csi-provisioner, csi-resizer, csi-snapshotter) now carry fsGroupChangePolicy: OnRootMismatch and seLinuxOptions on every container.
• Remaining residual: four DaemonSets. One is longhorn-csi-plugin; the other three are engine-image-ei-*, one per engine version.
• Why the engine-image criterion could never be met: automatic engine upgrades are off (Longhorn's default of 0), so 10 of 14 volumes stay on v1.11.1 or v1.12.0 indefinitely and every engine image stays referenced. security(longhorn): enable automatic engine upgrades so stale engine-image DaemonSets can retire #3966 turns the upgrade on (draft fix(longhorn): enable automatic engine upgrades so stale engine images can retire #3967). Once it deploys, the two old images should lose their last reference and be removed, which leaves longhorn-csi-plugin and the default engine-image DaemonSet for a scoped retrofit like security(longhorn): retrofit C-0211 onto the four CSI sidecar Deployments #3953's.