security(platform): the KSail-installed node autoscaler is admitted before the kube-system LimitRange and declares no CPU limit

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

#3788 caps the Talos-bootstrapped CoreDNS after a DR rebuild, and #3789/#3793 declared CPU limits on the Flux-delivered kube-system add-ons. Both close their half of #3567. Neither reaches the node autoscaler, which is delivered by a third mechanism nobody enumerated.

Read live against prod kube-system on 2026-09-20T23:07Z:

• Deployment/cluster-autoscaler-hetzner-cluster-autoscaler stores requests: {cpu: 50m, memory: 128Mi} and limits: {memory: 256Mi} — no limits.cpu. Its admitted pod carries limits.cpu: 2, which is the kube-system default-limitrange default, not a declared value.
• It is the only kube-system Deployment or DaemonSet besides coredns whose stored spec has no CPU limit. Every other one declares its own.
• It is not a Flux HelmRelease in this repository. ksail.prod.yaml says so directly: "KSail installs, configures, and reconciles the autoscaler — no Flux manifests needed." Its Helm release name is cluster-autoscaler, and no helm-release.yaml here produces it.
• So it is admitted by ksail cluster create, in the same window as Talos's CoreDNS: before Flux is installed, and long before the bootstrap Kustomization creates the LimitRange. security(platform): Talos-bootstrapped CoreDNS on prod is admitted before the kube-system LimitRange and is never rolled by Flux #3567's ordering guarantee does not cover it for the same reason it does not cover CoreDNS.
• The shipped mitigation does not reach it either. scripts/restart-uncapped-talos-coredns.sh is hard-scoped by its own header — "This touches deployment/coredns and nothing else."
• scripts/tests/test-kube-system-addon-cpu-limits.sh pins 16 containers across six HelmReleases. The autoscaler is not among them, so the guard that exists to stop exactly this regressing is blind to it.

Who this affects, and impact

On a fresh prod bring-up — including the DR rebuild path, which runs ksail cluster create on fresh servers — the autoscaler pod runs uncapped. Nothing recreates it: KSail reconciles the release but does not roll a healthy pod, and the DR CoreDNS sweep skips it by design. Today's prod pod is capped only incidentally, because deploy-prod suspends autoscaling to zero replicas for the gated Cilium rollout and restores it, and that restore re-admits the pod under the LimitRange.

An uncapped autoscaler is the worst single candidate in the namespace to leave uncapped: it is the component that reacts to cluster-wide scheduling pressure, so the moment it is busiest is the moment nothing bounds its CPU. It also keeps a real subject under kubescape C-0270 and CKV_K8S_11 in kube-system during the window #3567 set out to close.

Expected behaviour

The autoscaler carries a CPU limit on a fresh cluster without depending on the LimitRange existing first, and a guard fails if that declaration is removed.

Options

1. Declare the limit in ksail.prod.yaml's autoscaler block, if KSail exposes a resources passthrough. Structural and matches what fix(security): declare CPU limits on the non-datapath kube-system add-ons #3793 did for the Flux-delivered add-ons. Needs KSail support confirmed first; if it is absent, that is a KSail feature request, not a platform change.
2. Widen the DR sweep from deployment/coredns to the bootstrap-admitted set. The existing script's narrowness was deliberate — a generic "restart whatever is uncapped" would roll Cilium — but the population is separable without that risk: a workload is bootstrap-admitted when it carries neither a kustomize.toolkit.fluxcd.io/name label nor a Flux-owned HelmRelease behind it. Measured against prod today that predicate selects coredns and the autoscaler and nothing else. Also needs the sweep wired into deploy-prod, which only dr-rebuild.yaml runs today.

Option 1 is the honest end state and leaves nothing to a post-hoc restart; option 2 is the fallback if KSail cannot express it. Check option 1 first.

Acceptance criteria

• Deployment/cluster-autoscaler-hetzner-cluster-autoscaler declares limits.cpu in its stored spec, read from the cluster rather than inferred from a manifest.
• The declared CPU request stays 50m, so the admitted 50m:2 ratio the VPA (InPlaceOrRecreate) holds is unchanged — the same constraint fix(security): declare CPU limits on the non-datapath kube-system add-ons #3793 recorded for the other add-ons.
• scripts/tests/test-kube-system-addon-cpu-limits.sh, or an equivalent guard, fails when the declaration is removed, with the ablation recorded.
• If KSail cannot express this, the KSail issue is filed and linked here, and option 2 ships instead.

Rough size: S (option 1) / M (option 2). Found while verifying #3567's acceptance criteria against main.

[devantler]

🤖 Generated by the Agentic Engineer

Option 1 is not expressible today — checked so the next run does not have to. NodeAutoscalerConfig in devantler-tech/ksail (pkg/apis/cluster/v1alpha1/autoscaler.go) carries enabled, pools, expander, maxNodesTotal, scaleDownUnneededTime, scaleDownUtilizationThreshold, capacityBuffers, ignoreDaemonsetsUtilization, skipNodesWithLocalStorage and skipNodesWithSystemPods. There is no resources field, and no generic chart-values passthrough beside it, so ksail.prod.yaml has nowhere to declare the limit. ksail.prod.yaml's own autoscaler block confirms it from the other side: every key it sets is one of the above.

So the work splits, and the split is what makes each half small:

1. A KSail feature request — let the node autoscaler's resources be declared, in the shape the other add-ons already use (an explicit request alongside the limit, so the admitted ratio is preserved rather than collapsed to 2:2). That is the end state this issue wants, and it is a KSail change, not a platform one.
2. Meanwhile, option 2 here — widen the bootstrap-admitted sweep beyond deployment/coredns so a fresh cluster is not left with an uncapped autoscaler while the KSail change is outstanding.

One correction to the body's option 2 while it is being planned: the label predicate it proposes ("neither a kustomize.toolkit.fluxcd.io/name label nor a Flux-owned HelmRelease") does not select the autoscaler. Measured on prod, the autoscaler carries meta.helm.sh/release-name: cluster-autoscaler and app.kubernetes.io/managed-by: Helm from KSail's own Helm install, so a Helm-annotation test excludes it exactly as it excludes the Flux-delivered add-ons. The two are only separable by who owns the release — Flux's HelmRelease inventory versus KSail's — not by anything on the Deployment. A sweep written against the labels alone would look correct, run clean, and skip the one workload this issue is about.

The predicate that does hold, and is the one to build on: a bootstrap-admitted workload is one no HelmRelease in this repository produces. coredns has no Helm release at all; the autoscaler's release name (cluster-autoscaler) matches no helm-release.yaml here. That is a static check against the repo, not a label read, and it is decidable at review time.

[devantler]

🤖 Generated by the Agentic Engineer

Status 2026-09-21: the KSail half has shipped; what remains is a version bump and a live read

• Option 1 is now expressible without a new field. KSail declares the limit itself: fix(autoscaler): declare a CPU limit on the cluster autoscaler ksail#7145 (fixing ksail#7144, which links here) sets the autoscaler's values to requests: {cpu: 50m, memory: 128Mi} and limits: {cpu: 2, memory: 256Mi}. That keeps the 50m:2 ratio AC2 asks for, and it applies on every cluster, whether or not a LimitRange exists yet.
• AC3's guard lives where the declaration lives. KSail's installer values test pins the full resources shape, so removing the CPU limit fails that test. A platform-side test could only pin a version, and it would fail until the cooldown below ends.
• Option 2 (widening the DR sweep) is no longer needed. A fresh ksail cluster create on ≥ v7.186.2 admits the autoscaler with its limit already declared. Nothing is left for a later restart to fix.
• What's left is AC1, and it has a date. The fix first shipped in v7.186.2 (published 2026-09-21T05:19Z). Platform pins KSAIL_VERSION: "7.184.5", and Renovate's repository-wide minimumReleaseAge: 7 days for privileged deployment tooling holds that release back until 2026-09-28. I'm deliberately not bypassing that cooldown. Once the grouped ksail bump merges and deploy-prod reconciles the release, read Deployment/cluster-autoscaler-hetzner-cluster-autoscaler from the cluster. Close this issue once limits.cpu appears in its stored spec.

Moving this to 📊 Verifying: the fix is merged, and the measurement is due on or after 2026-09-28.

[devantler]

🤖 Generated by the Agentic Engineer

Next step has a date: 2026-09-28. The autoscaler CPU limit (devantler-tech/ksail#7145, merged 2026-09-21T04:28Z) first ships in KSail v7.186.2 (published 2026-09-21T05:19Z). Platform pins KSail 7.184.9 in its workflows, and .github/renovate.json holds every update behind a global minimumReleaseAge: "7 days", which is why Renovate is still stepping through 7.184.x.

So Renovate can propose v7.186.2 or later from 2026-09-28T05:19Z. Bumping by hand before then would bypass that supply-chain gate, so this waits. After the bump merges and the next deploy re-admits the autoscaler, the remaining check is the live read: the stored Deployment spec carries limits.cpu.