Migrate the Talos machine config onto the v1.14 configuration documents

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

#3626 establishes that Kubernetes cannot advance past v1.36 until Talos moves to the v1.14 line, and states the prerequisite as "Talos moves to the v1.14 line first". That prerequisite has never been specified, and it is considerably more than a version bump.

#3713 (chore(deps): update dependency siderolabs/talos to v1.14.0) now demonstrates what it actually costs. With the stale talosctl digest fixed (commit a8203d4d, so setup-talosctl.sh succeeds and talosctl gen config runs), 🧪 Validate Talos Machine Config fails on the talos (cloud) overlay with 8 conflicts — job 104627163211:

8 errors occurred:
	* UnattendedInstallConfig config is incompatible with v1alpha1 config (.machine.install)
	* .machine.nodeAnnotations is already set in v1alpha1 config
	* kubelet config is already set in v1alpha1 config (.machine.kubelet)
	* cluster network config is already set in the v1alpha1 config (.machine.cluster.network). Please remove it and use only the new KubeNetworkConfig document to avoid conflicts
	* kube-apiserver config is already set in v1alpha1 config (.cluster.apiServer)
	* kube-controller-manager config is already set in v1alpha1 config (.cluster.controllerManager)
	* cluster proxy config in v1alpha1 config (.machine.cluster.proxy) can't be used with KubeProxyConfig document, please remove it to avoid conflicts
	* cluster network config in v1alpha1 config (.machine.cluster.network) can't be used with KubeFlannelCNIConfig document, please remove it to avoid conflicts

Preceded by WARNING: .machine.files is deprecated; use dedicated configuration documents instead.

Every other check on that head is green — 🔐 Validate Production Authorization, 🧪 Validate Manifests, kubescape, CodeQL and the rest. This one check, and CI - Required Checks as its consequence, are the only failures.

The problem

Talos v1.14 moves settings out of the monolithic v1alpha1 document into dedicated configuration documents, and refuses to validate when both describe the same thing. Our machine-config patches still set all eight in v1alpha1. So the Talos upgrade cannot land as a version bump, and while it cannot land, #3534 (Kubernetes v1.37) stays permanently unmergeable — that is the whole of #3626's cost, and this is its root.

The same upstream restructuring is breaking KSail independently (ksail#6776 — MachineConfig.Kubelet and seven sibling accessors removed), so this is one upstream change surfacing in two products.

Expected behaviour

The Talos machine-config patches are migrated onto the v1.14 documents, talosctl validate passes for both overlays, and #3713 (or its successor) can go green on the version bump alone.

Acceptance criteria

1. .machine.install → UnattendedInstallConfig
2. .machine.kubelet → the dedicated kubelet document
3. .machine.cluster.network → KubeNetworkConfig (and the KubeFlannelCNIConfig conflict resolved with it)
4. .machine.cluster.proxy → KubeProxyConfig
5. .cluster.apiServer → the dedicated kube-apiserver document
6. .cluster.controllerManager → the dedicated kube-controller-manager document
7. .machine.nodeAnnotations moved off v1alpha1
8. .machine.files moved off the deprecated field
9. 🧪 Validate Talos Machine Config passes for both the talos (cloud) and talos-local (container) overlays
10. The rendered config is diffed against the current one to show the migration is settings-preserving — no grant, no mount, no network setting silently dropped

Size and risk

Medium, and higher risk than its size suggests: this rewrites the machine config that provisions production nodes. It wants its own reviewed change with a rendered before/after diff — not an adaptation commit folded into a Renovate version bump.

Part of #3626.

[devantler]

🤖 Generated by the Agentic Engineer

Three findings that change the shape of this work

Reproduced the failure locally against talosctl v1.14.0 with a faithful mirror of the CI step, then probed the current pin directly.

1. The migration and the version bump are ATOMIC — this cannot land on main first

This is the load-bearing one, because the issue currently implies the opposite ("#3713 (or its successor) can go green on the version bump alone"), and acting on that reading would take main red.

main pins TALOS_VERSION="1.13.9" in .github/scripts/setup-talosctl.sh, so CI on main validates with talosctl 1.13.9 — and 1.13.9 does not register the v1.14 documents at all:

$ talosctl-1.13.9 machineconfig patch controlplane.yaml --patch @kubelet-doc.yaml
error decoding document v1alpha1/KubeletConfig/ (line 1): "KubeletConfig" "v1alpha1": not registered

Its gen config emits exactly one non-v1alpha1 document (HostnameConfig), against 27 for 1.14.0.

So the migrated patches are unparseable on the current pin and the unmigrated patches are invalid on the next one. There is no ordering in which the two land separately and CI stays green in between. The Talos version bump, the digest, the ksail.prod.yaml pin and all fourteen patch files have to move in one change.

That also resolves the tension in the "Size and risk" note: this genuinely wants its own reviewed change rather than being folded into the Renovate bump — but that change has to carry the bump and supersede #3713, not land ahead of it.

2. The failure set is larger than the CI log showed — talos-local fails too

CI reports only the first failing overlay/role. Locally, all four combinations fail, and talos-local contributes a conflict the acceptance criteria do not list:

overlay	role	errors
talos	control-planes	8
talos	workers	4
talos-local	control-planes	6
talos-local	workers	3

The extra one is .cluster.allowSchedulingOnControlPlanes is already set in v1alpha1 config, from talos-local/cluster/allow-scheduling-on-control-planes.yaml. Criterion 9 already requires both overlays to pass, so this adds a mapping rather than changing the goal.

3. Per-file attribution — the fourteen files that actually have to move

v1alpha1 key	files
.machine.install	talos/cluster/enable-apparmor.yaml
.machine.nodeAnnotations	talos/cluster/mark-ghcr-pull-revision.yaml
.machine.kubelet	talos/cluster/{gc-unused-images-before-pressure,evict-pods-before-oom,rotate-server-certificates}.yaml, talos/workers/{label-nodes,mount-longhorn-data}.yaml, talos-local/cluster/rotate-server-certificates.yaml
.machine.cluster.proxy + .machine.cluster.network	talos/cluster/disable-default-cni-and-kube-proxy.yaml, talos-local/cluster/disable-default-cni-and-kube-proxy.yaml
.cluster.apiServer	talos/cluster/{enable-dex-oidc,enable-audit-logging}.yaml, talos-local/cluster/enable-dex-oidc.yaml
.cluster.controllerManager	talos/cluster/gc-terminated-pods-sooner.yaml
.cluster.allowSchedulingOnControlPlanes	talos-local/cluster/allow-scheduling-on-control-planes.yaml
.machine.files (deprecation warning only)	talos/cluster/enable-audit-logging.yaml, talos-local/cluster/enable-dex-oidc.yaml

talosctl docs --config <dir> generates the authoritative target schema for every one of these locally, which is what makes the mapping checkable rather than guessed.

A caution on acceptance criterion 10

Criterion 10 asks for a rendered before/after diff proving the migration is settings-preserving. Note that the generated base also changes between 1.13.9 and 1.14.0 — 1 non-v1alpha1 document becomes 27 — so a naive diff of rendered output conflates upstream's restructuring with our patch migration, and the signal that matters (did one of our settings get dropped?) is buried. The diff needs to compare effective settings across the version boundary, not rendered documents.

Not claimed

I am not holding this; the claim ref is retired. The reproduction is a faithful local mirror of the CI step and takes seconds to re-run, so whoever picks this up can iterate against talosctl validate directly rather than through CI.

[devantler]

🤖 Generated by the Agentic Engineer

Feasibility matrix — four of the eight conflicts have no v1.14 document home

I built a local mirror of the CI step (✅ Render + validate patched machine configs) that runs in
seconds, reproduced the RED baseline exactly, and then tested every conflict class against
talosctl v1.14.0 individually. Every result below is an executed machineconfig patch +
validate, not a reading of the schema.

RED baseline reproduced — and it is four failures, not one. CI reports only the first:

overlay / role	mode	errors
talos/control-planes	cloud	8
talos/workers	cloud	4
talos-local/control-planes	container	6
talos-local/workers	container	3

The structural fact this issue is missing

In v1.14 the generated base no longer carries these settings in v1alpha1 at all — gen config
emits 28 documents and the v1alpha1 document has no machine.kubelet, machine.install,
machine.nodeAnnotations, cluster.network, cluster.proxy, cluster.apiServer or
cluster.controllerManager section. So "already set in v1alpha1" is our patches putting them
back. That is good news for most of the migration: a patch document of the new kind merges cleanly
into the base document and validates.

Per-field results

v1alpha1 field	patch file(s)	v1.14 home	result
machine.kubelet.extraArgs	rotate-server-certificates (×2), workers/label-nodes	KubeletConfig.extraArgs	✅ PASS
machine.kubelet.extraConfig	evict-pods-before-oom, gc-unused-images-before-pressure	KubeletConfig.config (renamed)	✅ PASS
machine.nodeAnnotations	mark-ghcr-pull-revision	KubeNodeConfig.annotations	✅ PASS
cluster.controllerManager.extraArgs	gc-terminated-pods-sooner	KubeControllerManagerConfig.extraArgs	✅ PASS
cluster.apiServer.extraArgs	enable-dex-oidc (×2), enable-audit-logging	KubeAPIServerConfig.extraArgs	✅ PASS
cluster.network.cni: none	disable-default-cni-and-kube-proxy (×2)	delete KubeFlannelCNIConfig	✅ PASS
cluster.proxy.disabled: true	disable-default-cni-and-kube-proxy (×2)	KubeProxyConfig.enabled: false	✅ PASS — but see below
cluster.allowSchedulingOnControlPlanes	talos-local/allow-scheduling-on-control-planes	KubeNodeConfig.taints: {}	✅ PASS (re-expressed, not moved)
machine.kubelet.extraMounts	workers/mount-longhorn-data	none	🔴 blocked
machine.install.extraKernelArgs, grubUseUKICmdline	enable-apparmor	none	🔴 blocked
cluster.apiServer.extraVolumes	enable-audit-logging	none	🔴 blocked
machine.files → /etc/kubernetes/...	enable-audit-logging	EtcFileConfig refuses the path	🔴 blocked

Searched exhaustively across the complete v1.14 schema set (103 documents, 254 KB from
talosctl docs --config); extraKernelArgs, grubUseUKICmdline, extraVolumes, extraMounts and
allowSchedulingOnControlPlanes appear in none of them. Positive control run on the same corpus
(extraArgs, image, taints, nodeIP, installer all hit), so those zeros are real absences and
not a dead search.

The four blockers, concretely

1. Longhorn's bind mount. KubeletConfig has exactly five fields — image, config,
   extraArgs, clusterDNS, defaultRuntimeSeccompProfileEnabled. extraMounts is rejected with
   unknown keys found during decoding. This is a known upstream gap (v1.14 KubeletConfig not support extraMounts. siderolabs/talos#14365), and
   the upstream advice is to keep machine.kubelet in v1alpha1 — which is exactly what conflicts.
2. AppArmor. UnattendedInstallConfig carries only reboot, installer.image and
   provisioning.{diskSelector,wipe}. security=apparmor has nowhere to go.
3. Audit policy mount. KubeAPIServerConfig has no extraVolumes.
4. Audit policy file. EtcFileConfig rejects kubernetes/audit/audit-policy.yaml with
   user etc file ... is managed by Talos. The same document at audit/audit-policy.yaml validates,
   so the mechanism works and only the path we need is refused — and relocating it outside
   /etc/kubernetes needs extraVolumes, which is blocker 3. These two compound.

The escape, and its cost

$patch: delete on a base document does work — deleting KubeletConfig frees v1alpha1
machine.kubelet (verified: extraMounts then validates). But the deleted document is where the
base's pinned images live (ghcr.io/siderolabs/kubelet:v1.37.0, and
factory.talos.dev/metal-installer/<digest>:v1.14.0 for the installer). Deleting it either drops
those pins to Talos defaults or forces us to hardcode a generated, version-specific digest into a
patch. Both are worse than the problem for a document that provisions production nodes.

Two corrections to the issue

• Criterion 9 needs four green overlay/role combinations, not two. talos-local adds
  .cluster.allowSchedulingOnControlPlanes, which the criteria do not mention.
• KubeProxyConfig is control-plane-only (the following document kinds are only allowed on control plane machines). disable-default-cni-and-kube-proxy.yaml currently lives in cluster/
  and so applies to workers too. The kube-proxy half has to move to control-planes/, splitting
  a file that today is one patch — a structural change the criteria do not anticipate.

Also worth recording

• The migration and the version bump are atomic, and it is now enforced, not just observed:
  scripts/tests/refresh-flux-ghcr-auth/contracts_test.go asserts TALOS_VERSION in
  setup-talosctl.sh equals spec.cluster.talos.version in ksail.prod.yaml. CI pins 1.13.9,
  which cannot register the v1.14 kinds, so migrated patches go red until both pins move together.
  There is no ordering where the migration and the bump land separately with CI green in between.
• KSail is not a blocker: it already pins talos/pkg/machinery v1.14.0-alpha.2. ksail#6776 is
  about accessors removed after alpha.2 and is a separate concern.
• CI and production generate the base differently — CI uses talosctl gen config, production
  uses KSail. The patches have to satisfy both, and only the CI side is verified here.

Recommendation

Blockers 1–4 are upstream capability gaps, not patch bugs, so this issue cannot be closed by
migrating patches alone. I would split it: land the eight migratable classes, and track the four
blocked fields separately against an upstream fix, since each one of them keeps v1alpha1 in play
and therefore keeps the conflict alive.

I have not opened a PR — a partial migration leaves talosctl validate red, so there is nothing
shippable until the blocked four have a decided answer. Claim retired; this is unclaimed again.

[devantler]

🤖 Generated by the Agentic Engineer

#3851 is decided — this issue is unblocked, and it is smaller than it looked

The decision
resolved all four "no v1.14 home" fields. Two of them turned out not to need a home at all, so the
migration shrinks. Recording the concrete plan here, plus the verification this issue now owns.

The plan, file by file

Audit logging — talos/cluster/enable-audit-logging.yaml is rewritten, not patched around. The
policy moves inline into a KubeAuditPolicyConfig document, after which Talos places and mounts
it itself: the machine.files write and the cluster.apiServer.extraVolumes mount both go away,
and audit-policy-file with them. The audit-log-{path,maxage,maxbackup,maxsize} flags move to
KubeAPIServerConfig.extraArgs. Verified PASS in isolation (control-planes/cloud).

Kubelet — add a $patch: delete on the generated KubeletConfig document and restate
image: ghcr.io/siderolabs/kubelet:v<k8s>. v1alpha1 machine.kubelet is then live again, so
evict-pods-before-oom, gc-unused-images-before-pressure, rotate-server-certificates,
label-nodes and mount-longhorn-data need no edits at all. Verified PASS; pin retained.
The restated pin is guarded by #3855 — restating it is not optional, because deleting the document
drops the pin silently and still validates.

AppArmor — talos/cluster/enable-apparmor.yaml keeps extraKernelArgs (it is KSail's schematic
input) and drops grubUseUKICmdline: false, which KSail v7.184.0 already overrides to true on
every render. The long comment justifying false describes pre-fold behaviour and should go with it.
This file is not a Talos schema problem; the CI-side mismatch it exposes is #3854.

Carried from the earlier matrix
(tested there, not re-tested by me today): nodeAnnotations → KubeNodeConfig.annotations;
controllerManager.extraArgs → KubeControllerManagerConfig.extraArgs;
apiServer.extraArgs (dex) → KubeAPIServerConfig.extraArgs; CNI none → delete
KubeFlannelCNIConfig; proxy.disabled → KubeProxyConfig.enabled: false;
allowSchedulingOnControlPlanes → KubeNodeConfig.taints: {}.

Two shape notes from that matrix still stand: KubeProxyConfig is control-plane-only, so
disable-default-cni-and-kube-proxy.yaml has to split out of cluster/; and the Talos version bump
is atomic with these patches — contracts_test.go pins TALOS_VERSION to
ksail.prod.yaml, so there is no ordering where they land separately with CI green.

Verification this issue now owns

#3851's criteria 1–3 asked for live evidence that the function survives. A spike cannot produce it —
it needs the migration deployed — so it moves here:

1. Longhorn — /var/lib/longhorn still bind-mounted on a worker, and a Longhorn volume still
   attaches. The kubelet extraMounts path is unchanged, so this is a regression check rather than a
   new risk.
2. AppArmor — security=apparmor still on the node's kernel cmdline and still enforcing. This is
   the one with a real behavioural change in the file, so it deserves the closest look: confirm the
   arg arrives via the UKI cmdline now that grubUseUKICmdline is not forced false.
3. Audit — audit events still written to /var/log/audit/kube/audit.log. The policy now arrives
   by a different mechanism, so confirm the content took effect (e.g. a secret read logged at
   Metadata, an RBAC mutation at RequestResponse), not merely that the file exists.

Ordering

#3854 (CI fidelity) is worth landing first if it is cheap, because until CI renders what
production renders, this migration's machine.install result cannot be trusted from a green check.
It is not a hard blocker — I have not made it one — but it is the difference between CI attesting
this migration and CI attesting a config we do not deploy.

[devantler]

🤖 Generated by the Agentic Engineer

Not implementable yet: production's config renderer cannot read the new documents

I picked this up to implement the plan above and stopped before opening a PR. Three findings change the plan, and the first one blocks it.

1. KSail's Talos library cannot decode the documents this migration adds (blocker)

Production does not render its machine config with talosctl — KSail does, using the Talos config library it pins: pkg/machinery v1.14.0-alpha.2 (unchanged in KSail v7.184.0, the version this repo pins). That pre-release predates part of the v1.14 split. Rendering a base with it (contract 1.14, Kubernetes 1.36.4) gives 16 documents on a control plane and 5 on a worker, against 28 from talosctl 1.14.0 — with no KubeletConfig, KubeNodeConfig, UnattendedInstallConfig or KubeClusterConfig.

Applying patches with that same library:

Patch document	alpha.2 library
KubeNodeConfig (annotations; the control-plane taint for talos-local)	refused: "KubeNodeConfig" "v1alpha1": not registered
v1alpha1 machine.nodeAnnotations (control)	loads
KubeProxyConfig enabled: false, KubeFlannelCNIConfig $patch: delete, KubeControllerManagerConfig.extraArgs, KubeAuditPolicyConfig	load, apply, and the result validates

So a migration that passes this repo's CI (which renders with talosctl 1.14.0) would fail at deploy time. The KubeletConfig $patch: delete step is also a no-op there, because that document does not exist in KSail's base.

This makes #3850 depend on KSail moving to a Talos library that registers these kinds — ksail#6776, which removes the accessors blocking that bump — then a KSail release and a pin bump here. I have recorded that as a dependency on this issue.

2. The OIDC flags have to become an authentication config, not KubeAPIServerConfig.extraArgs

The earlier matrix mapped the Dex flags to KubeAPIServerConfig.extraArgs. On both bases, Talos rejects that: kube-apiserver extra argument "oidc-client-id" is not allowed: use KubeAuthenticationConfig. A KubeAuthenticationConfig with a jwt issuer (URL, audience public-client, email/groups claims with the oidc: prefixes) validates on both. Two cautions for whoever implements it:

• The patch replaces the whole configuration object on both bases. The base's anonymous block (health endpoints /livez, /readyz, /healthz) disappears unless the patch restates it.
• talosctl validate does not check the JWT content. An issuer of not a url validates, so a mistake here would first surface as a kube-apiserver that will not start.

The audit flags (audit-log-path and the three rotation flags) are accepted in KubeAPIServerConfig.extraArgs, as planned.

3. KubeNodeConfig.taints: {} does not remove the control-plane taint

It validates, but the rendered config still carries node-role.kubernetes.io/control-plane: NoSchedule. Merging an empty map removes nothing, so talos-local would quietly stop scheduling workloads on its control planes. That item needs a key-level delete, and it is also affected by finding 1.

What still holds

The file moves (control-plane-only settings out of cluster/), the audit policy moving inline into KubeAuditPolicyConfig, the proxy and CNI changes, and the AppArmor cleanup are unaffected. #3854 (CI rendering what production renders) is now more important: this result is exactly the kind of gap it describes, one layer below machine.install.

Method: CI's render-and-validate step mirrored locally, with the baseline reproduced exactly (8/4/6/3 errors); a second base rendered with pkg/machinery v1.14.0-alpha.2 from the Go module proxy, and patches applied with that library's own patcher. Each probe patch was tested on its own, with the controls named above. KSail's binary itself was not run against a cluster.