Skip to content

Talos bumps land red: the digest and machinery pins never move with the version #3734

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

A Talos upgrade needs four coordinated edits. Renovate makes two of them, so every Talos
bump arrives red and has to be repaired by hand before it can even be evaluated.

edit who makes it today
TALOS_VERSION in .github/scripts/setup-talosctl.sh Renovate (custom manager)
spec.cluster.talos.version in ksail.prod.yaml Renovate (custom manager)
TALOSCTL_SHA256, on the line below TALOS_VERSION nobody
github.com/siderolabs/talos/pkg/machinery in go.mod Renovate, but as a separate PR

Observed on #3713 (siderolabs/talos → v1.14.0), opened 2026-09-10 and red ever since:

  1. 🧪 Validate Talos Machine Config failed at the install step — "the pinned talosctl digest
    is stale for v1.14.0 — the served bytes match this release's published sha256sum.txt, so
    the version was bumped without its digest."
  2. With the digest supplied, the same job failed one step later — "Kubernetes v1.36.4 is not
    verified compatible with Talos v1.14.0: compatibility with version 1.14.0 is not
    supported"
    — because machinery was still pinned at v1.13.9 and has no knowledge of the
    1.14 family.

Both guards behaved correctly and failed closed; the problem is that the PR cannot reach them
in a passing state on its own.

Why this costs something

The siderolabs/talos packageRule is otherwise meticulous about what a human must finish —
it documents the Hetzner iso id and the lockstep kubernetesVersion explicitly. These two
pins are simply absent from that list, so the completion step is under-specified and the
failure is rediscovered from scratch each time. #3713 sat red for two days; the diagnosis took
a full CI cycle per layer because the second fault is invisible until the first is fixed.

It also degrades a guard's signal: a job that is always red on Talos bumps for pin-drift
reasons trains readers to skip its output, which is exactly where a genuine
digest-mismatch-that-is-not-stale would appear.

Expected behaviour

A Talos bump arrives with every pin that can be derived already moved, and the packageRule
documents the ones that genuinely cannot be.

Options

  1. Group machinery with the Talos custom manager so both move in one PR. machinery is
    an ordinary gomod dependency whose version tracks the Talos release line exactly, so this
    is a groupName addition rather than new machinery.
  2. Derive TALOSCTL_SHA256 automatically. Renovate cannot compute it, but the release
    publishes sha256sum.txt, so a small CI step (or a postUpgradeTasks equivalent) could
    fetch and write it. Failing that, the packageRule description should name it alongside the
    iso id as a required manual completion.
  3. At minimum, document both in the siderolabs/talos packageRule, so the completion list
    is accurate even if nothing is automated.

Option 1 is cheap and removes one of the two faults outright; option 2 removes the other.

Acceptance criteria

  • A siderolabs/talos bump PR contains the machinery bump in the same PR, or the
    packageRule documents that it must be completed manually
  • TALOSCTL_SHA256 is either written automatically or named in the packageRule's
    completion list alongside the iso id
  • Verified against a real bump: the next Talos PR reaches
    validate-talos-kubernetes-compatibility without hand edits to the two pins above

Size

Small — a renovate.json change plus, for option 2, one CI step. The verification is the
slower part, since it needs a real bump to land.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions