Skip to content

fix: only wait on the machines an update brings up - #66

Merged
riadelhajjaji-scality merged 2 commits into
mainfrom
bugfix/MK8S-519-wait-only-on-new-machines
Nov 4, 2026
Merged

riadelhajjaji-scality merged 2 commits into
mainfrom
bugfix/MK8S-519-wait-only-on-new-machines

Conversation

@riadelhajjaji-scality

@riadelhajjaji-scality riadelhajjaji-scality commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Why

Updating a platform waits on every machine again, including the ones the update left alone:

node-2 booted badly, cloud-init exited 1
PermissionError: [Errno 13] Permission denied: '/run/cloud-init/cloud.cfg'

On Rocky 9, cloud-init ships a tmpfiles rule that keeps /run/cloud-init to root (0700), and systemd runs every tmpfiles rule again whenever a package ships one. Once something has been installed, cloud-init status can no longer read its runtime config as the SSH user (canonical/cloud-init#6680).

#65 (released in v3.2.3) asks through sudo -n only. That fixed MetalK8s bootstrap-restore but broke ARTESCA spawns from snapshots of clusters installed before scality/artesca#5818, where artesca-os is not allowed to run cloud-init through sudo:

sudo: a password is required
node-1 booted badly, cloud-init exited 1

(ARTESCA nightly upgrade-from-snapshots (single-node, previous, 4.3.4), run 41751747994. The same job passed on v3.2.2.)

What it does

  • Each machine gets its own wait-for-boot-<machine>, triggered by that machine's instance only. An update now waits only on the machines it brings up, which are freshly booted, and leaves the others alone. The bastion setup still waits on all of them.
  • The status is asked as the SSH user first, which works on a freshly booted machine. If that fails, it is asked again through sudo -n, if sudo -l says it is allowed. Otherwise the first answer is reported, so a sudo refusal is not mistaken for a bad boot.

Once ARTESCA versions without scality/artesca#5818 are no longer spawned, the first attempt goes and the status is only asked through sudo.

Since the waits are renamed, updating a platform spawned before this change waits once more on every machine.

Once merged, a GA patch release (v3.2.4) moves v3, so callers get the fix without changing anything on their side.

Closes MK8S-519

@riadelhajjaji-scality
riadelhajjaji-scality requested a review from a team as a code owner October 29, 2026 11:08
@riadelhajjaji-scality riadelhajjaji-scality changed the title ix: only wait on the machines an update brings up fix: only wait on the machines an update brings up Oct 29, 2026
@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as draft November 3, 2026 08:22
`wait-for-boot` was a single command checking every machine, re-run
whenever any instance changed. Removing one node therefore asked
cloud-init again on machines that had been in use for an hour.

Each machine now gets its own `wait-for-boot-<machine>`, triggered by
its own instance only. The bastion setup still waits on all of them.

The status is still asked through `sudo -n` first (#65), but only if
sudo allows it, checked with `sudo -l`. Otherwise it falls back to
asking as the SSH user, which works on a freshly booted machine. The
fallback covers ARTESCA machines installed before artesca-os was
allowed to run `cloud-init status --wait` through sudo
(scality/artesca#5818). It is meant to go once those versions are no
longer spawned.
@riadelhajjaji-scality
riadelhajjaji-scality force-pushed the bugfix/MK8S-519-wait-only-on-new-machines branch from 943d9ee to 4b3edc9 Compare November 3, 2026 13:33
@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as ready for review November 3, 2026 13:35
@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as draft November 4, 2026 08:36
@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as ready for review November 4, 2026 08:38
@riadelhajjaji-scality riadelhajjaji-scality self-assigned this Nov 4, 2026
Only new machines are waited on, and the SSH user can read the status
of a freshly booted one. That is also the only way on ARTESCA machines
installed before artesca-os was allowed to run cloud-init through sudo
(scality/artesca#5818).

When asking as the SSH user fails, the status is asked again through
`sudo -n`, if `sudo -l` says it is allowed. Otherwise the first answer
is reported, so that a refusal is not mistaken for a bad boot.
@riadelhajjaji-scality
riadelhajjaji-scality merged commit ad5fa94 into main Nov 4, 2026
14 checks passed
@riadelhajjaji-scality
riadelhajjaji-scality deleted the bugfix/MK8S-519-wait-only-on-new-machines branch November 4, 2026 14:32
@riadelhajjaji-scality
riadelhajjaji-scality restored the bugfix/MK8S-519-wait-only-on-new-machines branch November 4, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants