From 10c575cdfe9540fe883ab19d1a7b87e4dec76311 Mon Sep 17 00:00:00 2001 From: HardMax71 Date: Mon, 19 Oct 2026 02:31:35 +0200 Subject: [PATCH] ci: nothing execs into k3s before it has written its kubeconfig At startup k3s moves every process out of the container's root cgroup and then enables the cgroup controllers there. Both pre-pull loops started docker exec k3s crictl info the moment the container ran, so an exec'd process could land in the root cgroup in between: k3s logged "failed to enable controller ... device or resource busy", the kubelet exited on "cgroup [kubepods] has some missing controllers", and Finalize k3s timed out (3 of 49 E2E jobs). The loops now wait for /etc/rancher/k3s/k3s.yaml first. --- .github/actions/e2e-boot/action.yml | 5 +++-- docs/operations/cicd.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/actions/e2e-boot/action.yml b/.github/actions/e2e-boot/action.yml index 387a2cb8..65de445c 100644 --- a/.github/actions/e2e-boot/action.yml +++ b/.github/actions/e2e-boot/action.yml @@ -58,8 +58,9 @@ runs: ' > /tmp/kueue-download.log 2>&1 & echo $! > /tmp/kueue-download.pid + # No docker exec before k3s writes its kubeconfig: one during its root cgroup evacuation kills the kubelet nohup bash -c ' - timeout 120 bash -c "until docker exec k3s crictl info >/dev/null 2>&1; do sleep 2; done" + timeout 120 bash -c "until [ -s /etc/rancher/k3s/k3s.yaml ] && docker exec k3s crictl info >/dev/null 2>&1; do sleep 2; done" docker exec k3s crictl pull docker.io/library/python:3.14-slim echo $? > /tmp/crictl-python.exit ' > /tmp/crictl-python.log 2>&1 & @@ -72,7 +73,7 @@ runs: run: | KUEUE_VERSION="${KUEUE_VERSION:-v0.20.1}" nohup bash -c ' - timeout 120 bash -c "until docker exec k3s crictl info >/dev/null 2>&1; do sleep 2; done" + timeout 120 bash -c "until [ -s /etc/rancher/k3s/k3s.yaml ] && docker exec k3s crictl info >/dev/null 2>&1; do sleep 2; done" docker exec k3s crictl pull registry.k8s.io/kueue/kueue:'"$KUEUE_VERSION"' echo $? > /tmp/crictl-kueue.exit ' > /tmp/crictl-kueue.log 2>&1 & diff --git a/docs/operations/cicd.md b/docs/operations/cicd.md index 1c5c3f7e..c988da5b 100644 --- a/docs/operations/cicd.md +++ b/docs/operations/cicd.md @@ -78,7 +78,7 @@ The pushes run in parallel and each failure is reported on its own. Fork PRs can Each E2E job gets its own runner with k3s and the full compose stack, pulling the images it just built. Two composite actions set that up, and the frontend job installs Node and Playwright between them so that work overlaps with k3s booting. -`e2e-boot` logs in to GHCR, starts `docker compose pull` and then `mongo redis shared-ca kafka` in the background, and starts k3s in a privileged container named `k3s`, from the `rancher/k3s` image, with its network policy controller off. No E2E test checks policy enforcement, and the worker still creates the NetworkPolicy object. The container publishes 6443 and mounts the runner's `/etc/rancher/k3s`, so the kubeconfig lands where the cert-generator service reads it, the same as with a host install. k3s runs in a container because its pods then get their network interfaces inside it. With k3s on the runner itself, every executor pod changed the runner's network, and Chromium aborted Playwright navigations in flight with `net::ERR_NETWORK_CHANGED`. Also in the background, it downloads the Kueue manifest and pre-pulls `python:3.14-slim`, the default runtime and the only one the tests execute, into k3s with `docker exec k3s crictl`, so the first executor pods don't wait on an image pull. Jobs that install Kueue also pre-pull `registry.k8s.io/kueue/kueue` that way, and `e2e-ready` waits for that pull before it waits for the controller, since the image alone once took over five minutes from registry.k8s.io. +`e2e-boot` logs in to GHCR, starts `docker compose pull` and then `mongo redis shared-ca kafka` in the background, and starts k3s in a privileged container named `k3s`, from the `rancher/k3s` image, with its network policy controller off. No E2E test checks policy enforcement, and the worker still creates the NetworkPolicy object. The container publishes 6443 and mounts the runner's `/etc/rancher/k3s`, so the kubeconfig lands where the cert-generator service reads it, the same as with a host install. k3s runs in a container because its pods then get their network interfaces inside it. With k3s on the runner itself, every executor pod changed the runner's network, and Chromium aborted Playwright navigations in flight with `net::ERR_NETWORK_CHANGED`. Also in the background, it downloads the Kueue manifest and, once k3s has written its kubeconfig, pre-pulls `python:3.14-slim`, the default runtime and the only one the tests execute, into k3s with `docker exec k3s crictl`, so the first executor pods don't wait on an image pull. Nothing execs into the container before that kubeconfig exists: at startup k3s moves every process out of the container's root cgroup and then enables the cgroup controllers there, and a `docker exec` landing in between made that fail with "device or resource busy", so the kubelet exited and the node never came up. Jobs that install Kueue also pre-pull `registry.k8s.io/kueue/kueue` that way, and `e2e-ready` waits for that pull before it waits for the controller, since the image alone once took over five minutes from registry.k8s.io. `e2e-ready` copies k3s's kubeconfig for the runner's kubectl, waits for the node, writes `backend/kubeconfig.yaml` pointing at `host.docker.internal:6443` and creates the `integr8scode` namespace. Unless `install-kueue` is `false`, it applies the Kueue manifest, waits for the controller and creates the `ResourceFlavor`, `ClusterQueue` and `LocalQueue` (`kueue.x-k8s.io/v1beta2`), then waits for the image pre-pulls. Kueue starts with a certificate bootstrap that answers `/readyz` with 404 until the kubelet has mounted its new certificate secret, and it checks for the file with an exponential backoff, so on a slow run the controller takes over two minutes. If it isn't Available within 300 seconds, the step prints the `kueue-system` pods, their descriptions, events and the controller's last 300 log lines before it fails. It copies config.test.toml over config.toml, waits for the background infra start, runs `docker compose up -d --no-build` and polls `/api/v1/health/live` (and the frontend, when asked). It then waits until the five Kafka consumer groups report `Stable` in `kafka-consumer-groups --describe --state`. On a fresh Kafka most of them start at the newest message, so a request sent while they were still joining would never be read, and an execution test would time out.