Repository navigation
ci: nothing execs into k3s before it has written its kubeconfig - #406
Conversation
At startup k3s moves every process out of the container's root cgroup and then enables the cgroup controllers there. Both pre-pull loops started docker exec k3s crictl info the moment the container ran, so an exec'd process could land in the root cgroup in between: k3s logged "failed to enable controller ... device or resource busy", the kubelet exited on "cgroup [kubepods] has some missing controllers", and Finalize k3s timed out (3 of 49 E2E jobs). The loops now wait for /etc/rancher/k3s/k3s.yaml first.
github-actions
Bot
added
documentation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
2 participants
Summary
Since #390, 3 of 49 E2E jobs timed out in "Finalize k3s" with the node never Ready, the latest on main in run 39179973449. Its k3s log, kept since #400, shows the cause. At 00:18:58 k3s logged "failed to move process 96 to cgroup /init" and "failed to enable controller ... device or resource busy". At 00:19:06 the kubelet exited on "cgroup [kubepods] has some missing controllers: cpu, cpuset, hugetlb, memory", and k3s died with it.
Changes
At startup k3s moves every process out of the container's root cgroup and then enables the controllers there. Our two pre-pull loops ran
docker exec k3s crictl infothe moment the container started (00:18:58.46 in that run), so an exec'd process could land in the root cgroup in between. Both loops now wait for/etc/rancher/k3s/k3s.yaml, which k3s writes seconds after that setup. cicd.md says why.Type of Change
Test Plan
cd backend && uv run pytest tests/unit)cd frontend && npm run test)docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')cd frontend && npm run test:e2e)./deploy.sh openapi, then./deploy.sh types), never edited by handValidation: with the same k3s image locally, execs already in flight at startup gave "device or resource busy" in 5 of 6 trials. No exec gave none. Here the node still came up, since this Docker host apparently has the controllers enabled already; the runner evidently didn't.
Checklist
pre-commit run --all-filesand all checks passSummary by cubic
Fixes E2E jobs timing out in "Finalize k3s" because
docker execinto the k3s container raced k3s's startup cgroup setup.The pre-pull loops ran
docker exec k3s crictl infothe moment the container started. At startup k3s moves every process out of the container's root cgroup before enabling controllers there, so an exec landing in between made that fail with "device or resource busy", the kubelet exited, and the node never became Ready./etc/rancher/k3s/k3s.yamlbefore exec'ing and document why.Written for commit 10c575c. Summary will update on new commits.