Repository navigation
ci: wait up to 300 s for Kueue's controller and print its log on failure - #402
Conversation
Kueue v0.20.1 starts with a cert bootstrap manager that serves the health probe port without a readyz check, so /readyz answers 404 until the certs are mounted. cert-controller polls for the mounted file with exponential backoff, and the kubelet can take a minute to mount the new secret, so a slow start runs past two minutes: run 39177020215 failed at 120 s, and a local controller took 1 min 44 s.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|



Summary
Main's run 39177020215 failed before any test: Kueue's controller started at once from the pre-pulled image, then answered every readiness probe with 404 for 110 s, and the 120 s wait gave up. This happens on a normal start, just a slow one.
Changes
Kueue v0.20.1 first runs a cert bootstrap manager on the health probe port with only a
healthzcheck (BootstrapCerts), so/readyzis a 404 until bootstrap ends. Bootstrap waits for the kubelet to mount the new cert secret, and cert-controller v0.16.0 polls for that file with a doubling backoff from 1 s, so a slow mount pushes readiness past two minutes. A local controller took 1 min 44 s (20:33:30 "Waiting for certificate generation", 20:35:14 "Certificate bootstrap complete"). The wait is now 300 s, and on failure the step also prints the controller's last 300 log lines. cicd.md explains both.Type of Change
Test Plan
cd backend && uv run pytest tests/unit)cd frontend && npm run test)docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')cd frontend && npm run test:e2e)./deploy.sh openapi, then./deploy.sh types), never edited by handValidation: the log command ran against a local Kueue controller (300 lines). This PR's own E2E run uses the new step.
Checklist
pre-commit run --all-filesand all checks passSummary by cubic
Fixes CI failures where Kueue's controller never became ready within the 120-second wait, aborting E2E runs before any test started.
Kueue v0.20.1 boots a cert bootstrap manager that serves only
/healthzuntil the new certificate secret is mounted, so/readyzreturns 404 and the readiness probe fails. The bootstrap polls for the mounted file with exponential backoff, and a slow kubelet mount can push readiness past two minutes—a local controller took 1 min 44 s.docs/operations/cicd.mdto explain both changes.Written for commit 8b579d8. Summary will update on new commits.