Skip to content

ci: wait up to 300 s for Kueue's controller and print its log on failure - #402

Merged
HardMax71 merged 1 commit into
mainfrom
ci/kueue-controller-log-on-failure
Oct 18, 2026
Merged

HardMax71 merged 1 commit into
mainfrom
ci/kueue-controller-log-on-failure

Conversation

@HardMax71

@HardMax71 HardMax71 commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Summary

Main's run 39177020215 failed before any test: Kueue's controller started at once from the pre-pulled image, then answered every readiness probe with 404 for 110 s, and the 120 s wait gave up. This happens on a normal start, just a slow one.

Changes

Kueue v0.20.1 first runs a cert bootstrap manager on the health probe port with only a healthz check (BootstrapCerts), so /readyz is a 404 until bootstrap ends. Bootstrap waits for the kubelet to mount the new cert secret, and cert-controller v0.16.0 polls for that file with a doubling backoff from 1 s, so a slow mount pushes readiness past two minutes. A local controller took 1 min 44 s (20:33:30 "Waiting for certificate generation", 20:35:14 "Certificate bootstrap complete"). The wait is now 300 s, and on failure the step also prints the controller's last 300 log lines. cicd.md explains both.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: the log command ran against a local Kueue controller (300 lines). This PR's own E2E run uses the new step.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes CI failures where Kueue's controller never became ready within the 120-second wait, aborting E2E runs before any test started.

Kueue v0.20.1 boots a cert bootstrap manager that serves only /healthz until the new certificate secret is mounted, so /readyz returns 404 and the readiness probe fails. The bootstrap polls for the mounted file with exponential backoff, and a slow kubelet mount can push readiness past two minutes—a local controller took 1 min 44 s.

  • Increases the readiness wait from 120 s to 300 s.
  • Prints the controller's last 300 log lines when the wait fails, so failures are diagnosable without rerunning.
  • Updates docs/operations/cicd.md to explain both changes.

Written for commit 8b579d8. Summary will update on new commits.

View guided diff Turn on auto-fix

Kueue v0.20.1 starts with a cert bootstrap manager that serves the health probe port without a readyz check, so /readyz answers 404 until the certs are mounted. cert-controller polls for the mounted file with exponential backoff, and the kubelet can take a minute to mount the new secret, so a slow start runs past two minutes: run 39177020215 failed at 120 s, and a local controller took 1 min 44 s.
@HardMax71
HardMax71 enabled auto-merge (squash) October 18, 2026 23:52
@coderabbitai

coderabbitai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d7a07506-81d6-45ee-ab83-abbf01b5de42
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added documentation Improvements or additions to documentation area: ci GitHub workflows, test infrastructure, repo tooling labels Oct 18, 2026
@sonarqubecloud

Copy link
Copy Markdown

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag Coverage Δ
backend-e2e 82.76% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

Components Coverage Δ
Backend 92.14% <ø> (-0.02%) ⬇️
Frontend 92.60% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HardMax71
HardMax71 merged commit eebccf0 into main Oct 18, 2026
21 checks passed
@HardMax71
HardMax71 deleted the ci/kueue-controller-log-on-failure branch October 18, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci GitHub workflows, test infrastructure, repo tooling documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants