diff --git a/.github/actions/e2e-ready/action.yml b/.github/actions/e2e-ready/action.yml index cdaa067f..6a12cb21 100644 --- a/.github/actions/e2e-ready/action.yml +++ b/.github/actions/e2e-ready/action.yml @@ -53,10 +53,12 @@ runs: kubectl apply --server-side -f "$KUEUE_MANIFEST" rm -f "$KUEUE_MANIFEST" - if ! kubectl wait --for=condition=Available --timeout=120s deployment/kueue-controller-manager -n kueue-system; then + # Kueue's cert bootstrap answers /readyz with 404 until the kubelet mounts its new secret, which can take minutes + if ! kubectl wait --for=condition=Available --timeout=300s deployment/kueue-controller-manager -n kueue-system; then kubectl -n kueue-system get pods -o wide kubectl -n kueue-system describe pods kubectl -n kueue-system get events --sort-by=.lastTimestamp + kubectl -n kueue-system logs deployment/kueue-controller-manager --all-containers --tail=300 exit 1 fi diff --git a/docs/operations/cicd.md b/docs/operations/cicd.md index debeafe7..1c5c3f7e 100644 --- a/docs/operations/cicd.md +++ b/docs/operations/cicd.md @@ -80,7 +80,7 @@ Each E2E job gets its own runner with k3s and the full compose stack, pulling th `e2e-boot` logs in to GHCR, starts `docker compose pull` and then `mongo redis shared-ca kafka` in the background, and starts k3s in a privileged container named `k3s`, from the `rancher/k3s` image, with its network policy controller off. No E2E test checks policy enforcement, and the worker still creates the NetworkPolicy object. The container publishes 6443 and mounts the runner's `/etc/rancher/k3s`, so the kubeconfig lands where the cert-generator service reads it, the same as with a host install. k3s runs in a container because its pods then get their network interfaces inside it. With k3s on the runner itself, every executor pod changed the runner's network, and Chromium aborted Playwright navigations in flight with `net::ERR_NETWORK_CHANGED`. Also in the background, it downloads the Kueue manifest and pre-pulls `python:3.14-slim`, the default runtime and the only one the tests execute, into k3s with `docker exec k3s crictl`, so the first executor pods don't wait on an image pull. Jobs that install Kueue also pre-pull `registry.k8s.io/kueue/kueue` that way, and `e2e-ready` waits for that pull before it waits for the controller, since the image alone once took over five minutes from registry.k8s.io. -`e2e-ready` copies k3s's kubeconfig for the runner's kubectl, waits for the node, writes `backend/kubeconfig.yaml` pointing at `host.docker.internal:6443` and creates the `integr8scode` namespace. Unless `install-kueue` is `false`, it applies the Kueue manifest, waits for the controller and creates the `ResourceFlavor`, `ClusterQueue` and `LocalQueue` (`kueue.x-k8s.io/v1beta2`), then waits for the image pre-pulls. If the controller isn't Available within 120 seconds, the step prints the `kueue-system` pods, their descriptions and events before it fails. It copies config.test.toml over config.toml, waits for the background infra start, runs `docker compose up -d --no-build` and polls `/api/v1/health/live` (and the frontend, when asked). It then waits until the five Kafka consumer groups report `Stable` in `kafka-consumer-groups --describe --state`. On a fresh Kafka most of them start at the newest message, so a request sent while they were still joining would never be read, and an execution test would time out. +`e2e-ready` copies k3s's kubeconfig for the runner's kubectl, waits for the node, writes `backend/kubeconfig.yaml` pointing at `host.docker.internal:6443` and creates the `integr8scode` namespace. Unless `install-kueue` is `false`, it applies the Kueue manifest, waits for the controller and creates the `ResourceFlavor`, `ClusterQueue` and `LocalQueue` (`kueue.x-k8s.io/v1beta2`), then waits for the image pre-pulls. Kueue starts with a certificate bootstrap that answers `/readyz` with 404 until the kubelet has mounted its new certificate secret, and it checks for the file with an exponential backoff, so on a slow run the controller takes over two minutes. If it isn't Available within 300 seconds, the step prints the `kueue-system` pods, their descriptions, events and the controller's last 300 log lines before it fails. It copies config.test.toml over config.toml, waits for the background infra start, runs `docker compose up -d --no-build` and polls `/api/v1/health/live` (and the frontend, when asked). It then waits until the five Kafka consumer groups report `Stable` in `kafka-consumer-groups --describe --state`. On a fresh Kafka most of them start at the newest message, so a request sent while they were still joining would never be read, and an execution test would time out. The backend E2E job runs `pytest tests/e2e` inside the backend container after `uv sync --group test --no-dev --frozen --no-install-project --no-build`. The frontend E2E job runs as two shards. It caches Playwright browsers by package-lock.json, always runs `npm run test:e2e:deps`, runs `npm run test:e2e:browsers` on a cache miss, skips Kueue, and runs `npm run test:e2e -- --shard=N/2`. On failure both jobs upload compose logs, the k3s container's log and `kubectl describe nodes`; the backend one adds `kubectl get events`.