From bd12121e838b20d7c9b27afefb2cfda682bce716 Mon Sep 17 00:00:00 2001 From: HardMax71 Date: Mon, 19 Oct 2026 01:24:49 +0200 Subject: [PATCH] ci: failed E2E jobs keep the k3s log and node description Three of 49 E2E jobs since k3s moved into a container timed out in Finalize k3s with the node never Ready, and none of them kept anything from k3s. Both E2E jobs now save docker logs k3s and kubectl describe nodes with their other failure logs. --- .github/workflows/ci.yml | 4 ++++ docs/operations/cicd.md | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3568e63d..f15fcde9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -571,6 +571,8 @@ jobs: docker compose logs --timestamps "$svc" > "logs/$svc.log" 2>&1 || true done kubectl get events --sort-by='.metadata.creationTimestamp' -A > logs/k8s-events.log 2>&1 || true + docker logs k3s > logs/k3s.log 2>&1 || true + kubectl describe nodes > logs/k8s-nodes.log 2>&1 || true - name: Upload logs if: failure() @@ -662,6 +664,8 @@ jobs: for svc in backend frontend; do docker compose logs --timestamps "$svc" > "logs/$svc.log" 2>&1 || true done + docker logs k3s > logs/k3s.log 2>&1 || true + kubectl describe nodes > logs/k8s-nodes.log 2>&1 || true - name: Upload logs if: failure() diff --git a/docs/operations/cicd.md b/docs/operations/cicd.md index 419ac689..debeafe7 100644 --- a/docs/operations/cicd.md +++ b/docs/operations/cicd.md @@ -82,7 +82,7 @@ Each E2E job gets its own runner with k3s and the full compose stack, pulling th `e2e-ready` copies k3s's kubeconfig for the runner's kubectl, waits for the node, writes `backend/kubeconfig.yaml` pointing at `host.docker.internal:6443` and creates the `integr8scode` namespace. Unless `install-kueue` is `false`, it applies the Kueue manifest, waits for the controller and creates the `ResourceFlavor`, `ClusterQueue` and `LocalQueue` (`kueue.x-k8s.io/v1beta2`), then waits for the image pre-pulls. If the controller isn't Available within 120 seconds, the step prints the `kueue-system` pods, their descriptions and events before it fails. It copies config.test.toml over config.toml, waits for the background infra start, runs `docker compose up -d --no-build` and polls `/api/v1/health/live` (and the frontend, when asked). It then waits until the five Kafka consumer groups report `Stable` in `kafka-consumer-groups --describe --state`. On a fresh Kafka most of them start at the newest message, so a request sent while they were still joining would never be read, and an execution test would time out. -The backend E2E job runs `pytest tests/e2e` inside the backend container after `uv sync --group test --no-dev --frozen --no-install-project --no-build`. The frontend E2E job runs as two shards. It caches Playwright browsers by package-lock.json, always runs `npm run test:e2e:deps`, runs `npm run test:e2e:browsers` on a cache miss, skips Kueue, and runs `npm run test:e2e -- --shard=N/2`. On failure both jobs upload compose logs; the backend one adds `kubectl get events`. +The backend E2E job runs `pytest tests/e2e` inside the backend container after `uv sync --group test --no-dev --frozen --no-install-project --no-build`. The frontend E2E job runs as two shards. It caches Playwright browsers by package-lock.json, always runs `npm run test:e2e:deps`, runs `npm run test:e2e:browsers` on a cache miss, skips Kueue, and runs `npm run test:e2e -- --shard=N/2`. On failure both jobs upload compose logs, the k3s container's log and `kubectl describe nodes`; the backend one adds `kubectl get events`. ## Docker Scan & Promote