ci: E2E waits until the Kafka consumer groups are stable

.github/actions/e2e-ready/action.yml

@@ -119,6 +119,15 @@ runs:
         echo "Waiting for backend health..."
         timeout 120 bash -c 'until curl -ksf https://localhost/api/v1/health/live 2>/dev/null; do sleep 1; done'
         echo "Backend ready"
+        # Most groups start at the newest message on a fresh Kafka, so anything sent before they join is never read
+        for group in k8s-worker result-processor saga-orchestrator notification-service sse-bridge-pool; do
+          echo "Waiting for consumer group $group..."
+          timeout 120 bash -c "until docker compose exec -T kafka kafka-consumer-groups --bootstrap-server kafka:29092 --describe --group $group --state 2>/dev/null | grep -qw Stable; do sleep 2; done" || {
+            docker compose exec -T kafka kafka-consumer-groups --bootstrap-server kafka:29092 --describe --all-groups --state
+            exit 1
+          }
+        done
+        echo "Consumer groups ready"
         if [ "$WAIT_FOR_FRONTEND" = "true" ]; then
           echo "Waiting for frontend health..."
           timeout 60 bash -c 'until curl -ksf https://localhost:5001 2>/dev/null; do sleep 1; done'

docs/operations/cicd.md

@@ -80,7 +80,7 @@ Each E2E job gets its own runner with k3s and the full compose stack, pulling th
 
 `e2e-boot` logs in to GHCR, starts `docker compose pull` and then `mongo redis shared-ca kafka` in the background, and starts k3s in a privileged container named `k3s`, from the `rancher/k3s` image, with its network policy controller off. No E2E test checks policy enforcement, and the worker still creates the NetworkPolicy object. The container publishes 6443 and mounts the runner's `/etc/rancher/k3s`, so the kubeconfig lands where the cert-generator service reads it, the same as with a host install. k3s runs in a container because its pods then get their network interfaces inside it. With k3s on the runner itself, every executor pod changed the runner's network, and Chromium aborted Playwright navigations in flight with `net::ERR_NETWORK_CHANGED`. Also in the background, it downloads the Kueue manifest and pre-pulls `python:3.14-slim`, the default runtime and the only one the tests execute, into k3s with `docker exec k3s crictl`, so the first executor pods don't wait on an image pull. Jobs that install Kueue also pre-pull `registry.k8s.io/kueue/kueue` that way, and `e2e-ready` waits for that pull before it waits for the controller, since the image alone once took over five minutes from registry.k8s.io.
 
-`e2e-ready` copies k3s's kubeconfig for the runner's kubectl, waits for the node, writes `backend/kubeconfig.yaml` pointing at `host.docker.internal:6443` and creates the `integr8scode` namespace. Unless `install-kueue` is `false`, it applies the Kueue manifest, waits for the controller and creates the `ResourceFlavor`, `ClusterQueue` and `LocalQueue` (`kueue.x-k8s.io/v1beta2`), then waits for the image pre-pulls. If the controller isn't Available within 120 seconds, the step prints the `kueue-system` pods, their descriptions and events before it fails. It copies config.test.toml over config.toml, waits for the background infra start, runs `docker compose up -d --no-build` and polls `/api/v1/health/live` (and the frontend, when asked).
+`e2e-ready` copies k3s's kubeconfig for the runner's kubectl, waits for the node, writes `backend/kubeconfig.yaml` pointing at `host.docker.internal:6443` and creates the `integr8scode` namespace. Unless `install-kueue` is `false`, it applies the Kueue manifest, waits for the controller and creates the `ResourceFlavor`, `ClusterQueue` and `LocalQueue` (`kueue.x-k8s.io/v1beta2`), then waits for the image pre-pulls. If the controller isn't Available within 120 seconds, the step prints the `kueue-system` pods, their descriptions and events before it fails. It copies config.test.toml over config.toml, waits for the background infra start, runs `docker compose up -d --no-build` and polls `/api/v1/health/live` (and the frontend, when asked). It then waits until the five Kafka consumer groups report `Stable` in `kafka-consumer-groups --describe --state`. On a fresh Kafka most of them start at the newest message, so a request sent while they were still joining would never be read, and an execution test would time out.
 
 The backend E2E job runs `pytest tests/e2e` inside the backend container after `uv sync --group test --no-dev --frozen --no-install-project --no-build`. The frontend E2E job runs as two shards. It caches Playwright browsers by package-lock.json, always runs `npm run test:e2e:deps`, runs `npm run test:e2e:browsers` on a cache miss, skips Kueue, and runs `npm run test:e2e -- --shard=N/2`. On failure both jobs upload compose logs; the backend one adds `kubectl get events`.