test(cilium): measure ExternalAuth cross-node delivery on a test-only route

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

• What Cilium 1.20.0-pre.3: Gateway API ExternalAuth ext_authz black-holes cross-node under WireGuard (blocks #1881) #2284 needs: a datapath test, because the ipcache diagnostic read INCONCLUSIVE with node encryption off (run https://github.com/devantler-tech/platform/actions/runs/34783415149). The maintainer approved running it on production on 2026-09-13.
• Today's SSO: it goes through oauth2-proxy as each route's backend, then to auth-proxy. No production route uses the Gateway API ExternalAuth filter.
• The earlier cutover: it moved six real routes at once, which is what turned the black-hole into a production login outage.

Audience and problem

The platform wants native ExternalAuth SSO (#1881), but nobody knows whether Cilium 1.20.1 still loses the ext_authz subrequest when Envoy and the auth backend are on different nodes. Testing that on a real login route risks another outage.

Hypothesis

With one test-only route on ExternalAuth, a probe can measure cross-node delivery without any real sign-in.

• Why no login is needed: an unauthenticated request still makes Envoy call oauth2-proxy, which answers 401 or 302 at once.
• What each outcome looks like: a delivered subrequest comes back fast, while a black-holed one times out.

Success signal

Over at least 100 unauthenticated probe requests, spread across gateway nodes:

• Fixed: every response is an oauth2-proxy 401 or 302, with no timeouts.
• Still broken: timeouts roughly in proportion to requests landing cross-node from the oauth2-proxy replicas.
• Guardrail: no user-facing route changes, and the component stays unreferenced unless the test is running.

Smallest useful change

1. Default-off component: a Kustomize component in the Hetzner provider overlay, following the existing components/ pattern. It holds:
   • an HTTPRoute on a dedicated probe hostname, routed to the existing whoami backend;
   • an ExternalAuth filter pointing at oauth2-proxy, with a non-empty allowedResponseHeaders;
   • the network policy and ReferenceGrant it needs.
2. Probe workflow: a dispatch-only workflow that sends N unauthenticated requests to the probe hostname. It reports the count of 401/302, other statuses and timeouts, and prints no addresses.
3. Separate enable PR: reference the component only while the test runs, and remove it afterwards.

Acceptance criteria

• The component renders and validates, but nothing references it on merge, so the production render is unchanged.
• Static tests pin the probe workflow: read-only, timeout-bounded, no identifying output.
• One measured run is recorded on Cilium 1.20.0-pre.3: Gateway API ExternalAuth ext_authz black-holes cross-node under WireGuard (blocks #1881) #2284 with its counts and a decision: fixed (re-attempt refactor(auth): replace Traefik auth-proxy with native Gateway API ExternalAuth filter (GEP-1494, Cilium 1.20) #1881 behind a flag), or still broken (keep auth-proxy and file the upstream report).

Rough size: M.

Part of #2284

Blocker: production dispatch of the merged Probe Cilium ExternalAuth Cross-Node workflow, plus confirmation of the agent-written approval (maintainer authority) | authority | last-verified 2026-09-15: workflow on main, 0 runs | asked slack 2026-09-15

[devantler]

🤖 Generated by the Agentic Engineer

The first design can't answer #2284. Codex's review of #3785 showed why, and all six findings hold.

• It can't see the path. Each node runs its own gateway Envoy, and the load balancer picks the node. A public client can't see or pin the node that makes the ExternalAuth check, or which oauth2-proxy replica it reaches. An all-302 run could therefore stay on one node the whole time.
• It can't trust its failures. Gateway routes are published Cloudflare-proxied, so an edge or TLS failure looks exactly like a lost subrequest.

What a conclusive test needs:

1. Know the topology during the run. Read which nodes host the oauth2-proxy replicas before and after, and refuse the run if they move. The ipcache diagnostic already does this with its reviewed production access.
2. Make the path cross-node on purpose. Send requests to the gateway on a node that runs no oauth2-proxy replica, inside the cluster and not through Cloudflare. Every check then has to cross nodes, and a same-node control run gives the baseline.
3. Attribute failures to the origin. Only failures that reach Envoy count as lost. Transport and edge failures stay inconclusive.
4. Have a working enable path. The route component has to be referenceable in a PR that passes CI, with the probe bounded by the job timeout and its inputs strictly validated.

Point 2 is the hard part, because the gateway Service load-balances across nodes. That needs a closer look at how the Cilium gateway exposes its per-node listener. #3785 stays a draft until that design is settled.

[devantler]

🤖 Generated by the Agentic Engineer

A concrete way to make the path cross-node (point 2 above), from the current manifests.

• How traffic reaches a node: the gateway is a Cilium-generated LoadBalancer Service, cilium-gateway-platform. The Hetzner load balancer reaches each node on its private address through that Service's node port. In production the gateway Envoy doesn't use the host network, and node ports only listen on private addresses.
• The option: a client inside the cluster can target one chosen node directly. Send HTTPS to that node's private address and node port, with --resolve so TLS still uses the probe hostname. That node's Envoy then makes the ExternalAuth check, and Cloudflare is out of the path. Pointing it at a node without an oauth2-proxy replica forces the cross-node case, and a node with one gives the same-node control.
• The trade-off: that client has to run inside production, as a short-lived pinned pod or a broader exec. That is more access than the ipcache diagnostic's single read-only exec, so the redesign has to justify the footprint and bound it just as tightly.

[devantler]

🤖 Generated by the Agentic Engineer

Redesign decision: an in-cluster, node-pinned probe with a same-path control

I'm picking this up (claimed 2026-09-14). The design below answers all six #3785 findings. It rests on facts I read from production today, read-only, counts only:

• One Envoy per node. cilium-envoy is ready on 8 of 8 nodes (external-envoy-proxy=true). Pod socket load balancing is off (bpf-lb-sock=false), so a pod's connection to the gateway Service is handled by the Envoy on the pod's own node.
• Envoy does not prefer local endpoints (envoy-node-locality-enabled=false). The two oauth2-proxy replicas run on 2 of the 8 nodes.
• external-dns publishes only hostnames inside the Cloudflare zone. The gateway's http listener accepts any hostname, so a route on a hostname outside the zone gets no DNS record, no Cloudflare hop and no TLS.

How each finding is answered

1. Enable path (P1). There's no GitOps component to activate. The dispatch-only workflow has the same shape as the ipcache diagnostic: main only, a confirmation phrase, and the prod-deploy lock. It applies its short-lived test objects itself, labels them, and deletes them in an exit trap. Nothing enters the Flux tree, so CI never validates an "activated" state.
2. Cross-node path (P1).
   • Forced cross-node: a short-lived client pod pinned to a node with no oauth2-proxy replica. Its local Envoy makes every ExternalAuth call, and both replicas are remote.
   • Same-node control: a second client pinned to a replica node.
   • Stability: replica placement and the node set are read before and after the run. Any change makes the run INCONCLUSIVE.
3. Origin attribution (P2).
   • Where requests go: in-cluster, plain HTTP, hostnames under externalauth-probe.invalid.
   • Plain-route control: from each client, a plain route with the same backend and no ExternalAuth filter is probed first. If it doesn't answer, the run is INCONCLUSIVE, because the path or policy is broken, not ext_authz.
   • Delivered: a 302 whose Location is the Dex login URL, which only oauth2-proxy produces.
   • Lost: no response within the per-request timeout.
   • Anything else is counted and never decides the verdict.
4. Budget (P2). The request count is capped so the all-timeout worst case still fits inside the job timeout. The script enforces the cap and the test pins it.
5. Decimal parsing (P2). Numeric inputs must be plain decimal with no leading zero, and are rejected with the usage exit before any arithmetic.
6. Hostname binding (P2). There's no hostname input. The script owns its fixed hostnames.

Verdicts

Verdict	When	Exit
FIXED	Plain routes answer, and every cross-node ExternalAuth request is delivered	0
FAULT-PERSISTS	Plain routes answer and same-node ExternalAuth is delivered, but cross-node ExternalAuth requests time out	0
INCONCLUSIVE	Anything else	3

The public log prints counts only.

Access, stated plainly

Unlike the read-only ipcache diagnostic, this writes to production while it runs:

• two HTTPRoutes and one egress policy for the probe clients, in whoami;
• one ReferenceGrant, in oauth2-proxy;
• two pods.

All of them are removed on exit, and no existing route, grant or policy is modified. The approval to run this test on production is recorded in this issue's body (2026-09-13).