ci(e2e): add e2e-gpu-rad3 lane for the scale-to-zero R9700 runner

[tomastola]

What

Adds e2e-gpu-rad3 to e2e-selfhosted.yml, targeting a self-hosted runner on the rad3 cluster (Radeon AI PRO R9700S, gfx1201, 32GB).

The runner is a Kubernetes pod that KEDA scales 0<->1 off the GitHub job queue, so the card is reserved only while a job exists rather than around the clock — that's the point of the exercise (silogen/dev-tooling EAI-8065). The runner side is deployed from silogen/dev-tooling; this PR is only the workflow lane that uses it.

How it differs from e2e-gpu

runs-on	[self-hosted, linux, r9700]
uv cache	pinned to /var/tmp/rocm-e2e-uv-cache (a separate PVC in the rad3 overlay, kept off the work PVC)

Triggers and gating are identical to the other self-hosted lanes: automatic on push/PR/merge_queue when the serve set changed, and on dispatch for platform=all or platform=rad3. continue-on-error: true, like the rest, so a rad3 flake never blocks a PR.

The label names the hardware (r9700) rather than reusing amd-gpu, because GitHub assigns a job to any runner whose label set is a superset of the job's. Distinct per-hardware labels keep the pools separately addressable — the same convention the strix lanes already follow — so rad3 only picks up work aimed at it, not MI300X-targeted jobs.

Scale-to-zero, and why it's transparent to CI

The runner has no pod most of the time. A queued job is what wakes it: KEDA sees the job, scales the StatefulSet 0->1, the pod connects within a poll interval, runs the job, and scales back to 0 after a cooldown. From the workflow's side this is just a self-hosted runner that happens to have startup latency; no lane logic depends on it.

The one operational caveat is generic to self-hosted runners, not specific to scale-to-zero: a job queued while a runner is offline can't be cancelled by GitHub until a runner connects. Scale-to-zero makes "no pod" the normal state, so the mitigation that matters is a health alert on the ScaledObject (tracked in EAI-8150) — a dead scaler is the only way the wake never happens. The older concurrency-starvation footgun (PR #138) is already handled by this workflow having its own concurrency group.

Testing

Validated end to end against the real ROCm/rocm-cli CI on the rad3 R9700, via workflow_dispatch with a single-scenario name_filter:

• dispatch -> KEDA Active=True within one 30s poll
• StatefulSet 0->1, pod admitted with amd.com/gpu: 1, Running in ~28s (warm node)
• job E2E tests (rad3 R9700) completed Succeeded (examine scenario 3, GPU + driver detection — a real GPU check)
• job end -> Active=False, cooldownPeriod: 300 held the pod, then scaled 1->0; pod gone, GPU released, no stranded pod and no manual cleanup

Earlier bring-up against a private scratch repo also exercised a job longer than cooldownPeriod (ran to completion untouched) and a cancelled job (released on the same path).

Risk

Low. The lane is continue-on-error, so it can't block merges, and it mirrors the established self-hosted lane structure. Fork-PR execution on self-hosted runners in this public repo is a repo-wide consideration that already applies to the app-dev and strix lanes; it is not specific to this one.

[rominf]

Review

Nice lane — the r9700 label reasoning and the scale-to-zero validation evidence are the strongest parts. Two things I'd fix before merge, one policy question, and an answer to "is there too much duplication here" since this PR is a good moment to ask it.

Blocking

1. The canonical artifact list is now self-contradictory, and a guard test cements the wrong side.

• xtask/src/e2e_report.rs:163-171 now lists 6 artifacts including e2e-gpu-rad3-report.
• docs/ci-hardware-testing.md:93-95 still lists 5.
• xtask/src/workflow_contract.rs:744-759 asserts the doc list equals exactly those 5.

So the repo states two different canonical lists, each pinned by a test — and the test actively blocks the fix, because adding rad3 to the doc list turns workflow_contract.rs red. The same test also pins three statements this PR makes stale: the "exactly four e2e-selfhosted.yml rows" table (:721-742), docs/ci-hardware-testing.md:49 and :142 ("The four self-hosted jobs…"), and tests/e2e-cucumber/README.md:162 ("four non-blocking jobs"). Separately, docs/ci-hardware-testing.md:121-124 documents the dispatch platform choices as all, app-dev-gpu, strix-ubuntu, strix-windows, strix-wsl — rad3 was added at e2e-selfhosted.yml:36 but isn't documented, so nobody reading the docs will know the option exists.

Suggested doc row:

| `e2e-gpu-rad3` | `e2e-selfhosted.yml` | Radeon AI PRO R9700 (gfx1201) on Linux | self-hosted `[self-hosted, linux, r9700]` |

…plus "e2e-gpu-rad3-report" into the workflow_contract.rs:744-759 expectation, and a rename off ..._all_four_self_hosted_platforms.

Worth a follow-up regardless: that test only compares the docs against themselves — it never reads e2e-selfhosted.yml/nightly.yml, so it can't detect a new lane. It only fires when someone tries to fix the docs, which is backwards. Deriving the expected rows from the actual job list (the helpers already exist in that file) would make it a real guard.

2. e2e-gpu-rad3 opts out of merge-queue coverage, seemingly by accident.

e2e-selfhosted.yml:872-874 sets only E2E_SERVE_TIMEOUT_SECS and E2E_INCLUDE_NIGHTLY. All four siblings also set:

      E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"

(:169, :375, :525, :695). merge_group is a trigger of this workflow (:21-23), and per the comment at :164-169 that variable is what enables the heavier default-engine/readiness serves. As written, rad3 runs the cheap PR canary during merge-queue while every other platform gets the full serve set — under-covering this hardware at the highest-stakes gate. workflow_contract.rs:626-644 only checks e2e-wsl, so nothing catches it. (The nightly lane correctly omits it — no merge_group trigger there.)

If it's deliberate, could you note it in the comment and in the difference table? Right now it reads as a dropped line.

Worth an explicit decision

167a2a26 moves rad3 from dispatch-only to pull_request-triggered; ec9ec870 ("trim stale rad3 lane comments") then removes the comment block that had documented why that was deliberately deferred — that on a public repo, pull_request on self-hosted hardware needs a fork-approval policy decision first.

The rationale for dropping it is reasonable — e2e-gpu and the Strix lanes already run on pull_request, so it is a repo-wide concern rather than this lane's — and the PR body does disclose it under Risk. But the marginal exposure isn't identical for a pooled, cache-persistent runner, and the record of why it mattered left the repo in the same PR that took the risk. Could someone with admin access confirm the "Require approval for fork pull requests" setting, and record the outcome in docs/ci-hardware-testing.md (which needs editing for #1 anyway) rather than deleting it? Happy to be told it's already settled.

Non-blocking

• e2e-gpu-nightly-rad3 drops the pre-warm diagnostics its own comment claims to mirror. nightly.yml:423 says it "Mirrors the per-PR e2e-gpu-rad3 lane", but :497-506 reduces the per-PR block (e2e-selfhosted.yml:950-963) to a bare if [ ! -d … ], losing both the "pre-warm did not produce a runtimes registry" warning and the else-branch. If pre-warm silently fails on rad3, nothing prints. Looks templated from the file-neighbour e2e-gpu-nightly rather than the named lane; the three Strix nightly lanes do have the full if/else.
• "R9700" vs "R9700S". crates/e2e-report/src/lib.rs:415 renders "R9700", but every comment the PR adds says R9700S. Shortening is stylistically consistent with "MI300X", but since "R9700" appears nowhere in prose it may be unintended.
• gfx1201 is reachable — I traced normalize_family → platform_hardware_slug → derive_platform_slug and it does fall through to the raw family, so e2e_report.rs:134 fires correctly. Just noting for later: derive_platform_slug only appends an OS suffix for strix-halo, so an R9700 Windows lane would collapse into the same gfx1201 slug and share a report column.
• Two out-of-repo path references (e2e-selfhosted.yml:936, nightly.yml:488) point at an overlay path that doesn't exist in this repo. Stripping the org prefix in ec9ec870 was the right instinct; consider going one step further and phrasing them self-containedly — "a separate PVC mounted at exactly this path by the runner's cluster overlay; if you change this path, change the overlay too" — so a public reader isn't left with an unfollowable pointer.
• nightly.yml has no concurrency: block at all, unlike e2e-selfhosted.yml:46-53. Pre-existing and orthogonal to rad3 (the nightly lanes target disjoint hardware), but a manual dispatch overlapping the 06:00 cron would put two rad3 jobs on the same label contending for /var/tmp/rocm-e2e-uv-cache.
• GPU_PREFLIGHT_MIN_FREE_GIB / GPU_PREFLIGHT_CEILING_SECS are never set anywhere in the repo — all 18 occurrences are ${VAR:-default} reads. Inert escape hatches rather than live knobs; not introduced here, just worth not mistaking for configuration.

On duplication — not this PR's problem, but this PR is a good prompt for it

Measured across the 10 e2e lanes in e2e-selfhosted.yml + nightly.yml: 62% of step-body lines (502 of 806) sit inside a block that appears verbatim in at least one other lane. The GPU preflight script exists in 5 copies differing in exactly one integer (MIN_FREE_GIB 16 vs 8); Upload E2E report in 10 copies differing only in name:. The main Run E2E tests script escapes exact matching only because of path substitutions — by sequence similarity it's 97.1% identical between e2e-gpu and e2e-gpu-rad3, and 97.6% between their nightly equivalents.

To be clear, this PR isn't the offender — the two new lanes sit at 56%/68%, right at the existing norm, and that 97% identity with e2e-gpu is exactly why the lane is low-risk. Following the established pattern was the right call here.

For a follow-up, the shape I'd suggest is a composite action, not a matrix:

• A matrix can preserve check names (codeql.yml:19 already proves the pattern in this repo), but matrix isn't in scope at jobs.<id>.if, so the per-lane inputs.platform == 'rad3' gating would have to move into a step gate — which means waking a scale-to-zero runner just to no-op. It also needs fail-fast: false or one flaky self-hosted box takes the others down, and the PowerShell lane can't join anyway.
• A composite action can't touch name:, if:, needs:, runs-on:, continue-on-error: or strategy:, so it's provably invisible to branch protection. Eight Linux call sites collapse onto one definition; the rad3 lane goes from 139 lines to ~25 with a byte-identical job header.
• workflow_call looks tempting but shouldn't bridge ci.yml and e2e-selfhosted.yml — that split exists precisely so an offline self-hosted runner can't stall a required check, and it's enforced by ci_yml_schedules_no_self_hosted_job and workflows_use_distinct_concurrency_groups.

Note that "one or two jobs" isn't reachable: ci.yml's E2E tests / E2E consolidated report are required checks, the self-hosted split is enforced by those two tests, Windows is a hard job split, and nightly is a separate tier. Realistic floor is 8 from 14 — and honestly, keeping all 14 jobs and applying composite actions alone removes ~500 duplicated lines at zero structural risk, which seems like the better trade.

Tracked separately in #294 so it doesn't block you here.

Things done well

• The r9700 label choice is right and the reasoning at :857-859 is worth having in the file — label matching is superset-based, so reusing amd-gpu would let a 32GB card pick up MI300X work.
• Both xtask guard tests satisfied deliberately, with 80e68197 naming them. Uploading the same artifact name from both workflows is exactly what keeps the_nightly_lanes_publish_the_same_platforms_as_the_per_pr_lanes green.
• timeout-minutes: 90 is correct and the "do not tighten below the Instinct lane's cap" comment is a useful guard. Cold start is a non-issue since the job clock starts at pickup, not queue — worth stating somewhere so nobody "optimises" it later.
• Validation evidence is concrete and falsifiable, and including the longer-than-cooldown and cancelled-job cases is the boundary probing this needed.

One note: the self-hosted lanes are serve-gated and this PR touches no serve code, so e2e-gpu-rad3 was correctly skipped on this PR — meaning the automatic-trigger path 167a2a26 introduces is itself unexercised. Might be worth one dispatch run on a branch that does touch serve, before or right after merge.

Branch is 3 commits behind main (all Dependabot, none touching these files) — no rebase risk.

[tomastola]

Thanks — thorough review, and the report/contract gap was a real miss. Addressed in 83275bb (workflow) and d61b37b (docs/contract).

Blocking — both fixed

1. Self-contradictory canonical list. The report code + its unit guards knew about e2e-gpu-rad3 but the doc-contract layer didn't, so the repo pinned two different lists. Reconciled:

• docs/ci-hardware-testing.md: added the e2e-gpu-rad3 job-table row, the rad3 dispatch platform option, and e2e-gpu-rad3-report to the canonical artifact list.
• Dropped the hardcoded "four self-hosted jobs" counts (both doc spots + tests/e2e-cucumber/README.md) rather than bumping them to five, so the next platform doesn't reopen this.
• workflow_contract.rs: added rad3 to the table + artifact assertions, updated the README-string assertion, and renamed ..._all_four_self_hosted_platforms → ..._all_self_hosted_platforms. All 18 contract + 8 report tests green.
• Good call on the follow-up (the test only compares docs against themselves) — agree it should derive expected rows from the actual job list; that's worth its own change and I've left it out of scope here.

2. Missing E2E_MERGE_QUEUE. Confirmed a dropped line, not intentional — added it so rad3 gets the full serve matrix at the merge gate like the four siblings.

Non-blocking — fixed

• nightly pre-warm else branch — restored the post-install registry check + else so a silent pre-warm failure prints. (Noting for the record: the MI300X nightly lane at nightly.yml:403 has the same reduced block — pre-existing, left out of this PR's scope.)
• Out-of-repo path refs — reworded both to "the runner's cluster overlay" so a public reader isn't left with an unfollowable pointer.

Non-blocking — kept as-is (with reasoning)

• R9700 vs R9700S — keeping R9700; it matches the MI300X shortening convention and never appears in prose, as you noted.
• derive_platform_slug OS suffix — good catch that an R9700 Windows lane would collide on the gfx1201 slug; noting for whenever a Windows R9700 lane appears, not a concern for this Linux-only one.
• nightly concurrency:, inert GPU_PREFLIGHT_* vars, the duplication refactor (ci(e2e): deduplicate the self-hosted e2e lanes into a composite action #294) — all pre-existing / out of scope, agreed.

Fork-approval policy — deferring

I can't check the "Require approval for fork pull requests" setting (no admin). Since e2e-gpu and the Strix lanes already run on pull_request, fork exposure on self-hosted hardware is a repo-wide property rather than something this lane introduces — but you're right the marginal exposure for a pooled, cache-persistent runner isn't identical, and the PR body discloses it under Risk. Rather than block here, I'd track confirming the setting + recording the outcome as a separate task. Flagging to the team to confirm.

Re: the unexercised automatic-trigger path — agreed, will do one dispatch on a serve-touching branch around merge to exercise it.

[r0x0r]

Approving. This adds a fifth self-hosted E2E lane for the rad3 R9700 (gfx1201) across both the per-PR (e2e-selfhosted.yml) and nightly (nightly.yml) workflows, and it faithfully mirrors the established lane structure rather than inventing a new one: same reclaim -> GPU-preflight -> prewarm -> cargo xtask e2e shape as e2e-gpu, same serve/dispatch if: gating, continue-on-error: true, and the existing include_nightly / E2E_MERGE_QUEUE wiring.

The [self-hosted, linux, r9700] label choice is correct — GitHub matches a job to any runner whose labels are a superset of the job's, so a distinct hardware label is what stops this card from picking up MI300X-targeted work; it follows the strix convention.

Crucially the naming contract stays green end to end: the new e2e-gpu-rad3-report artifact is threaded through parse_descriptor (gpu-rad3 -> R9700/Linux), the gfx1201 slug mapping and CANONICAL_REPORT_ARTIFACTS in xtask, and the docs-contract test in workflow_contract.rs. I verified there are no stale "four self-hosted/nightly platforms" references left behind — every occurrence is rewritten in lockstep. Third-party actions are pinned to full SHAs with version comments.

Two non-blocking notes: (1) the one red check, E2E tests (Strix Halo, Windows), is on a lane this PR does not touch and is continue-on-error, so it cannot gate the merge — reads as the known pre-existing Strix-Windows flake; please confirm. (2) The reclaim/preflight/prewarm block is now duplicated a fifth time; pre-existing debt this PR merely follows, but the file is at the size where a composite action would pay off (fine as a follow-up). Everything else is green.