From 58cc1c348738e017be4400a4d32c8cd6e17b6d8c Mon Sep 17 00:00:00 2001 From: MDA2AV Date: Mon, 24 Aug 2026 19:14:24 +0100 Subject: [PATCH] validate: a skipped check is an environment gap, not a failed entry #1304 made a run with no assertions fail, to stop h3-only entries showing a green check while nothing had been verified. That was right for the case it was aimed at and wrong for the one CI actually hits. The per-framework validate job runs on ubuntu-latest. Load-generator images are built by benchmark.sh on the self-hosted runner, so h2load-h3 does not exist there and never will under the current workflow. The h3 checks therefore skip, the run ends 0 passed / 0 failed, and #1304 turned that into a hard failure -- zix-http3 now fails CI for a missing image on the runner rather than for anything about the entry: SKIP [h3]: no h2load-h3 image ... === Results: 0 passed, 0 failed, 1 skipped === FAIL: no checks ran for zix-http3 ... Split the two causes. If coverage exists and its tool was absent, say plainly that the entry is UNVALIDATED and pass -- nothing was proven, but nothing is wrong with the entry either. Fail only when validate.sh genuinely has no checks for anything the entry subscribes to. Verified both reachable paths against zix-http3: h2load-h3 present -> 2 passed, 0 failed, exit 0 h2load-h3 absent -> 0 passed, 0 failed, 1 skipped, WARNING, exit 0 Getting real h3 coverage on CI needs the image to exist there -- either built and cached in the workflow, or pulled from a registry. Neither is in this change. --- scripts/validate.sh | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/scripts/validate.sh b/scripts/validate.sh index 73d5f8864..d1292e45e 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -2923,12 +2923,24 @@ else fi # An entry that ran no assertions at all is unvalidated, not validated-clean. -# Exiting 0 here is what let H3-only entries show a green check while nothing -# had been verified about them; say so plainly and fail instead. +# Exiting 0 there is what let H3-only entries show a green check while nothing +# had been verified about them. +# +# But "no assertions" has two very different causes, and only one of them is the +# entry's problem. If coverage exists and the tool it needs was simply absent, +# that is an environment gap: the validate job runs on ubuntu-latest, which never +# builds the load-generator images (benchmark.sh does that, on the self-hosted +# runner), so h3 checks skip there and would fail every h3-only entry for a +# reason that has nothing to do with the entry. Warn loudly and pass. +# Fail only when validate.sh genuinely has no checks for anything subscribed. if [ "$PASS" -eq 0 ] && [ "$FAIL" -eq 0 ]; then echo "" - echo "FAIL: no checks ran for $FRAMEWORK — every subscribed test ($TESTS) is one validate.sh has no coverage for, so this run proves nothing" - exit 1 + if [ "$SKIPPED" -ne 0 ]; then + echo "WARNING: $FRAMEWORK is UNVALIDATED — the only coverage for its subscribed tests ($TESTS) was skipped for want of a tool on this machine. Nothing here was verified about the entry; run it where the load-generator images exist to get a real verdict." + else + echo "FAIL: no checks ran for $FRAMEWORK — every subscribed test ($TESTS) is one validate.sh has no coverage for, so this run proves nothing" + exit 1 + fi fi if [ "$FAIL" -ne 0 ]; then