From 760c2884bdc2fef9fb2fd19f435a46d6e5c2f120 Mon Sep 17 00:00:00 2001 From: Lann Martin Date: Tue, 11 Aug 2026 20:21:29 -0400 Subject: [PATCH 1/2] Runner execution policy: the consolidated doc; --only reports deselected MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/runner-policy.md is the document #22 owed — the granularity/ replication/budget/reporting doctrine in one place, carrying both measurement campaigns (the webcrypto 11.5k corpus ladder from the issue thread, the 10k-synthetic wizen numbers from findings 22-24), the "K=1 at scale -> wizen" row and the JS-leg anti-recommendation (#85), and the selection-vs-capability rule. Guidance, not contract: the L1 surface permits every granularity; README commitments win on conflict. Writing "cost tiers are deselected, not not-applicable" forced the rule to become true: no runner emitted deselected, so a subset run could not participate in aggregation at all (omitted census = failed coverage). --only now reports the unselected remainder of the census as deselected wire events — never executed, no provenance, detail naming the selection — so filtered runs fold and aggregate cleanly with the subsetting visible as policy. Capability wins over selection: a tags-excluded case stays not-applicable even outside the filter, which is what aggregate's applicability policing requires of scheduled streams (deselected on a non-applicable case remains the error it should be — selection must not hide capability). Human mode stays silent per deselected case (a narrow --only over a large corpus must not bury the selected output); the summary line grows a ", N deselected" segment only when nonzero, so all goldens stay byte-identical. Empty selection stays a run error. The JS leg's `only` still omits (predates the rule): filed #89, noted in the doc. Also rides along: the #85 wasm-opt spike, concluded negative (findings.md 25). binaryen v124 refuses components outright (binaryen#6728), and the bound holds regardless: the wizened bench artifact's growth is snapshot data (1.18MB across 10,002 segments vs 93KB code), so total code deletion reclaims <=7.2%; measured on the extracted core module, wasm-opt -Oz saves 19.5KB (1.5%) — the case bodies stay live through the snapshotted case table, so "init-only code goes dead" is immaterial. Verified: just check, just all (all goldens byte-identical), just test-wasm (new: --only census test; fold coverage with deselected rows; aggregate selection-vs-capability both directions). Review caught and fixed a bad multiple in the doc: full isolation is ~1.7x the incumbent adapter but ~3x the fastest shared-instance mode; both now stated with their referents. --- AGENTS.md | 3 + components/sample-suite/README.md | 4 +- crates/component-test-cli/tests/cli.rs | 32 +++ .../component-test-formats/src/aggregate.rs | 49 +++++ crates/component-test-runner/src/lib.rs | 120 ++++++++--- crates/component-test-runner/tests/runner.rs | 59 +++++ docs/findings.md | 15 ++ docs/runner-policy.md | 203 ++++++++++++++++++ 8 files changed, 450 insertions(+), 35 deletions(-) create mode 100644 docs/runner-policy.md diff --git a/AGENTS.md b/AGENTS.md index 2e1895f..4b60f76 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,6 +79,9 @@ js/viewer/ results viewer + live browser harness (#46/#5): the examples/compose/ wac composition walkthrough (bundle-then-plug) expected/ golden outputs diffed by the verify recipes docs/findings.md toolchain findings log +docs/runner-policy.md runner execution policy (#22): granularity, + replication, budgets, wizening — guidance with + the measurement campaigns, not contract ``` **WIT deps are symlinks; suites have none.** `components/provider` and diff --git a/components/sample-suite/README.md b/components/sample-suite/README.md index 7c3564d..81783fc 100644 --- a/components/sample-suite/README.md +++ b/components/sample-suite/README.md @@ -85,4 +85,6 @@ snapshot freezes whatever init observed (env, entropy, clocks), and the wizened artifact must be the one used everywhere downstream (runners, `lock --check`). Suites that import a system-under-test drive `component_test_runner::wizen::wizen_with` with their own -linker instead. +linker instead. When wizening pays — and when instance granularity or +corpus preprocessing is the better lever — is the runner policy doc's +call: see `docs/runner-policy.md`. diff --git a/crates/component-test-cli/tests/cli.rs b/crates/component-test-cli/tests/cli.rs index 5d664c7..c8988a8 100644 --- a/crates/component-test-cli/tests/cli.rs +++ b/crates/component-test-cli/tests/cli.rs @@ -179,6 +179,38 @@ fn fold_duplicate_case_fails_coverage() { assert!(out.stdout.contains("more than once"), "{}", out.stdout); } +/// Deselected rows (a runner's `--only` subsetting, #22) satisfy +/// coverage: the whole census is reported, the subsetting is visible, +/// and nothing failed — exit 0. +#[test] +fn fold_deselected_rows_cover() { + let lock = tmpfile("deselected.lock", LOCK_XY); + let jsonl = stream( + &[ + r#"{"case":"a/x","status":"pass"}"#, + r#"{"case":"a/y","status":"deselected","detail":"--only x"}"#, + ], + true, + ); + let out = fold_with(&lock, &jsonl); + assert_eq!( + out.code, 0, + "stdout: {}\nstderr: {}", + out.stdout, out.stderr + ); + assert!( + out.stdout.contains("DESELECTED: a/y — --only x"), + "{}", + out.stdout + ); + assert!( + out.stdout + .contains("2 results (terminated): 1 deselected, 1 pass"), + "{}", + out.stdout + ); +} + #[test] fn fold_unknown_status_fails() { let lock = tmpfile("unknown.lock", LOCK_XY); diff --git a/crates/component-test-formats/src/aggregate.rs b/crates/component-test-formats/src/aggregate.rs index 4651248..1d41440 100644 --- a/crates/component-test-formats/src/aggregate.rs +++ b/crates/component-test-formats/src/aggregate.rs @@ -622,6 +622,55 @@ mod tests { assert_eq!(agg.targets, ["native", "sim"]); } + /// Corpus subsetting is selection policy (#22): an applicable case + /// reported `deselected` (a runner's `--only`, a smoke tier) is + /// coverage-complete, tolerated by applicability policing, and + /// non-failing — the subset target aggregates cleanly with the + /// subsetting visible. Capability still cannot hide behind + /// selection: `deselected` on a tags-excluded case remains the + /// applicability error (runners must report those not-applicable, + /// which takes precedence over deselection). + #[test] + fn deselected_is_selection_policy_not_capability() { + let sim_subset = doc( + "sim", + vec![ + result("suite/add", Status::Deselected, Some("--only declined")), + result("suite/hsm/attest", Status::NotApplicable, Some("hsm")), + result("suite/hsm/declined", Status::Pass, None), + ], + ); + let agg = aggregate( + &corpus_lock(), + &manifest(MANIFEST), + &[("native".into(), native_doc()), ("sim".into(), sim_subset)], + ); + assert!(agg.errors.is_empty(), "{:?}", agg.errors); + assert!(agg.ok()); + + let sim_hiding = doc( + "sim", + vec![ + result("suite/add", Status::Pass, None), + result("suite/hsm/attest", Status::Deselected, Some("--only x")), + result("suite/hsm/declined", Status::Pass, None), + ], + ); + let agg = aggregate( + &corpus_lock(), + &manifest(MANIFEST), + &[("native".into(), native_doc()), ("sim".into(), sim_hiding)], + ); + assert!( + agg.errors + .iter() + .any(|e| e.contains("suite/hsm/attest") && e.contains("not applicable")), + "{:?}", + agg.errors + ); + assert!(!agg.ok()); + } + #[test] fn artifact_hashes_are_provenance_not_identity() { // Suite builds are not reproducible across environments, so a diff --git a/crates/component-test-runner/src/lib.rs b/crates/component-test-runner/src/lib.rs index d9e4a42..0217125 100644 --- a/crates/component-test-runner/src/lib.rs +++ b/crates/component-test-runner/src/lib.rs @@ -6,6 +6,11 @@ //! (component-model-async), so calls go through wasmtime's concurrent //! API; host-side `diagnostic` is a concurrent host function. //! +//! The execution policy implemented here — instance granularity, +//! replication/striping, layered budgets, no retries, selection vs +//! capability — is documented with its measurements in +//! `docs/runner-policy.md`. +//! //! Env knobs: `COMPONENT_TEST_PROFILE=1` prints per-session enumeration //! timings to stderr (double-enumerates each session to separate //! registry construction from lifting cost). @@ -187,6 +192,9 @@ pub enum OutputMode { #[derive(Debug, Default)] pub struct Summary { pub not_applicable: usize, + /// Census cases outside the `only` selection: reported (as + /// `deselected` wire events), never executed. + pub deselected: usize, pub passed: usize, pub failed: usize, pub skipped: usize, @@ -194,7 +202,7 @@ pub struct Summary { impl Summary { pub fn total(&self) -> usize { - self.passed + self.failed + self.skipped + self.not_applicable + self.passed + self.failed + self.skipped + self.not_applicable + self.deselected } } @@ -650,9 +658,12 @@ impl Runner { .await } - /// `only`: run only cases whose name contains the substring (a - /// dev-loop filter; filtered cases are omitted from output, so - /// filtered runs will not aggregate cleanly — by design). + /// `only`: run only cases whose name contains the substring. The + /// rest of the census is reported as `deselected` (never executed), + /// so filtered runs still fold and aggregate cleanly, with the + /// subsetting visible as selection policy — never conflated with + /// capability (`not-applicable`, which takes precedence for cases + /// excluded by tags). A filter matching nothing is a run error. /// /// `case_execution_budget_secs`: budget on actual wasm *execution* /// per case — the executing thread's CPU time, sampled at epoch @@ -850,16 +861,31 @@ impl Runner { } } - // Scheduler pre-pass: census order, each entry either runs or - // is not-applicable (with the excluding tag). + // Scheduler pre-pass: census order, each entry runs, is + // not-applicable (with the excluding tag), or is deselected by + // `--only`. Applicability wins over deselection: capability is + // a property of the target and is reported truthfully + // regardless of selection, while deselection only narrows the + // runnable set (#22's cost-tier rule — subsetting is selection + // policy, visible as such; it is also what aggregate's + // applicability policing expects of scheduled streams). enum Action { Run, NotApplicable(String), + Deselected, + } + // Same normative rule as the zero-enumeration guard above: a + // `--only` substring matching nothing is an empty selection (a + // typo'd filter must not exit green with the whole census + // deselected). + if let Some(o) = only { + if !names.iter().any(|n| n.contains(o)) { + bail!("--only `{o}` matches no cases (empty selection is a run error)"); + } } let plan: Vec<(usize, &String, Action)> = names .iter() .enumerate() - .filter(|(_, name)| only.is_none_or(|o| name.contains(o))) .map(|(index, name)| { let action = match (&inventory, tags_of(name)) { (Some(_), Some(tags)) if !tags.applies(missing_features) => { @@ -869,20 +895,12 @@ impl Runner { .unwrap_or_default(), ) } + _ if !only.is_none_or(|o| name.contains(o)) => Action::Deselected, _ => Action::Run, }; (index, name, action) }) .collect(); - // Same normative rule as the zero-enumeration guard above: a - // `--only` substring matching nothing is an empty selection (a - // typo'd filter must not exit green with "0 total"). - if plan.is_empty() { - bail!( - "--only `{}` matches no cases (empty selection is a run error)", - only.unwrap_or_default() - ); - } // Parallel path: workers own stores; results are collected and // emitted in census order below. @@ -934,24 +952,51 @@ impl Runner { let mut session: Option> = None; for (index, enumerated_name, action) in plan { - if let Action::NotApplicable(mark) = action { - summary.not_applicable += 1; - if human { - println!("test {enumerated_name}: N/A ({mark})"); - } else { - let event = Event::Case(CaseResult { - case: enumerated_name.clone(), - status: Status::NotApplicable, - provenance: None, - detail: Some(mark), - seed: None, - duration_ms: None, - diagnostics: vec![], - diagnostics_complete: true, - }); - println!("{}", serde_json::to_string(&event)?); + match action { + Action::NotApplicable(mark) => { + summary.not_applicable += 1; + if human { + println!("test {enumerated_name}: N/A ({mark})"); + } else { + let event = Event::Case(CaseResult { + case: enumerated_name.clone(), + status: Status::NotApplicable, + provenance: None, + detail: Some(mark), + seed: None, + duration_ms: None, + diagnostics: vec![], + diagnostics_complete: true, + }); + println!("{}", serde_json::to_string(&event)?); + } + continue; } - continue; + // Deselected cases stay silent per-case in human mode + // (a narrow `--only` over a large corpus must not bury + // the selected cases under thousands of lines; the + // summary carries the count) but every one is a wire + // event: coverage folds and aggregation need the whole + // census reported, and the rows make the subsetting + // visible as selection policy rather than silence. + Action::Deselected => { + summary.deselected += 1; + if !human { + let event = Event::Case(CaseResult { + case: enumerated_name.clone(), + status: Status::Deselected, + provenance: None, + detail: only.map(|o| format!("--only {o}")), + seed: None, + duration_ms: None, + diagnostics: vec![], + diagnostics_complete: true, + }); + println!("{}", serde_json::to_string(&event)?); + } + continue; + } + Action::Run => {} } if human { @@ -1025,8 +1070,15 @@ impl Runner { } if human { + // The deselected segment appears only when a selection was + // in force, so unfiltered output stays byte-identical. + let deselected = if summary.deselected > 0 { + format!(", {} deselected", summary.deselected) + } else { + String::new() + }; println!( - "\nresult: {} passed, {} failed, {} skipped, {} not applicable, {} total", + "\nresult: {} passed, {} failed, {} skipped, {} not applicable{deselected}, {} total", summary.passed, summary.failed, summary.skipped, diff --git a/crates/component-test-runner/tests/runner.rs b/crates/component-test-runner/tests/runner.rs index db0b966..6942cb3 100644 --- a/crates/component-test-runner/tests/runner.rs +++ b/crates/component-test-runner/tests/runner.rs @@ -156,6 +156,65 @@ fn missing_feature_flips_applicability() { assert_eq!(status_of(&cases, "fixture/hsm/declined")["status"], "pass"); } +/// `--only` reports the unselected census as `deselected` instead of +/// omitting it (#22's cost-tier rule: subsetting is selection policy, +/// visible as such), so filtered runs keep full coverage. Capability +/// wins over selection: a tags-excluded case stays `not-applicable` +/// even outside the filter (what aggregate's applicability policing +/// requires of scheduled streams). +#[test] +#[ignore = "needs built components: run via `just test-wasm`"] +fn only_reports_the_rest_of_the_census_deselected() { + let wasm = fixture_wasm(); + let run = ct_runner(&[ + wasm.to_str().unwrap(), + "--jsonl", + "--missing", + "hsm", + "--only", + "gen", + ]); + // The deliberate trap is outside the selection: nothing failed. + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + let (_, cases, terminated) = parse_jsonl(&run.stdout); + assert!(terminated); + + // Full census: 8 events, in census order. + assert_eq!(cases.len(), 8, "full census reported"); + assert_eq!(status_of(&cases, "fixture/gen/tc1")["status"], "pass"); + assert_eq!(status_of(&cases, "fixture/gen/tc2")["status"], "pass"); + let boom = status_of(&cases, "fixture/trap/boom"); + assert_eq!(boom["status"], "deselected"); + assert_eq!(boom["detail"], "--only gen"); + assert!(boom.get("provenance").is_none(), "never executed"); + // Applicability beats selection. + let attest = status_of(&cases, "fixture/hsm/attest"); + assert_eq!(attest["status"], "not-applicable"); + assert_eq!(attest["detail"], "hsm"); + + // Human mode: deselected cases are silent per-case; the summary + // carries the count and the census total. + let run = ct_runner(&[wasm.to_str().unwrap(), "--missing", "hsm", "--only", "gen"]); + assert_eq!(run.code, 0, "stderr: {}", run.stderr); + assert!( + !run.stdout.contains("fixture/trap/boom"), + "deselected cases stay silent in human mode:\n{}", + run.stdout + ); + assert!( + run.stdout.contains( + "result: 2 passed, 0 failed, 0 skipped, 1 not applicable, 5 deselected, 8 total" + ), + "{}", + run.stdout + ); + + // A filter matching nothing remains an empty selection: run error. + let run = ct_runner(&[wasm.to_str().unwrap(), "--only", "zzz"]); + assert_eq!(run.code, 2, "stderr: {}", run.stderr); + assert!(run.stderr.contains("empty selection"), "{}", run.stderr); +} + #[test] #[ignore = "needs built components: run via `just test-wasm`"] fn suite_artifact_flag_rebinds_the_envelope() { diff --git a/docs/findings.md b/docs/findings.md index f02011c..011c5ed 100644 --- a/docs/findings.md +++ b/docs/findings.md @@ -230,4 +230,19 @@ bench-suite artifact built with its `wizer-init` feature. one-off). Wizening pays on JS legs only when enumeration is genuinely expensive; instance-granularity K>1 remains the lever there. +25. **wasm-opt post-pass on wizened artifacts: no meaningful + clawback, and blocked at the component level anyway.** binaryen + v124 refuses components outright (binaryen#6728). The bound holds + regardless: the wizened bench artifact's growth is snapshot + *data* — 1.18MB across 10,002 data segments vs 93KB of code in + the 1.29MB artifact (`wasm-tools objdump`) — so even total code + deletion reclaims ≤7.2%. Measured on the extracted core module + (byte range from objdump; no supported round-trip back into a + component exists): `wasm-opt -Oz` saves 19.5KB, 1.5% of the + artifact. #25's "init-only code goes dead after snapshotting" + hypothesis is immaterial in this shape: the case bodies stay live + through the snapshotted case table — only the one-shot + registration driver dies, and it is small. CoW already makes the + on-disk growth near-free at runtime (finding 23); transport + compression covers the wire. diff --git a/docs/runner-policy.md b/docs/runner-policy.md new file mode 100644 index 0000000..9e11b0b --- /dev/null +++ b/docs/runner-policy.md @@ -0,0 +1,203 @@ +# Runner execution policy + +Guidance for L3 runner implementers and for consumers tuning the +reference runners' knobs. Nothing here is contract: the L1 WIT surface +(`wit/tests.wit`) constrains none of it, and both instance +granularities below are legal implementations of it. Where this +document and a design commitment in the README disagree, the README +wins. The numbers come from two measurement campaigns: the +webcrypto-conformance corpus (11,578 generated cases, 17-core linux, +issue #22's thread) and a 10k-case synthetic +(`components/bench-suite`, findings.md 19–24). + +Most of this policy ships as *defaults* in the reference wasmtime +runner (`crates/component-test-runner`) rather than advice; the +sections say which knob, if any, moves each behavior. + +## Instance granularity: instance-per-case is the default + +A fresh instance per case (`cases_per_instance = 1`, ct-runner's +default) buys, in one mechanism: + +- **isolation as mechanical fact** — no shared guest state exists to + leak, so case self-containment stops being a discipline; +- **trivial trap containment** — a trap kills one case's store: record + a trap-provenance `fail`, instantiate the next case. No resume + protocol, no case-trap vs harness-error distinction, no + re-instantiation loops to guard against; +- **replication as the parallelism axis** (see below) with sharding + granularity 1. + +The knob (`--cases-per-instance`): `1` = per-case; `K` = a fresh +instance every K cases; `0` = one instance for the whole run (or per +worker, under parallelism). A trap abandons the current instance +regardless of K — the next case gets a fresh session, resumed +positionally at the next census index (deterministic `all()` order +across instances is a contract guarantee; the JS harness's positional +relocation additionally name-verifies, finding 21). The fold +synthesizes `not-reached` only for cases an abandoned *run* never got +to. Diagnostics already emitted for an abandoned case stand, marked as +a prefix (`diagnostics-complete: false`). + +Shared-instance modes (K > 1, K = 0) are the documented fallback for +targets where per-case setup dominates — browser boots are the +canonical example, and load-induced per-case boot timeouts are their +flake signature. The cost is that poisoning forfeits instance state +for the remainder of the recycle window; the reference runner's +session-slot machinery (poison + re-create + resume by index) makes +that a per-case trap rather than a run failure. + +### What the default costs, measured + +Profiling the 11.5k-case corpus under K=1: **95% of wall time was +`all()`** — per-instance registry construction plus lifting 11.5k +handles (6.9 ms/case) — while instantiation was 64 µs (copy-on-write +images; the pooling allocator added no measurable win), case work +358 µs, teardown 57 µs. Enumeration, not instantiation, is the K=1 tax. + +The mitigation ladder, each step removing its own layer (full-run wall +clock, same corpus): + +| change | full run | +|---|---| +| baseline: JSON corpus parsed per instance, K=1, sequential | 11:36 | +| build-time corpus preprocessing (postcard) | 3:35 | +| zero-copy corpus (rkyv `access_unchecked`) | 1:23 | +| K=1 sequential, all of the above | 0:36 | +| **K=1, `--jobs 8` — full per-case isolation** | **7.3s** | +| K=0, `--jobs 8` — instance-per-worker | 2.3s | +| (incumbent single-instance concurrent adapter, for scale) | (4.3s) | + +Doctrine: make registry construction cheap *first* (preprocess corpora +at build time; keep construction O(cases served), not O(suite) — see +findings 19–21 for the quadratic traps). After that, full isolation +runs within ~1.7× of the incumbent single-instance concurrent adapter +it replaced (7.3s vs 4.3s) and ~3× of the fastest shared-instance +replication mode (K=0, `--jobs 8`: 2.3s) — a price worth paying by +default. Tune K high or 0 only for cheap pure-compute corpora where +even that multiple matters, or for targets whose per-instance boot is +the dominant term. + +### K=1 at scale: wizen instead of relaxing isolation + +Wizer pre-initialization (`component-test wizen`, #25/#85) runs the +suite's own `all()` at build time and snapshots the heap: every fresh +instance is born with the registry built, removing the enumeration tax +without giving up isolation. On the 10k synthetic under the wasmtime +runner: per-instance `all()` 3.15 ms → 663 µs; instantiation unchanged +(~19 µs — CoW absorbs the 122 KB → 1.29 MB artifact growth); K=1 +full-isolation run 30.8s → 7.1s sequential, **1.14s at `--jobs 8`**. +Inventory, tag scheduling, and `lock --check` all work unchanged on +the wizened artifact (finding 22). + +When it does *not* pay: + +- **JS legs (deltic, browser): net ~1.5× at best.** V8 has no + copy-on-write memory images, so the 1.29 MB active data segment is + copied at every instantiation (0.78 ms → 2.17 ms), eating most of + the enumeration win (finding 24). Instance granularity K>1 remains + the JS-leg lever. +- **K=0 topologies: irrelevant.** The registry builds once (~12 ms on + the 11.5k corpus with a preprocessed corpus); there is nothing left + to amortize. + +Caveats the wizened artifact carries: the snapshot freezes whatever +init observed (env, entropy, clocks — baked at wizen time for every +future instance), and the wizened artifact must be the one used +everywhere downstream — runners, lockfile checks, hashing — never +mixed with the unwizened build. Suites with SUT imports drive +`component_test_runner::wizen::wizen_with` with their own linker; see +`components/sample-suite/README.md` for the suite-author view. + +## Parallelism: replication, striping, byte-stable output + +Intra-instance concurrency is cooperative-only under the component +model, so **replication is the parallelism axis**: `--jobs N` workers +share the compiled engine, own their stores, and each runs the modulo +stripe `runnable-index % jobs == worker`. Striping, not contiguous +chunking: expensive cases cluster by construction in generated corpora +(by algorithm, by key size), so chunks systematically imbalance. +`cases_per_instance` applies per worker; K=0 under parallelism means +instance-per-worker — the replication doctrine literally. + +Results are emitted in census order regardless of completion order, so +parallel runs produce byte-identical result streams — parallelism is +invisible to everything downstream (goldens, folds, aggregation). +Under `--jobs`, diagnostics print with their case's block rather than +live. + +Parallelism multipliers need empirical tuning per target class: +browser targets saturate at low multiples, and load-induced timeouts +are the flake signature to watch for when raising them. + +## Layered guards: execution budgets and wall timeouts + +Two guards per case, catching disjoint failure modes +(`--case-execution-budget`, default 10s; `--case-timeout`, default +120s; `0` disables either): + +- The **execution budget** meters actual wasm *execution* — the + executing thread's CPU time, sampled at epoch ticks — so it catches + CPU spins, which no wall timer can (the executor thread is stuck + inside wasm), and a contended machine stretching wall clock does not + eat a case's budget. +- The **wall timeout** covers suspension, so it catches async wedges — + a case awaiting something that never resolves — which the execution + budget cannot see (no wasm runs while wedged). + +Either trip fails the case with provenance `limit-exceeded()` +and abandons the instance: the same containment as a trap, and the +next case gets a fresh session. Layer any SUT-internal operation +timeout *below* the runner's guards, so genuine failures classify as +`failed` outcomes with their own diagnostics and only true hangs trip +the guard. Emit phase markers in diagnostics so an abandoned case's +prefix identifies the hung phase. (Budget escalation — a keepalive +that lets a legitimately slow case extend its lease — is #47's +territory.) + +## No retries + +No execution path retries anything, and there is deliberately no knob: +a second attempt masks exactly the flake a test system exists to +report. For network and timing nondeterminism, prefer deterministic +simulation environments over retries. + +## Reporting obligations + +- **Write-through**: forward diagnostics to the transport as they + arrive. Anything buffered inside a wasm runner core dies with the + store on a trap; the composed wasi:cli core streams stdout + write-through for exactly this reason. +- **Concurrent drain**: a runner observing a case's diagnostics stream + must consume it concurrently with `run` — a non-draining observer + wedges sync-lifted suites (finding 3) — and must drain biased, + taking completed messages before the verdict, or in-flight + diagnostics are lost with the future (finding 5). +- **Census truth**: one event per census case, in census order, ending + with the terminator. Cases the run never reached are the fold's to + synthesize (`not-reached`), not the runner's to invent. + +## Selection is not capability + +Two different absences, never conflated: + +- **`not-applicable`** is *capability*: the target's manifest declares + a feature missing, and tag scheduling excludes the case. It is a + property of the target and is reported truthfully regardless of any + filter. +- **`deselected`** is *selection policy*: cost tiers, smoke subsets on + expensive targets, a dev loop's `--only`. The runner reports the + unselected remainder of the census as `deselected` (never executed, + no provenance), so subset runs still fold and aggregate cleanly with + the subsetting visible in the results — restricting a + browser-boot-per-case target to a smoke tier is policy someone chose, + and the matrix shows it as such. + +Capability takes precedence: a tags-excluded case stays +`not-applicable` even when it also falls outside the selection — +aggregation's applicability policing rejects `deselected` where the +manifest says the case could never have run (selection must not hide +capability). An empty selection (`--only` matching nothing) is a run +error, not a vacuous green. (The JS leg's `only` option predates this +rule and still omits filtered cases instead of reporting them — +tracked in #89.) From 379564e40a58f88f77d3df0743b82c45a82b5a46 Mon Sep 17 00:00:00 2001 From: Lann Martin Date: Tue, 11 Aug 2026 21:31:12 -0400 Subject: [PATCH 2/2] =?UTF-8?q?findings=2025:=20wasm-tools=20strip=20measu?= =?UTF-8?q?red=20too=20=E2=80=94=200.38%,=20and=20a=20trap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prompted by review discussion: the spike had bounded DCE (wasm-opt) but not custom-section stripping. Measured on the wizened bench artifact: 4.9KB / 0.38% — [profile.release] strip = true already removed the heavy sections at build time. And the default invocation is actively harmful here: wasm-tools strip's keep list is name/component-type/dylink.0 only, so component-test:tags@0.1 dies — verified: the stripped artifact still executes (execute-everything, with the runner's no-inventory note), but --missing hard-errors and lock --check fails with the "sections stripped" message that anticipated exactly this. --- docs/findings.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/findings.md b/docs/findings.md index 011c5ed..9e12670 100644 --- a/docs/findings.md +++ b/docs/findings.md @@ -244,5 +244,13 @@ bench-suite artifact built with its `wizer-init` feature. through the snapshotted case table — only the one-shot registration driver dies, and it is small. CoW already makes the on-disk growth near-free at runtime (finding 23); transport - compression covers the wire. + compression covers the wire. `wasm-tools strip` is no substitute + and a trap besides: it *does* process components, but saves only + 4.9KB (0.38% — `[profile.release] strip = true` already removed + the heavy custom sections at build time), and its default keep + list is `name`/`component-type`/`dylink.0` only, so it deletes + `component-test:tags@0.1` — scheduling and `lock --check` break + exactly as the CLI's "sections stripped" error anticipates. If it + must run, name sections explicitly with `--delete`; never the + default form.