Runner execution policy: the consolidated doc; --only reports deselected - #90
Merged
Conversation
docs/runner-policy.md is the document #22 owed — the granularity/ replication/budget/reporting doctrine in one place, carrying both measurement campaigns (the webcrypto 11.5k corpus ladder from the issue thread, the 10k-synthetic wizen numbers from findings 22-24), the "K=1 at scale -> wizen" row and the JS-leg anti-recommendation (#85), and the selection-vs-capability rule. Guidance, not contract: the L1 surface permits every granularity; README commitments win on conflict. Writing "cost tiers are deselected, not not-applicable" forced the rule to become true: no runner emitted deselected, so a subset run could not participate in aggregation at all (omitted census = failed coverage). --only now reports the unselected remainder of the census as deselected wire events — never executed, no provenance, detail naming the selection — so filtered runs fold and aggregate cleanly with the subsetting visible as policy. Capability wins over selection: a tags-excluded case stays not-applicable even outside the filter, which is what aggregate's applicability policing requires of scheduled streams (deselected on a non-applicable case remains the error it should be — selection must not hide capability). Human mode stays silent per deselected case (a narrow --only over a large corpus must not bury the selected output); the summary line grows a ", N deselected" segment only when nonzero, so all goldens stay byte-identical. Empty selection stays a run error. The JS leg's `only` still omits (predates the rule): filed #89, noted in the doc. Also rides along: the #85 wasm-opt spike, concluded negative (findings.md 25). binaryen v124 refuses components outright (binaryen#6728), and the bound holds regardless: the wizened bench artifact's growth is snapshot data (1.18MB across 10,002 segments vs 93KB code), so total code deletion reclaims <=7.2%; measured on the extracted core module, wasm-opt -Oz saves 19.5KB (1.5%) — the case bodies stay live through the snapshotted case table, so "init-only code goes dead" is immaterial. Verified: just check, just all (all goldens byte-identical), just test-wasm (new: --only census test; fold coverage with deselected rows; aggregate selection-vs-capability both directions). Review caught and fixed a bad multiple in the doc: full isolation is ~1.7x the incumbent adapter but ~3x the fastest shared-instance mode; both now stated with their referents.
Prompted by review discussion: the spike had bounded DCE (wasm-opt) but not custom-section stripping. Measured on the wizened bench artifact: 4.9KB / 0.38% — [profile.release] strip = true already removed the heavy sections at build time. And the default invocation is actively harmful here: wasm-tools strip's keep list is name/component-type/dylink.0 only, so component-test:tags@0.1 dies — verified: the stripped artifact still executes (execute-everything, with the runner's no-inventory note), but --missing hard-errors and lock --check fails with the "sections stripped" message that anticipated exactly this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22. The remaining substantive box on #85 (the policy doc), plus the wasm-opt spike box, concluded negative.
docs/runner-policy.md
The consolidated runner-implementer document #22 owed, gathering what was split across README commitments / AGENTS invariants / runner rustdoc / the issue thread:
Guidance, not contract: the doc defers to README commitments and the frozen L1 surface explicitly. Cross-linked from the runner rustdoc, the sample-suite README ("Large suites" now points here as promised in the #85 thread), and AGENTS.md.
--onlynow reportsdeselected(the doc forced the rule to become true)#22's "cost tiers are
deselected, notnot-applicable" was unexercised — no runner emitted it, so a subset run couldn't participate in aggregation at all (omitted census = failed coverage). Now:deselectedevents (never executed, no provenance, detail names the selection) — subset runs fold and aggregate cleanly with the subsetting visible as selection policynot-applicableeven outside the filter (aggregate's applicability policing on scheduled streams requires exactly this; unit-tested in both directions — selection tolerated, capability can't hide behind selection), N deselectedsegment only when nonzero — all goldens byte-identicalonlypredates the rule and still omits: filed JS-legonlyfilter omits unselected cases instead of emitting deselected #89, noted in the docwasm-opt spike (#85), concluded negative — findings.md 25
binaryen v124 refuses components (binaryen#6728), and the bound holds regardless: the wizened bench artifact's growth is snapshot data (1.18MB / 10,002 segments vs 93KB code), so total code deletion reclaims ≤7.2%; measured on the extracted core module,
wasm-opt -Ozsaves 19.5KB (1.5%). The "init-only code goes dead" hypothesis is immaterial — case bodies stay live through the snapshotted case table.Verification
just check,just all(goldens byte-identical),just test-wasmincl. new tests: runner--onlycensus emission + precedence + empty-selection, fold coverage over deselected rows, aggregate selection-vs-capability. Independent review verified every number in the doc against findings 19–25 and the #22 thread; it caught one bad multiple (~1.7× had the wrong referent — now stated against both the incumbent adapter and the fastest shared-instance mode).