You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up from #86's review (non-blocking finding #3).
The CLI bakes in size-optimized builds of components/runner-cli and components/provider (crates/component-test-cli/embedded/, regenerated by just embed-update) as the compose-runner/run defaults. Freshness is enforced behaviorally: just verify-cli diffs the embedded artifacts' composed output against the Path-2 goldens. Byte-comparing a rebuild against the committed copies is deliberately off the table — builds are not reproducible across environments (#44).
The gap: a change to runner-cli or provider whose behavior the sample-suite goldens don't exercise — trap handling in the composed core, diagnostics edge cases, provider stream corner cases, JSONL fields only emitted on unusual paths — leaves embedded/ stale with no gate tripping. Today's mitigation is the embed-update reminder in AGENTS.md and the justfile recipe comments.
Options, in increasing machinery:
Accept + document (status quo; this issue is the record).
Widen the goldens: add a composed-run leg over a second suite to verify-cli. Note the trap fixture doesn't transplant: the composed core is single-instance execute-everything, so fixture-suite's deliberate trap would poison the whole run — which is exactly why Path 2 has no fixture leg today. A non-trapping fixture (generated rows + tagged pair) would still widen coverage of the wire format.
Advisory rebuild job in CI: rebuild with the embed profile and warn (never fail, per Lockfile artifact-sha256 binding vs non-reproducible suite builds #44) when bytes differ from the committed copies. Noisy across toolchain bumps; possibly still a useful nudge on PRs touching those components.
Not blocking anything — filed so the decision is recorded rather than re-derived.
Follow-up from #86's review (non-blocking finding #3).
The CLI bakes in size-optimized builds of components/runner-cli and components/provider (
crates/component-test-cli/embedded/, regenerated byjust embed-update) as thecompose-runner/rundefaults. Freshness is enforced behaviorally:just verify-clidiffs the embedded artifacts' composed output against the Path-2 goldens. Byte-comparing a rebuild against the committed copies is deliberately off the table — builds are not reproducible across environments (#44).The gap: a change to runner-cli or provider whose behavior the sample-suite goldens don't exercise — trap handling in the composed core, diagnostics edge cases, provider stream corner cases, JSONL fields only emitted on unusual paths — leaves
embedded/stale with no gate tripping. Today's mitigation is theembed-updatereminder in AGENTS.md and the justfile recipe comments.Options, in increasing machinery:
fixture-suite's deliberate trap would poison the whole run — which is exactly why Path 2 has no fixture leg today. A non-trapping fixture (generated rows + tagged pair) would still widen coverage of the wire format.Not blocking anything — filed so the decision is recorded rather than re-derived.