Skip to content

Embedded CLI components: freshness gate only sees sample-suite-visible behavior #88

Description

@lann

Follow-up from #86's review (non-blocking finding #3).

The CLI bakes in size-optimized builds of components/runner-cli and components/provider (crates/component-test-cli/embedded/, regenerated by just embed-update) as the compose-runner/run defaults. Freshness is enforced behaviorally: just verify-cli diffs the embedded artifacts' composed output against the Path-2 goldens. Byte-comparing a rebuild against the committed copies is deliberately off the table — builds are not reproducible across environments (#44).

The gap: a change to runner-cli or provider whose behavior the sample-suite goldens don't exercise — trap handling in the composed core, diagnostics edge cases, provider stream corner cases, JSONL fields only emitted on unusual paths — leaves embedded/ stale with no gate tripping. Today's mitigation is the embed-update reminder in AGENTS.md and the justfile recipe comments.

Options, in increasing machinery:

  1. Accept + document (status quo; this issue is the record).
  2. Widen the goldens: add a composed-run leg over a second suite to verify-cli. Note the trap fixture doesn't transplant: the composed core is single-instance execute-everything, so fixture-suite's deliberate trap would poison the whole run — which is exactly why Path 2 has no fixture leg today. A non-trapping fixture (generated rows + tagged pair) would still widen coverage of the wire format.
  3. Advisory rebuild job in CI: rebuild with the embed profile and warn (never fail, per Lockfile artifact-sha256 binding vs non-reproducible suite builds #44) when bytes differ from the committed copies. Noisy across toolchain bumps; possibly still a useful nudge on PRs touching those components.

Not blocking anything — filed so the decision is recorded rather than re-derived.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciGitHub Actions packagingtoolingCLI, lockfiles, composition helpers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions