test(e2e): pin regression coverage for verified-but-unpinned CLI behaviours (EAI-8072) - #293
Open
fredespi wants to merge 3 commits into
Open
test(e2e): pin regression coverage for verified-but-unpinned CLI behaviours (EAI-8072)#293fredespi wants to merge 3 commits into
fredespi wants to merge 3 commits into
Conversation
…rs (EAI-8072) Add GREEN mock-lane cucumber scenarios protecting CLI behaviours that were manually verified correct during the walkthrough but had no scenario guarding them. All run on the no-GPU/no-network mock lane every PR and assert host-invariant contracts (exit code + stable output) verified against the real Linux binary. - config mutations: default engine/runtime set+clear, telemetry, permissions, set-engine target requirement, provider enable/disable, local-provider guard, and provider-key save failing without leaking the key - logs: search match-count reporting, absent-term, service+search conflict - automations: enable/disable watcher mode confirmation, unknown-watcher refusal (background daemon spawn suppressed via a planted live-pid runtime state) - runtime lifecycle: activate/rollback/uninstall/import over planted read-only tarball runtimes (no SDK download or GPU needed) - update: report distinguishes configured from not-configured feeds (run with no runtimes so it stays offline) - networking: public-bind refused pre-flight without --allow-public-bind - serve: --runtime-id/--env-id mutually-exclusive selector conflict Adds a run_rocm_with_stdin helper for stdin-driven commands. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…ertions - Remove private internal workflow identifiers from public comments and fixture tokens (AGENTS.md permits EAI ids only): use EAI-8072 / neutral e2e tokens. - Make the provider-key no-secure-storage premise deterministic: force the Linux Secret Service unreachable via a bogus DBUS_SESSION_BUS_ADDRESS and scope the scenario @requires-os:linux, so it no longer relies on the runner happening to lack a session bus (Windows/macOS credential stores cannot be disabled the same way). Still verifies the key is never echoed. - Bind each update-feed status to its own feed line so a status attributed to the wrong feed fails, instead of matching feed names and statuses as independent substrings anywhere in the output. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Build the planted log body with writeln! (std::fmt::Write) instead of push_str(&format!(...)), and hoist the import to the top to avoid items_after_statements. Matches CI's pedantic/nursery clippy config, which the per-package check did not surface. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
fredespi
force-pushed
the
e2e-regression-coverage-for-walkthrough
branch
from
August 20, 2026 13:31
5953765 to
98ad79d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end regression scenarios that pin CLI behaviours which were manually verified as correct during the walkthrough but had no automated test guarding them. Everything here runs on the no-GPU/no-network mock lane (every PR) and asserts host-invariant contracts (exit code + stable output), so a regression surfaces in CI rather than in the field. Contracts were grounded by running the real Linux
rocmbinary, not read from source.Relates to EAI-8072 (linked from EAI-8024, EAI-7409, EAI-7404).
Changes
New feature files and step definitions:
set-enginerequiring a target; provider enable/disable; the local provider not being toggleable as a cloud provider; and saving a provider key failing without leaking the key when secure storage is unavailable.--searchreporting the matching-line count, an absent term reporting no matches, and the--service/--searchconflict being refused.--replace. Driven with planted read-only runtimes, so no SDK download or GPU is needed.--allow-public-bindis refused pre-flight (the EAI-7409 contract).--runtime-idand--env-idare rejected as a mutually-exclusive selector conflict.Also adds a small
run_rocm_with_stdintest helper (stdin + extra env) for commands that read from stdin.Test plan
engine-shell-marks-the-prompthost-environment failure, which also fails on unmodifiedmainin the same container and is green on CI's runner.cargo clippy --testsclean under-D warnings.Scope
This PR covers the mock-lane (no-GPU) regression set. The remaining GPU-only behaviours from the ticket (public-bind live auth/401, serve selector effects reflected in the serve plan,
--tool-call-parser,--no-smoke-test) require a live GPU serve to validate and are tracked separately.