test(e2e): pin the second README-walkthrough batch (EAI-8046..8049) - #292
Open
fredespi wants to merge 6 commits into
Open
test(e2e): pin the second README-walkthrough batch (EAI-8046..8049)#292fredespi wants to merge 6 commits into
fredespi wants to merge 6 commits into
Conversation
Add one behavioural scenario per defect, each asserting the CORRECT behaviour and registered as an xfail in expectations.toml citing its EAI id, so the mock lane stays green now and the row goes stale (XPASS) the day the bug is fixed. No product behaviour changed. - EAI-8046: `rocm chat` ignores a piped prompt (reads no stdin, sends no request, exits 0 with a status summary). New run_rocm_with_stdin helper since Command::output gives the child a null stdin; the load-bearing assertion is that the mock received the prompt, not stdout. - EAI-8047: automations list publishes no usable identifier, so the checks it lists cannot be enabled from what it shows. The scenario derives each id from the listing rather than hard-coding the real ids. - EAI-8048: `dash --replay <missing>` opens the interactive view instead of refusing; the step imposes its own deadline so the known hang xfails promptly rather than burning the CI budget. - EAI-8049: the engine shell's activation hint names a path that does not exist, because the recorded env_path already includes the interpreter directory. Phrased as path-exists so the row goes stale on any fix. All four re-confirmed in a plain Linux container against this branch's binary before writing the scenarios. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/expectations.toml # tests/e2e-cucumber/tests/e2e.rs
Address pre-PR review: each of the three step assertions pinned the bug's present symptom rather than the behaviour that must hold, so a valid fix could stay xfailed and the row would never go stale. - EAI-8047: parse the identifier the listing explicitly exposes (inline [id]/(id) on the header, or an `id:`/`identifier:` detail line), keeping the display-name slug only as the current-broken-output fallback. A fix that publishes the id now flips the row to XPASS. - EAI-8048: also assert the interactive view never opened, not just a prompt non-zero exit. Adds a latched entered_alternate_screen() to the PTY driver so an "open the TUI, then exit non-zero" impl no longer satisfies the contract. - EAI-8046: assert the user-role message content rather than messages[0], so a fix that prepends a system message still XPASSes. Mock lane unchanged: still 6 xfail / 0 XPASS; each scenario still fails on its assertion step alone. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Round-2 review: the latch checked the parser's alternate_screen() state only after processing a whole PTY read, so a chunk carrying both the enter (?1049h) and leave (?1049l) sequences left it false; and wait_for_any_exit returned before the reader drained, so a still-buffered enter could be missed. Either produced a false negative that would let a fixed dashboard open the TUI and still pass the "must not open the interactive view" half. - Detect the ?1049h enter sequence in the RAW byte stream, with a one-short-of-marker carry-over so a boundary-split sequence is still found. Independent of what follows it in the same read. - Drain the reader (bounded by DRAIN_TIMEOUT) after the child exits, before the caller reads the latch — same post-exit drain wait_for_screen uses. Mock lane unchanged: 6 xfail / 0 XPASS. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
fredespi
force-pushed
the
e2e-pin-the-contracts-for-the-second
branch
from
August 20, 2026 12:58
450d4c8 to
dd733fe
Compare
The CI Clippy (e2e harness + steps) job runs clippy -p e2e-cucumber --test e2e -D warnings, which the local container gate did not (its clippy step excludes e2e-cucumber). Four lints surfaced: - future-not-send: drain_reader took &self, and a shared &TuiSession held across .await is not Send (the struct holds !Sync PTY handles), poisoning every step awaiting it. Take &mut self like the sibling waits. - collapsible-if, or-fun-call (automations_steps) and map-unwrap-or (engines_steps): mechanical rewrites, no behaviour change. Verified with the exact CI commands in a Linux container: both clippy invocations clean, e2e mock lane unchanged (6 xfail / 0 XPASS). Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test-only. Pins the correct behaviour for the second README-walkthrough batch of defects (EAI-8046–8049, tracked upstream as EAI-8050), following the same shape as #216: one behavioural scenario per defect, each asserting the CORRECT behaviour and registered as an
xfailinexpectations.tomlciting its EAI id. The mock lane stays green now, and each row goes stale (XPASS) the day its bug is fixed. No product behaviour changed.All four defects were reproduced by hand on an MI300X app-dev pod, then re-confirmed in a plain Linux container against this branch's binary before the scenarios were written. Each scenario was verified to fail on its assertion step (setup steps green) for the stated reason — not a miswiring.
What each ticket now pins
rocm chatignores a piped prompt.chat.featurescenario 8 pipes a prompt with no--promptand asserts the model actually received it. The load-bearing assertion is the mock's received request (the bug's signature is that no request is made), not stdout. Needed a newrun_rocm_with_stdinhelper —Command::outputgives the child a null stdin, so no existing step could reach the piped path.rocm automations listcannot be enabled from what the listing shows. Newautomations.feature+ steps: the scenario derives each check's identifier from the listing (an explicitly-exposed id if present, else the display-name slug as today's fallback) and assertsautomations enable <id>succeeds. Deriving from the listing — rather than hard-coding the real ids — is the contract.rocm dash --replay <missing>opens the interactive dashboard instead of refusing.dash.featurescenario 10 drives it under a PTY, imposes its own deadline (the current behaviour hangs), and asserts both a prompt non-zero exit and that the alternate-screen interactive view never opened.rocm engines shellnames a path that does not exist (the recordedenv_pathalready includes the interpreter directory, so the hint doubles that segment).engine_shell.featurescenario 2 runs it under a PTY and asserts the hint names a file that exists — phrased as path-exists so the row goes stale on any correct fix.Notes
run_rocm_with_stdin, a newautomations_stepsmodule, and PTY-driver support (wait_for_any_exit,screen_logical_lines, and a race-free latch that detects the alternate-screen enter sequence in the raw byte stream with a bounded post-exit reader drain).expectations.tomlrows are allwhen = {}(platform-independent), each with a comment stating the scope and the removal condition.Test plan
e2emock job is green: pass-or-xfail, no XPASS. (Locally on a native-Linux container: clippy-D warningsclean, e2e-cucumber lib tests pass, mock lane reconciles to 6 xfail / 0 XPASS / 0 unexpected failures.)