Make E2E scenario responses.feature:353 resilient to non-deterministic LLM output #3081

What happened

On PR #2995, run 39968648001 failed in the E2E: library / ci / authorized job because responses.feature:353 ("forks to a new conversation when previous_response_id is not the latest turn") asserted that the response body contains the word "middle". The assertion failed on all 5 retries (the scenario is tagged @flaky). The test passed on a later commit (run 40922445615 on Oct 29) with no relevant change, confirming non-determinism. The E2E Tests workflow shows a ~20% failure rate in recent history (2 failures out of 10 runs), and this scenario is a likely contributor.

What could go better

Asserting on specific words in LLM-generated output is inherently non-deterministic. The @flaky tag and retry mechanism mask the underlying issue rather than fixing it. The test should validate deterministic behavior (correct HTTP status, response structure, conversation fork semantics) rather than checking for a specific word in the model's free-text answer. Confidence: moderate — I confirmed the failure/pass pattern across two runs on different commits with no relevant code change, but the exact contribution of this scenario to the overall E2E failure rate would need further investigation.

Proposed change

In tests/e2e/features/responses.feature, modify scenario at line 353 to assert on deterministic properties of the response (HTTP status code, response JSON structure, previous_response_id field value, conversation fork behavior) rather than asserting that the body contains a specific word from the LLM's free-text answer. If the test intent is to verify the content is contextually related to a prior turn, use a structural check (e.g., verify the response references the correct conversation history) instead of a string-contains assertion on LLM output. Remove the @flaky tag once the assertion is deterministic.

Validation criteria

The modified scenario passes consistently on 10 consecutive E2E test runs without needing the @flaky retry mechanism. The @flaky tag is removed from the scenario.

Related issues checked

Searched: flaky e2e test responses feature; responses feature middle non-deterministic LLM
No overlapping issues found.

Generated by retro agent from #2995