[thegdsks]

Publishes round 2 of the pre-registered agent test, with the decision rule applied as written. Refs #78.

Result

The rule fails. Steering e does not become the default.

• 650 main sessions on four repos (control, a, e, and e with small repo mode on the smallest). Notional spend 15.77 USD, no errors, no session over its cap.
• Cost against control, pooled over all four repos: a 1.17 [1.11, 1.24], e 1.09 [1.04, 1.14]. Without the contaminated featuredrop repo, e is 1.05 [1.00, 1.11].
• Per repo (e): levelrail 0.87 [0.81, 0.93], theauth-go 1.05 [0.95, 1.16], featuredrop 1.21 [1.14, 1.29], theauth 1.42 [1.36, 1.48].
• e costs less than a on cost at equal correctness (0.93 [0.90, 0.95]).
• Correctness is level: 97.5% control, 97.0% a, 97.5% e.
• Tool calls 0.76 and turns 0.83 of control. Grep and glob calls 0.53 with e, 0.22 with a.
• Rule outcomes: P1 fail, G2 fail, S1 fail by 0.01 on the cost bound; P2, P3, P4 pass; G1 passes but says nothing (control was already 100% on trace).

What this means

Stacklit lowers cost on the large repo in both rounds (round 1: 0.78 [0.70, 0.86]; round 2: 0.87 to 0.89). On the smaller repos tested it costs more. No general cost saving is claimed.

Notes

• Memorisation check: featuredrop was contaminated (3 of 6 answered with no tools); results are shown with and without it.
• Grading defects are listed and reported as a separate adjusted column, not used by the rule.
• Round 1 files are unchanged. Round 2 outputs live in bench/agent-ab/round2/ and results-round2/.
• Raw records were scanned for local paths and secrets.

[thegdsks]

Superseded by the combined PR that also corrects the site claims.