fix(test): prop_greedy_is_argmax reddened at random on tied logits - #2544
fix(test): prop_greedy_is_argmax reddened at random on tied logits#2544noahgift wants to merge 1 commit into
Conversation
7ba3db5 to
ec2de54
Compare
`workspace-test` failed on #2540 — a PR that removes a CLI route and touches nothing near sampling: prop_greedy_is_argmax panicked at kernels/sampling.rs:315 left: 0, right: 1 minimal failing input: logits = [9.086451, 9.086451] successes: 127 let argmax = logits.iter().enumerate() .max_by(|a, b| a.1.partial_cmp(b.1).unwrap()).unwrap().0; prop_assert_eq!(result, argmax); `Iterator::max_by` returns the LAST maximum. `greedy_scalar` keeps the FIRST (`if v > best_val`, strictly greater). On a tie they disagree BY CONSTRUCTION, so the test failed whenever proptest happened to draw duplicate maxima — 127 successes before it did here. Nondeterministic, seed-dependent, and able to red ANY pull request. `greedy_scalar` is correct: first-wins is the conventional argmax tie-break and it is deterministic, which sampling requires. let best = logits.iter().copied().fold(f32::NEG_INFINITY, f32::max); prop_assert_eq!(logits[result], best); // greedy returns A maximum let first_max = logits.iter().position(|&v| v == best).unwrap(); prop_assert_eq!(result, first_max); // and specifically the FIRST This is STRICTLY STRONGER than what it replaces: it pins the tie-break as part of the contract instead of inheriting whatever `max_by` happens to do, and it can be stated on ties at all — which the old form could not. greedy_scalar = 0 OLD max_by().0 = 1 -> old assert FAILS NEW value match = true NEW first_max = 0 -> new assert PASSES cargo test -p aprender-contracts --lib 1435 passed, 0 failed cargo fmt --check rc=0 Same class as the SIGPIPE false-red in check_beats_gated (#2542): a required check that goes red at random is worse than one that cannot go red at all, because it trains everyone to re-run until green — and a real failure then rides through on the retry that happens to pass.
ec2de54 to
40dd679
Compare
|
Superseded by #2613, the 0.64.0 integration batch. This PR's commits are merged into Why batched rather than landed individually: one Batching also found four defects that were invisible to every individual PR — most Closing now, deliberately: an open PR that merges first moves #2613's base and forces |
workspace-testfailed on #2540 — a PR that removes a CLI route and touches nothing near sampling:Cause — the property was over-specified, and only wrong on ties
Iterator::max_byreturns the last maximum.greedy_scalarkeeps the first (if v > best_val, strictly greater). On a tie they disagree by construction — so the test failed whenever proptest happened to draw duplicate maxima. It took 127 successes to hit one here.Nondeterministic, seed-dependent, and able to red any PR.
greedy_scalaris not the problem: first-wins is the conventional argmax tie-break, and it is deterministic — which sampling requires.Fix — assert what must hold, and pin the tie-break
This is strictly stronger than what it replaces. It pins the tie-break as part of the contract rather than inheriting whatever
max_byhappens to do, and it can be stated on ties at all — which the old form could not.Verified on the exact failing input
Not "the tests pass now" — the old assertion is shown failing and the new one passing on the input CI actually produced.
Why this matters beyond one PR
Same class as the SIGPIPE false-red in
check_beats_gated(#2542): a required check that goes red at random is worse than one that cannot go red at all. It trains everyone to re-run until green, and a genuine failure then rides through on the retry that happens to pass.Two such flakes surfaced in a single day, both in required checks. Worth watching for a third.