Skip to content

fix(test): prop_greedy_is_argmax reddened at random on tied logits - #2544

Closed
noahgift wants to merge 1 commit into
mainfrom
fix/greedy-argmax-tie-flake
Closed

fix(test): prop_greedy_is_argmax reddened at random on tied logits#2544
noahgift wants to merge 1 commit into
mainfrom
fix/greedy-argmax-tie-flake

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

workspace-test failed on #2540 — a PR that removes a CLI route and touches nothing near sampling:

prop_greedy_is_argmax panicked at kernels/sampling.rs:315
  left: 0, right: 1
  minimal failing input: logits = [9.086451, 9.086451]
  successes: 127

Cause — the property was over-specified, and only wrong on ties

let argmax = logits.iter().enumerate()
    .max_by(|a, b| a.1.partial_cmp(b.1).unwrap()).unwrap().0;
prop_assert_eq!(result, argmax);

Iterator::max_by returns the last maximum. greedy_scalar keeps the first (if v > best_val, strictly greater). On a tie they disagree by construction — so the test failed whenever proptest happened to draw duplicate maxima. It took 127 successes to hit one here.

Nondeterministic, seed-dependent, and able to red any PR.

greedy_scalar is not the problem: first-wins is the conventional argmax tie-break, and it is deterministic — which sampling requires.

Fix — assert what must hold, and pin the tie-break

let best = logits.iter().copied().fold(f32::NEG_INFINITY, f32::max);
prop_assert_eq!(logits[result], best);        // greedy returns A maximum

let first_max = logits.iter().position(|&v| v == best).unwrap();
prop_assert_eq!(result, first_max);           // and specifically the FIRST

This is strictly stronger than what it replaces. It pins the tie-break as part of the contract rather than inheriting whatever max_by happens to do, and it can be stated on ties at all — which the old form could not.

Verified on the exact failing input

greedy_scalar        = 0
OLD max_by().0       = 1   ->  old assert FAILS
NEW value match      = true
NEW first_max        = 0   ->  new assert PASSES

cargo test -p aprender-contracts --lib   1435 passed, 0 failed
cargo fmt --check                        rc=0

Not "the tests pass now" — the old assertion is shown failing and the new one passing on the input CI actually produced.

Why this matters beyond one PR

Same class as the SIGPIPE false-red in check_beats_gated (#2542): a required check that goes red at random is worse than one that cannot go red at all. It trains everyone to re-run until green, and a genuine failure then rides through on the retry that happens to pass.

Two such flakes surfaced in a single day, both in required checks. Worth watching for a third.

@noahgift
noahgift force-pushed the fix/greedy-argmax-tie-flake branch from 7ba3db5 to ec2de54 Compare August 20, 2026 19:01
`workspace-test` failed on #2540 — a PR that removes a CLI route and
touches nothing near sampling:

    prop_greedy_is_argmax panicked at kernels/sampling.rs:315
      left: 0, right: 1
      minimal failing input: logits = [9.086451, 9.086451]
      successes: 127

    let argmax = logits.iter().enumerate()
        .max_by(|a, b| a.1.partial_cmp(b.1).unwrap()).unwrap().0;
    prop_assert_eq!(result, argmax);

`Iterator::max_by` returns the LAST maximum. `greedy_scalar` keeps the
FIRST (`if v > best_val`, strictly greater). On a tie they disagree BY
CONSTRUCTION, so the test failed whenever proptest happened to draw
duplicate maxima — 127 successes before it did here.

Nondeterministic, seed-dependent, and able to red ANY pull request.
`greedy_scalar` is correct: first-wins is the conventional argmax
tie-break and it is deterministic, which sampling requires.

    let best = logits.iter().copied().fold(f32::NEG_INFINITY, f32::max);
    prop_assert_eq!(logits[result], best);        // greedy returns A maximum
    let first_max = logits.iter().position(|&v| v == best).unwrap();
    prop_assert_eq!(result, first_max);           // and specifically the FIRST

This is STRICTLY STRONGER than what it replaces: it pins the tie-break as
part of the contract instead of inheriting whatever `max_by` happens to
do, and it can be stated on ties at all — which the old form could not.

    greedy_scalar        = 0
    OLD max_by().0       = 1   -> old assert FAILS
    NEW value match      = true
    NEW first_max        = 0   -> new assert PASSES

    cargo test -p aprender-contracts --lib   1435 passed, 0 failed
    cargo fmt --check                        rc=0

Same class as the SIGPIPE false-red in check_beats_gated (#2542): a
required check that goes red at random is worse than one that cannot go
red at all, because it trains everyone to re-run until green — and a real
failure then rides through on the retry that happens to pass.
@noahgift

Copy link
Copy Markdown
Contributor Author

Superseded by #2613, the 0.64.0 integration batch.

This PR's commits are merged into batch/release-0-64-0 verbatim (--no-ff, never rebased),
and #2613's body carries the full provenance table — PR number, branch, merged head SHA, and
the issues each closes — so the detail survives the squash.

Why batched rather than landed individually: one workspace-test run is ~58 minutes on a
shared box. Thirteen PRs cost thirteen runs whether they go serially or in parallel; one
integration branch costs one. The same approach landed 24 branches previously.

Batching also found four defects that were invisible to every individual PR — most
notably the README contract count: #2548, #2549 and #2587 each add exactly one contract, each
is individually correct at 1779, and three +1s collide on one literal (correct value 1781).
That is the exact class that killed the previous batch.

Closing now, deliberately: an open PR that merges first moves #2613's base and forces
another full run. This is reversible and the branch is untouched — reopen if #2613 is
abandoned.

@noahgift noahgift closed this Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant