feat(adaptive): measure whether the learning is real - #68
Merged
Conversation
This crate accumulates knowledge in five ways — a ledger, lessons with help rates, workflow scores, repaired variants, a promotion gate — and had no way to answer the only question that matters about any of it: does an episode go better because earlier episodes happened? Derived from the eval design in a sibling PoC, which had already worked out the hard part: **a success rate cannot answer it.** On ten unrelated tasks solved once each, a learning loop and a plain retry loop produce identical numbers. Neither can a single arm — "it solved six of six" is compatible with the ledger being decorative. What answers it is a family of related tasks run twice in the same order, differing only in whether anything survives between episodes, and the quantity to read is the slope: attempts-to-success falling as the family progresses. Two pieces, because that experiment needed two things the crate lacked. **`evals::slope`** — `Episode`, two arms, the fit, and a comparison that refuses to call a bend a win unless the arm also converges (twenty attempts falling to ten loses to a flat two). `Episode::of` reads an episode off the ledger rows the loop already writes, rather than off a parallel bookkeeping nobody uses. **`evals::arms`** — the control. A loop with learning off is *not* a loop with its ledger removed: this ledger is load-bearing within an episode, holding the exclusion list that stops attempt four repeating attempt two. `Forgetful` blanks exactly the cross-episode reads — lessons, evidence, workflow scores, repair lineage — and leaves the rest. Writes still happen and are never read, so the control arm still pays for consolidation; an arm that skipped the writes would look cheaper for a reason unrelated to learning. Three departures from the design as derived, each one a defect the running example found: **An arm that solved nothing must not tie with an arm that solved four.** The example produced exactly that and reported no evidence, because both slopes read `0.0` — one meaning "flat", the other meaning "two points are needed for a line and there was one". Going from never to once is the largest fall in attempts-to-success there is. `trend()` and `mean_attempts()` are now `Option`, the report emits `null` rather than a flattering zero, and `compare` returns a `Verdict` naming which test it won on: `SolvedMore`, `Converged`, or `NoEvidence`. `SolvedMore` is guarded by "no worse per solve" so it cannot decay into the success-rate metric this design exists to avoid. **A wrong answer the judge accepted is not a success.** `Episode` carries an outside check separately from the verdict, and `Outcome` names all four states — solved, refused (right, judge would not accept), wrong (judge accepted, world refuted), failed. `None` for "not checked" is distinct from `Some(true)`, so a report can say which runs were taken at the judge's word. **A failed episode is a gap, not a shift.** Wins are fitted at their original positions, so a wasted episode makes the measured convergence shallower — the family took an extra episode to cover the same ground. The test states the direction explicitly because the intuition runs the other way. No task set ships here. A family has to be chosen for shared technique and checkable answers, and belongs to whoever runs the eval; a list baked into a host-agnostic crate would be an opinion about somebody else's domain. The example carries a simulated one, and says so.
|
Important Review skippedThis review includes 6 billable files. This on-demand review is free during your promotion. Your included review limit has been reached. Run
⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
How this change flows0 changed behaviours across 6 relationships. 5 surrounding behaviours are shown (60 graph nodes walked). 30 further behaviours left out to keep the diagram readable. flowchart LR
n0["ep"]:::impacted
n1["series"]:::impacted
n2["experiment"]:::impacted
n3["...pted_and_the_world_refuted_is_not_a_solve"]:::impacted
n4["...ut_refused_is_reported_apart_from_failing"]:::impacted
n1 -->|calls| n0
n2 -->|calls| n0
n3 -->|calls| n0
n3 -->|tests| n0
n4 -->|calls| n0
n4 -->|tests| n0
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge. |
tinysweeper
Bot
added
the
priority: p3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(adaptive): measure whether the learning is real
This crate accumulates knowledge in five ways — a ledger, lessons with
help rates, workflow scores, repaired variants, a promotion gate — and
had no way to answer the only question that matters about any of it: does
an episode go better because earlier episodes happened?
Derived from the eval design in a sibling PoC, which had already worked
out the hard part: a success rate cannot answer it. On ten unrelated
tasks solved once each, a learning loop and a plain retry loop produce
identical numbers. Neither can a single arm — "it solved six of six" is
compatible with the ledger being decorative. What answers it is a family
of related tasks run twice in the same order, differing only in whether
anything survives between episodes, and the quantity to read is the
slope: attempts-to-success falling as the family progresses.
Two pieces, because that experiment needed two things the crate lacked.
evals::slope—Episode, two arms, the fit, and a comparison thatrefuses to call a bend a win unless the arm also converges (twenty
attempts falling to ten loses to a flat two).
Episode::ofreads anepisode off the ledger rows the loop already writes, rather than off a
parallel bookkeeping nobody uses.
evals::arms— the control. A loop with learning off is not a loopwith its ledger removed: this ledger is load-bearing within an episode,
holding the exclusion list that stops attempt four repeating attempt two.
Forgetfulblanks exactly the cross-episode reads — lessons, evidence,workflow scores, repair lineage — and leaves the rest. Writes still
happen and are never read, so the control arm still pays for
consolidation; an arm that skipped the writes would look cheaper for a
reason unrelated to learning.
Three departures from the design as derived, each one a defect the
running example found:
An arm that solved nothing must not tie with an arm that solved four.
The example produced exactly that and reported no evidence, because both
slopes read
0.0— one meaning "flat", the other meaning "two points areneeded for a line and there was one". Going from never to once is the
largest fall in attempts-to-success there is.
trend()andmean_attempts()are nowOption, the report emitsnullrather than aflattering zero, and
comparereturns aVerdictnaming which test itwon on:
SolvedMore,Converged, orNoEvidence.SolvedMoreisguarded by "no worse per solve" so it cannot decay into the success-rate
metric this design exists to avoid.
A wrong answer the judge accepted is not a success.
Episodecarriesan outside check separately from the verdict, and
Outcomenames allfour states — solved, refused (right, judge would not accept), wrong
(judge accepted, world refuted), failed.
Nonefor "not checked" isdistinct from
Some(true), so a report can say which runs were taken atthe judge's word.
A failed episode is a gap, not a shift. Wins are fitted at their
original positions, so a wasted episode makes the measured convergence
shallower — the family took an extra episode to cover the same ground.
The test states the direction explicitly because the intuition runs the
other way.
No task set ships here. A family has to be chosen for shared technique
and checkable answers, and belongs to whoever runs the eval; a list baked
into a host-agnostic crate would be an opinion about somebody else's
domain. The example carries a simulated one, and says so.