feat(calibration): reject a contract a trivial baseline satisfies - #18
Merged
Conversation
A milestone contract says which progressions count. It does not say that reaching them is hard. deriveContract proves only that every mark fires on the reference run, so a derived contract can pin a memory channel that moves whenever the game runs at all. A constant button press then earns exactly what an evaluated agent earns, and the score measures elapsed frames. calibrateContract replays the reference and a suite of trivial policies through the same attestRun path: one constant policy per input word, a word the game cannot interpret, a round-robin cycle over the vocabulary, and a seeded pseudo-random walk over it. Every policy is deterministic in the seed. A contract separates only when at least one milestone is out of reach of every baseline and the reference verifies strictly more milestones than the strongest baseline. assertContractSeparates fails closed otherwise and names every trivial milestone with the baseline that earned it. The module stays pure, synchronous, and free of adapter and provider imports.
…tisfies calibration.test.mts runs offline in the repo test chain. A combination lock that opens on one six-word sequence must separate; a game whose channel moves whenever "a" is pressed must not, and the gate must throw naming constant:a. It also covers policy determinism across calls and seeds, report consistency, a one-word vocabulary, a zero-turn calibration, and the empty-vocabulary and negative-turn errors. The same gate runs on the native-2048 adapter and reds it: NATIVE_2048_REFERENCE is a fixed cycle of four directions, and 2048 merges tiles under almost any input, so a seeded pseudo-random walk of the same length reaches all seven milestones. That target exercises the execution and evidence paths; it does not measure skill, and the test now says so. pyboy-libbet.test.mts pins the finding that made this work exist. A live agent campaign of 70 turns on the packaged blind-discovery contract earned three milestones with a clean, replay-verified run. Over the same 70 turns: reference (packaged trajectory at the same progress level) 3 constant:a 3, the same set constant:start 3, the same set round-robin 3, the same set pseudo-random 3, the same set constant:select 2 constant:up, down, left, right, b 0 an unknown word 0 The regression keeps its existing ROM gate: it skips without the ROM and fails under PLAYPROOF_REQUIRE_ROM=1.
README gains a calibration section with the Libbet measurement as the worked example and the full baseline table. docs/adapters.md gains a matching section beside the replay-proof discussion, because replay proof answers "did this run happen" and never answers "was that hard". Blind-discovery contracts need the gate most: nothing in that pipeline asserts that a discovered memory channel means progress. CHANGELOG records the API and both measured findings under 0.4.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
deriveContractcan emit a contract that is not a valid benchmark, and nothing in the framework notices.It proves that every mark fires on the reference run. It never proves that reaching the mark is hard.
A contract that pins a memory channel which moves whenever the game runs at all produces scores, and those scores report how many frames elapsed.
This adds the fail-closed gate that catches it, and pins the measurement that found it.
What changes
calibration.ts, exported from the package root:BaselinePolicy— a deterministic input policy that never observes the game.trivialBaselines(vocabulary)— the standard suite: oneconstant:<word>policy per input word,constant:playproof-unknown-word,round-robin, andpseudo-random(a seeded LCG over the vocabulary). Per-word constants are the point: on Libbet onlyaandstartreach the agent's milestones, so a suite that pressed one representative button would have reported the contract healthy.calibrateContract(game, contract, { reference, vocabulary, seed?, baselines?, turns? })— replays the reference and every baseline through the sameattestRunpath at the same length and returns aCalibrationReportwithseparating,trivial,bestBaselineCount, andseparates.assertContractSeparates(report)— throws unless at least one milestone is out of reach of every baseline and the reference verifies strictly more milestones than the strongest baseline. The message names every trivial milestone with the baseline that earned it.deriveContract's doc comment now states that its output is a hypothesis until calibration shows it separates, and points atcalibration.ts.The module is pure, synchronous, and imports no adapter and no provider, so
check-boundary.mjscovers it as a production file.deriveContractbehaviour is unchanged. The package version is unchanged; CHANGELOG bullets go under## 0.4.0.The measured finding
A live agent campaign ran 70 turns on the free Game Boy game Libbet and the Magic Floor through
adapters/pyboy-genericwith the packagedpyboy/discovery-libbet.jsonblind-discovery document.It earned three milestones. Its verdict was clean and its run replay-verified.
Trivial policies over the same 70 turns, on the same ROM (md5
ce9716a3a431f9722d58d30947f26921) and the same derived contract:ch_c321-progressed,ch_c32d-progressed,ch_ff96-progressedconstant:aconstant:startround-robinpseudo-randomconstant:selectconstant:up,constant:down,constant:left,constant:right,constant:bPressing
aseventy times scores what the agent scored. A benchmark a constant button press satisfies as well as an evaluated agent does not measure competence.The gate found a second one in this repository.
NATIVE_2048_REFERENCEis a fixed cycle of four directions, and 2048 merges tiles under almost any input, so a seeded pseudo-random walk of the same length reaches all seven milestones,tile-32included. That target exercises the execution, evidence, checkpoint, and signing paths; it does not measure skill, and the test now says so out loud instead of leaving a reader to assume otherwise.Verification
pnpm run ciin a clean worktree, exit 0:PLAYPROOF_REQUIRE_ROM=1 pnpm test:pyboy-libbeton the pinned ROM:The agent's own 70-turn transcript is not packaged, so the regression uses the packaged exploration trajectory truncated to the same progress level (the inputs before the rank-0 channel first moves) and gives every baseline the campaign's full 70 turns. That handicap runs in the baselines' favour, so it can only understate the problem. The gate keeps its existing ROM handling: it skips without the ROM and fails under
PLAYPROOF_REQUIRE_ROM=1.