promote: differing same-name copies are an error, jax is considered, and a consumer series class (project-monorepo#74)

[mmcky]

Implements the promote-tool changes of QuantEcon/project-monorepo#74, the first work item of the plan of record, except the switch operation, which moved to its own work item (QuantEcon/project-monorepo#104) after the adversarial review of 2026-10-08. Includes #17 (merged into this branch), which closes #15.

What changes

• A same-name copy that differs is an error. A copy whose bytes differ from the anchor copy, after the ledger's asset-path normalisation, is refused with an error naming the lecture and every series holding a copy, with hashes. Only a committed map, sync/canonical.yml, settles it, by naming the winning copy; manifest priority still chooses among byte-identical holders, and run order never decides. The map is seeded with the 24 entries that were interim today, all dp-test collisions; they go when the dp-test stanza is removed (QuantEcon/project-monorepo#24).
• A stale copy is not a collision. A recorded source that is still at a version the pool already took (its own recorded digest, or the canonical copy's) while the canonical copy moved on stays a source, whatever anchors the run: a refresh, --toc of the lagging series, or an explicit promotion. A map line naming such a stale copy is refused, since it no longer settles a collision. The drift-check calls the same state stale-copy (info), and judges a non-canonical copy by the rule promotion admitted it under (drift-check: judge a non-canonical copy by promotion's own rule #17).
• The map is validated. Repeated keys, two keys naming one lecture, and an unknown or consumer series are errors before anything is planned; a slug that no canonical-eligible series holds is reported on every run. "Settles nothing and can be removed" is given only once every canonical-eligible series has been checked, and says to remove a line naming a lower-priority holder at once, since such a line is refused when that copy falls behind.
• jax is considered like any series: ALWAYS_EXCLUDED is removed and --exclude stays a per-run flag.
• A consumer class. A class: consumer series in the manifest is never canonical and contributes no lecture; its copies are recorded under a new superseded: field and the drift-check never watches them. dp is declared one.
• A refresh never skips silently. An entry whose canonical series the run cannot use (excluded, a consumer, or gone from the manifest) is skipped while its pin stands and fails once the pin moves, naming the reason. A canonical copy removed upstream is named as such.
• tools/tests/: 51 pytest tests on scratch repositories, run by a new tests.yml on pull requests and manual dispatch.

CI changes to note

• The daily refresh passes --exclude $PROMOTE_EXCLUDE, now dp jax (was dp): with jax considered, short_path (intro against jax) would otherwise fail today. Drop jax when its renames land (QuantEcon/project-monorepo#76); an empty list now works.
• A refresh entry promote refuses no longer holds the others back. The refresh pull request is opened with what did refresh, lists each refused entry with promote's reason, and the run goes red. An entry whose canonical copy moved is refused again every night, so the run stays red until the entry is settled, and a settlement applies the next night; drift-check.yml: a refused refresh entry is not retried until some canonical copy changes upstream, and the run goes green meanwhile #20 covers an entry refused when only its series' pin moved.
• A refresh that exits before its summary line (a configuration error or a crash) opens no pull request.

Proof

• Whole estate from scratch (all six series by toc, dp-test excluded, jax considered): the same 16 named collisions as before, about, arellano, autodiff, ifp_egm, inventory_dynamics, jax_intro, kesten_processes, lake_model, lln_clt, lucas_model, markov_asset, mle, pandas_panel, schelling, short_path and wealth_dynamics, and nothing else. In one batch with a map for them, all 297 promote, a repeat run gives a byte-identical ledger, and the drift-check is clean. (Run series by series, rob_markov_perf also needs its shared asset re-homed in the same batch, the hardening deferred to QuantEcon/project-monorepo#11.)
• Regression on the current pool and the real mirror: the drift-check gives main's output (52 lectures, 32 assets, clean). promote --refresh --exclude dp jax updates 41 promoted_at pins and nothing else, the pool stays byte-identical, a second run has nothing to do, and the drift-check stays clean.

Review. An adversarial review on 2026-10-08 (three rounds, 17 reviewers, every finding checked by three independent verifiers) confirmed 39 findings. This branch addresses the ones that belong here; the switch findings moved with the switch, and the rest are filed as #18 to #23. A second adversarial check of the fix commits (2026-10-09; 51 agents) confirmed 18 minor findings: 17 are fixed here, and one is filed as #24.

Opening or updating this pull request runs two jobs: drift-check (report-only on pull requests) and the test job. Neither writes anywhere.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🔵 Needs a closer look

It is a large, intricate change to core provenance tooling with subtle byte-stability and collision-resolution invariants plus operational CI impact (the daily refresh can now go red), which warrants final human review despite the extensive test coverage.

Review effort: Balanced
Findings: None

What changed in this PR

This PR reworks the tools/promote and tools/drift-check tooling that maintains the lecture pool and its provenance ledger, implementing the first work item of QuantEcon/project-monorepo#74. It changes how same-name collisions are resolved, how the jax series is treated, introduces a consumer series class, and adds a --switch cutover mode — plus a new pytest suite and CI job.

Changes:

• A same-name copy whose bytes differ (after asset-path normalisation) is now a hard error naming every series involved, resolvable only via a committed sync/canonical.yml map; run order never decides a winner. ALWAYS_EXCLUDED is removed so jax participates like any series.
• A class: consumer series (declared for dp) is never canonical, records copies under a new superseded: field, and is never watched by the drift-check; promote --switch SERIES stamps canonical: pool/switched_at: on a series' entries so the pool copy becomes source of truth.
• Adds tools/tests/ (24 scratch-repo pytest tests) with a tests.yml PR workflow, and switches the daily refresh to --exclude dp jax.

File	Description
tools/​promote	Core logic: load_manifest/load_canonical_map, collision handling in resolve_sources, normalised_text, switch_series, superseded ledger field, CLI --switch.
tools/​drift-check	Skips canonical: pool and consumer-series copies in series_in_ledger/check_lecture/check_asset.
sync/​manifest.yml	Documents class: and marks dp as consumer.
sync/​canonical.yml	New map seeded with 24 previously-interim dp-test collisions.
sync/​ledger.yml	Header comments updated for superseded/switched_at.
README.md	Documents consumer/switch/canonical-map behaviour and the new test suite.
.github/​workflows/​tests.yml	New PR/dispatch job running tools/tests via uv.
.github/​workflows/​drift-check.yml	PROMOTE_EXCLUDE: dp jax.
tools/​tests/​*.py	Scratch-repo harness and tests for collisions, consumer class, and switch.

I reviewed the collision-resolution logic (anchor selection, normalised_text equivalence, divergent/superseded recording), the --switch flow (stale-pin refusal, idempotency, asset-switching gated on all referencing lectures being switched), the overwrite-guard (blocked) interaction with switched assets, and cross-checked each test's assertions against the code paths. I did not find concrete defects, and the tests align with the implemented behaviour.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[quantecon-services]

Review summary

Read every changed file in full and ran the suite plus three extra scenarios against the branch's tools. The design holds: collisions are order-independent, the map picks a copy while priority picks the holder, consumers are handled consistently across promote, drift-check and the ledger, and the switch refuses stale pins and is idempotent. No blocking defects.

Addressed on this branch (two follow-on commits):

• c052dcc drift-check.yml: a refresh entry that promote refuses no longer withholds the refresh PR. The PR is opened with whatever did refresh, lists the failures under "Not refreshed", and the run stays red. Without this, the first drifted non-canonical copy whose canonical then moved would have blocked every other refresh.
• c23890c promote: switched lectures met via --toc are skipped rather than failed; the collision error groups copies into same/different; manifest comments updated (class: source, dp-test stanza). One test added, 25 pass.

Filed as follow-ups:

• #15: drift-check still compares non-canonical sources byte-for-byte, so a normalised-equal source moving in lockstep with its canonical copy raises a false diverged finding. Latent today.
• #16: a new lecture sharing a single-consumer asset with a switched lecture cannot be promoted, and the error's remedy is impossible after the switch. Mid-transition only.

Not reproduced here: the zero-diff ledger regeneration and the 297-name scratch run, both of which need the reconstructed mirror.

---

Generated by Claude Code

[mmcky]

Review follow-up: a stale copy was refused as a collision

The finding. resolve_sources compared every eligible series' current copy with the anchor, so a copy that had simply not moved while the canonical copy did was refused with differs across series. The drift-check calls the same state stale-copy (info), so the two tools disagreed, and the stale case is the common one, not an edge:

• The 2026-09-07 refresh (promote: refresh 14 pool lectures whose canonical source moved #9) re-copied 14 lectures. In all 14 the dp-test copy was unchanged since promotion, so under this PR's rule every one would have failed. They are the 14 "intermediate over dp-test" lines in the new sync/canonical.yml.
• 25 pool entries are still held identically by a source series and dp-test, with no map entry. Each would have turned the nightly refresh red on its next upstream edit, needing a hand-written canonical.yml line. The same would apply to pandas_panel and autodiff once their merges in QuantEcon/project-monorepo#76 leave them identical in two series.
• c052dcc's message and the earlier review framed the failure as "a drifted non-canonical copy, then the canonical copy moving". The commoner trigger needs no drift at all.

I reproduced it with this PR's own harness before changing anything: two series, one identical lecture, the canonical copy edited, --refresh fails.

Pushed to this branch:

• 7128fb1 promote: a stale copy stays a source; only an edited copy is a collision. When the anchor is the entry's recorded canonical series, a differing copy recorded under sources is stale if its digest equals its own recorded digest (it has not moved) or the canonical copy's recorded digest (it caught up, then the canonical copy moved on). It stays a source at its current digest, with no interim, and the refresh goes through and names it. A copy edited to anything else is still refused, and the error now lists stale copies separately. The drift-check gets the matching rule for the caught-up case, which it used to report as diverged. Docstrings, README and the ledger header say the same; the ledger header was regenerated and the committed ledger re-dumps byte-for-byte.
• 0b49216 drift-check.yml: open no refresh PR when promote stopped part-way. promote writes the pool lectures before the ledger, and since c052dcc the workflow commits whatever is on disk after a non-zero exit. A crash between those writes would have put a half-written pool into the refresh PR. A failing promote prints its summary only after every write, so a non-zero exit without the summary line now stops the job before anything is staged.

Tests. Three new tests: a stale copy refreshes cleanly (and the drift-check is clean afterwards), an edited copy still fails, and the caught-up-then-behind case. 28 pass locally and in CI; the stale and caught-up tests fail on c23890c.

Unchanged on purpose. The 24 canonical.yml entries stay. Those lectures record dp-test under divergent, not sources, so the stale rule does not reach them. They go with dp-test at QuantEcon/project-monorepo#24 step 1. Follow-ups #15 and #16 are unaffected.

---

Generated by Claude Code

[mmcky]

Update after the adversarial review (2026-10-08) and a check of the fixes (2026-10-09)

What changed on this branch since my last comment

• --switch is out of this pull request. The review confirmed twelve defects in it, six of them major. None can bite before the first cutover, so the switch lands, fixed, through its own work item before then (QuantEcon/project-monorepo#104). The reviewed implementation is kept on the promote/switch branch, and promote: a new lecture that shares a single-consumer asset with a switched lecture can never be promoted #16 moves with it.
• drift-check: judge a non-canonical copy by promotion's own rule #17 is in (squash-merged into this branch), so drift-check: a normalised-equal source copy is reported as drift when it moves in lockstep with the canonical copy #15 closes with this pull request.
• Seven new commits fix what belongs here:
  • A refresh now fails an entry it cannot use, rather than skipping it with a green run: one whose canonical series is excluded, is a consumer, or has left the manifest, once that series' pin moves.
  • Staleness is judged against the recorded canonical copy, whatever anchors the run. So --toc of a lagging series works, and a sync/canonical.yml line naming a stale copy is refused rather than followed.
  • sync/canonical.yml is validated: repeated keys, two keys for one lecture, and slugs no series holds. A line is called removable only when it is.
  • An empty PROMOTE_EXCLUDE works.
  • The refusal texts give promote's actual reason and the actual retry behaviour. An entry whose canonical copy moved stays red every night until it is settled; drift-check.yml: a refused refresh entry is not retried until some canonical copy changes upstream, and the run goes green meanwhile #20 covers the pin-only case.
• Tests: 51, up from 28. The new tests fail on the previous code, and each half of the stale rule is pinned by a test that fails without it.

Checked on real data (the mirror at the pinned SHAs):

• The drift-check is clean and matches main's output.
• The CI refresh changes 41 promoted_at pins and nothing else.
• A whole-estate promotion stops on the same 16 named collisions.
• All 297 lectures promote in one batch with a 16-entry map, byte-identically on a repeat run.

Filed from the review: #18, #19, #20, #21, #22, #23 and #24. None of them blocks this pull request.

The description above is updated to match.

---

Generated by Claude Code