Catch up co-core 0.37.0 → 0.39.1 ahead of the capture consumer #153

Activity

1. gregoryfoster commented on Oct 24, 2026

   

   gregoryfoster

   on Oct 24, 2026

   ContributorAuthor

   More actions

   Brief (coordinator, epic #154, 2026-10-24)

   Authorities: this issue's body and this comment. Where they disagree with the code, stop and post a correction before building.

   Pre-flight against origin/main 42b728c and .wheelhouse

   • The wheels exist and move together [measured]. co_core, co_core_aio and co_core_sync 0.39.1 are synced. Both co-core-aio and co-core-sync 0.39.1 declare Requires-Dist: co-core<0.40,>=0.39.
   • The capture types Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138 needs are in co_core/pure/models/changes.py in both 0.37.1 and 0.39.1: ContentCaptureCommand, CaptureStoredEvent / CaptureFailedEvent and CaptureStoredEmit / CaptureFailedEmit [measured, from the wheels]. Not in 0.37.0.
   • Sites naming the current pin (git grep 0.37 origin/main):
     • pyproject.toml:10-18 (the floor's rationale comment) and :189-191 (the three specifiers).
     • AGENTS.md:37 ("Pin the current minor — >=0.37.0,<0.38").
     • .socraticodecontextartifacts.json:156 (the manifest description). This means codebase_update after the merge (AGENTS.md, Consume content.fetch-policy — replace the single env default with Watcher's per-host numbers (cannobserv#285) #19).
     • docs/DEPLOYMENT.md:75-79. Re-read it: it may name the minor only by example.
     • Leave the historical 0.36.0/0.37.0 mentions alone: scripts/co_core_closure_diff.py's usage line, tests/test_co_core_closure_diff.py's real-wheel fixture, and tests/worker/test_policy*.py's "pre-0.37 producer". They describe history, not the pin.
   • AGENTS.md is close to its 6000-token ratchet (5991 on 2026-10-20 [stated]). The edit there should be a version swap of the same length. Measure exact after.

   Template

   #140 (PR #141): the last multi-minor catch-up. It used an isolated venv, the three pins together, a pin comment rewrite and a closure diff. #145's script now does the diff: uv run --no-project python scripts/co_core_closure_diff.py 0.37.0 0.39.1, after the wheel sync. Paste its report into the pin comment.

   Hazards, to weigh

   • The changelog says no surface Replicator uses moved [traced, from cannobserv's CHANGELOG]. 0.38's breaks are [extract]'s, which Drop the unused co-core [extract] extra #142 dropped. 0.39's are the co/v1 adapters'. 0.39.1 changes Python-mode dumps of two CRM models. Upgrade that to [measured] with the closure diff before relying on it. Pay particular attention to anything under co_core.pure.models.changes, co_core_aio.bus and co_core_aio.fetch, the three surfaces the worker imports most.
   • The shared venv. Work in an isolated venv (memory: develop-in-worktrees; the .pth flip). The service runs the main checkout's .venv, so the main checkout's uv sync --frozen at deploy is what moves production.
   • If the diff shows a real break, the fallback is >=0.37.1,<0.38. It's an in-minor patch that still unblocks Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138, and the minor move becomes its own child. Post the evidence and stop for the operator.

   Operator decision

   [Operator decision: floor >=0.39.0 (AGENTS.md's rule: the floor moves only for a feature the code depends on, and nothing here needs 0.39.1) or >=0.39.1 (as this issue's body says) — recommend >=0.39.0,<0.40, with the lock resolving 0.39.1. #138 raises nothing further: its types exist from 0.37.1.]

   Expected side effects

   • One worker restart at deploy, and a short consumer gap that the bus absorbs.
   • Nothing wire-visible: no new stream or field is consumed yet.

   Out of scope

   • Any capture code. That's Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138.
   • The Dependabot uv ecosystem (Adopt Dependabot: nothing proposes crossing this repo's dependency caps #76, parked on CannObserv/cannobserv#554).

   Post-merge host steps (for the issue, when shipping)

   1. Deploy: git pull --ff-only && uv sync --frozen && sudo systemctl restart replicator.
   2. Read back:
      • /run/replicator/build-id is the merge SHA;
      • uv pip show co-core co-core-aio co-core-sync shows 0.39.1 for all three;
      • the boot journal shows worker ready and the fetch-policy replay with no decode warnings;
      • a stored blob after worker ready.
   3. Run codebase_update from the main checkout, because the manifest description changed.
2. gregoryfoster commented on Oct 24, 2026

   

   gregoryfoster

   on Oct 24, 2026

   ContributorAuthor

   More actions

   Brief review (implementer, against origin/main 42b728c and .wheelhouse)

   Pre-flight confirmed: wheels synced; co-core-aio / co-core-sync at 0.39.0 and 0.39.1 both require co-core<0.40,>=0.39 [measured, METADATA]; the capture types are in 0.39.1's changes.py [measured]; production venv is 0.37.0 ×3 [measured].

   Corrections

   1. docs/DEPLOYMENT.md:75 names the minor outright, not by example ("Pin the current minor — >=0.37.0,<0.38, on all three"). It's a pin site; it gets the same swap as AGENTS.md. :79 also carries a version-anchored fact ("At 0.37.0 that's co-core-sync's two lazy __getattr__ sites"). Re-measured at 0.39.1: still exactly those two (drivers/__init__.py:89, drivers/blobstore/__init__.py:32), so the line moves to 0.39.1.
   2. "0.38's breaks are [extract]'s, which Drop the unused co-core [extract] extra #142 dropped" is half right [measured, co_core_closure_diff.py 0.37.0 0.39.1 --diff]. Drop the unused co-core [extract] extra #142 dropped the extra (the parsers), not the package: pure/extract/* is in the import closure through changes.py's import of extract.canonical, and seven of its files changed — JsonExtractConfig, ignore_selectors, refuse_unhonoured's signature (two new required kwargs), the jsonpath algorithm, .json in the extension map, spec_fingerprint routed through a new _canonical_json.py. None is reached from here: src/ and scripts/ import nothing from co_core.pure.extract and call none of spec_fingerprint, canonical_text, processor_version or extraction_config_from_spec. The only new Requires-Dist (python-jsonpath, soupsieve) is gated on extra == 'extract', which isn't installed. Conclusion stands; the reason in the pin comment changes.
   3. envelope.py moved two things the brief didn't list. (a) from_wire's global table gains content_capture / capture_stored / capture_failed. On 0.37.0 such a frame on a stream we read raised BusMessageUnknownEventTypeError; on 0.39 it decodes and falls to the isinstance guards, as the Watcher types did at Apply FetchPolicyState.generation only if newer #136. (b) idempotency_key for InfoItemPrimaryChangedEvent gains occurred_at. Replicator neither publishes that type nor calls idempotency_key; its own facts' keys are unchanged.
   4. The three surfaces the brief flags are clean: co_core_aio.bus and co_core_aio.fetch are byte-identical bar __version__ (all of co_core_aio is). In changes.py every model Replicator consumes or emits (ContentFetchCommand, BlobAvailableEvent, FetchFailedEvent, ReplicationFailedEvent, PersistFailedEvent, FetchPolicyState) has the same field list at both ends [measured, AST]. The --diff output appears to give PersistFailedEvent a new detail field. That's difflib alignment: the field was already there at 0.37.0.
   5. Operator decision, evidence for it: co_core_closure_diff.py 0.39.0 0.39.1 → three version-only __init__s, Requires-Dist unchanged [measured]. 0.39.1 brings nothing the closure sees, so I'm building the recommended >=0.39.0,<0.40, with the lock resolving 0.39.1. If you'd rather have >=0.39.1, it's a three-character swap before merge.
   6. Deployed build is cb0d7b9, not main's 42b728c [measured, /run/replicator/build-id]. The nine commits between them touch only docs, tests and skills-vendor, so nothing at runtime. This deploy moves the build-id across both.

   No real break → no fallback to >=0.37.1,<0.38. Proceeding.
3. gregoryfoster commented on Oct 24, 2026

   

   gregoryfoster

   on Oct 24, 2026

   ContributorAuthor

   More actions

   Implementation report (branch 153-co-core-0.39, not yet pushed)

   Commit	What
   5ed0219	co-core, co-core-aio[bus,gcs], co-core-sync[gcs] → >=0.39.0,<0.40. Lock moves co-core ×3 0.37.0 → 0.39.1, nothing else. Closure diff pasted verbatim into the pin comment
   774a761	AGENTS.md:37 (same-length swap, 14090 bytes), DEPLOYMENT.md:75 and :79, manifest description
   b4e515f	CR 2: tests/test_co_core_pin.py. Asserts the three specifiers are identical and one minor wide, and that AGENTS.md, DEPLOYMENT.md and the manifest each name the pin. Red against 5ed0219 (3 failed) and against one skewed specifier
   6ac5841	CR 1: the pin comment states the measured fact. Every class that already existed in changes.py at 0.37.0 is AST-identical at 0.39.1 apart from docstrings
   2790a53	CR 3: the pin comment names what a misrouted capture frame now costs (wrong-type DLQ or skip instead of a poison frame)

   Floor >=0.39.0, as the brief recommended. Operator: say if you want >=0.39.1 before merge. It's three characters, and the new test keeps the four doc sites in step.

   Verification

   • Isolated venv: the worktree's own .venv, created fresh, not linked to the deploy checkout's. uv pip show → 0.39.1 ×3.
   • uv run pytest, final gate after the CR fixes: 1463 passed, 6 skipped, coverage 99.89%.
   • -m integration tests/worker/test_oom_integration.py (the CI step): 16 passed.
   • The CI import smoke, run locally, plus ContentCaptureCommand / CaptureStoredEmit / CaptureFailedEmit / CONTENT_CAPTURE: OK.
   • ruff check / ruff format --check: clean.

   Skipped verification

   • AGENTS.md exact token count. No ANTHROPIC_API_KEY on the host, so measure-context.sh --exact fell back to the estimate: 5991/6000. The swap is byte-for-byte the same length (0.37.0,<0.38 → 0.39.0,<0.40).
   • CI. Not run yet, because nothing is pushed.
   • Other integration tests (-m integration apart from the OOM file). They need a scratch broker; CI doesn't run them either.
   • The baseline suite at 0.37.0 is contaminated. A uv run ruff re-synced the venv to the new lock mid-run, so the 7 failures in that run aren't attributable to 0.37.0. The three its tail names were two submodule-dependent tests (submodules weren't initialised in the fresh worktree) and test_main during the swap. After git submodule update --init, the full suite on 0.39.1 is green.

   Accepted limits

   • No behaviour test for the bump itself. It has no behaviour, as at Catch up co-core 0.19.6 → 0.36.0 ahead of the generation release #140. CR 2's test is the TDD hook.
   • A misrouted capture frame now dead-letters, or skips, under a different reason (wrong type rather than poison). It's still dead-lettered or skipped, never processed.
   • The pasted report puts lines over 100 characters into a TOML comment. No linter reads it.

   For #138:

   • The capture types and CONTENT_CAPTURE are present from 0.37.1 [measured, wheel], so Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138 needs no floor raise inside 0.39.
   • CONTENT_CAPTURE is a command kind in _STREAM_KINDS, so it gets one group with competing consumers, like fetch, replicate and persist.
   • idempotency_key for capture: the command keys on command_id, and CaptureStoredEvent / CaptureFailedEvent on command_id:occurred_at, the same pattern as persist.
   • ContentCaptureCommandEmit re-validates viewport / wait instances via model_dump(). Consume with the canonical ContentCaptureCommand (extra="ignore"), per AGENTS.md.
   • CI's import smoke doesn't name the capture types yet. Add them with the consumer.
   • tests/test_co_core_pin.py now guards the four pin sites. If Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138 raises the floor, it fails until AGENTS.md, DEPLOYMENT.md and the manifest move too.

   Post-merge host steps

   1. Deploy from the main checkout: git pull --ff-only && uv sync --frozen && sudo systemctl restart replicator. The main checkout's sync is what moves production; the worktree venv is separate.
   2. Read back:
      • cat /run/replicator/build-id = the merge SHA (it's cb0d7b9 today; the commits between that and 42b728c are docs/tests only);
      • uv pip show co-core co-core-aio co-core-sync → 0.39.1 for all three;
      • journalctl -u replicator -b (since restart): worker ready and the fetch-policy replay, with no decode / poison warnings;
      • one stored blob (blob_available) after worker ready, which discharges "no traffic yet".
   3. codebase_update from the main checkout (manifest description changed; AGENTS.md Consume content.fetch-policy — replace the single env default with Watcher's per-host numbers (cannobserv#285) #19). Cap the server launch per docs/COMMANDS.md (Replicator did not reconnect after the broker's 2026-09-16 outage and restart - restart it, then validate reconnection as a property #94).
   4. Remove the worktree: git worktree remove .worktrees/153-co-core-0.39, then git branch -d 153-co-core-0.39.
4. gregoryfoster

   closed this as completedin #155on Oct 24, 2026
5. gregoryfoster commented on Oct 24, 2026

   

   gregoryfoster

   on Oct 24, 2026

   ContributorAuthor

   More actions

   Shipped: 2790a53 (PR #155, fast-forward merged)

   Done when, verified:

   • CI on the merge SHA, by job: push run 40139262350: lint ✅, test ✅, gcs (T4) ✅. The PR run 40139003308 on the same SHA: ✅ ×3.
   • Deployed: uv sync --frozen in the main checkout moved co-core ×3 from 0.37.0 to 0.39.1. replicator.service restarted at 22:36:14Z. /run/replicator/build-id is BUILD_ID=2790a53. uv pip show reports 0.39.1 for co-core, co-core-aio and co-core-sync.
   • Boot journal: check_main_checkout: on main at 2790a53, the Redis floor check passed, wheelhouse in sync, the fetch-policy replay completed (508 messages), then worker ready. Zero WARNING or ERROR lines. NRestarts=0.
   • Traffic: stored a blob and published blob_available at 23:09:01Z (command 01M5R2SD9JH0H4MDZFRDPR38QH), after worker ready. That discharges "no traffic yet".
   • codebase_update, run from the main checkout under the COMMANDS.md cap: worker untouched, available memory flat. codebase_context shows pyproject.toml, AGENTS.md and docs/DEPLOYMENT.md re-indexed at 22:37Z, and the manifest description reads >=0.39.0,<0.40 since #153. Note: the vendored mcp-driver.mjs has no update subcommand, so I called the tool through a throwaway shim over the driver's withClient. Same pinned launch, same cap.

   Floor: >=0.39.0,<0.40, as recommended, accepted by the operator at ship. The lock resolves 0.39.1.

   Steps 1–3 of the post-merge list are done. Step 4: worktree and branch removed.