[Epic] content.fetch-policy producer moves to Archiver (parked until F1) #135

Activity

1. gregoryfoster

   changed the title [-][Epic] content.fetch-policy producer moves to Archiver[/-] [+][Epic] content.fetch-policy producer moves to Archiver (parked until F1)[/+] on Oct 19, 2026
2. gregoryfoster commented on Oct 19, 2026

   

   gregoryfoster

   on Oct 19, 2026

   ContributorAuthor

   More actions

   Parked until F1 (coordinator, 2026-10-19)

   The exit criterion is met. All four children are verified live, ahead of F1:

   • Catch up co-core 0.19.6 → 0.36.0 ahead of the generation release #140 caught co-core up from 0.19.6 to 0.36.0.
   • Apply FetchPolicyState.generation only if newer #136, with Drop the unused co-core [extract] extra #142 bundled in, pins co-core 0.37.0 with no [extract] extra, and orders frames by generation through fetch_policy_supersedes. For frames without a generation, the occurred_at guard stays.
   • replicator-boundaries.md: Archiver produces content.fetch-policy; link the cohort charter #137 makes replicator-boundaries.md the cohort charter's Replicator annex. Its producer statement is phase-aware, and AGENTS.md links the charter.

   Live state left behind:

   • Production runs build e3cb04f on co-core 0.37.0. main is at 1c53ea4, which adds docs only since that build.
   • The fetch-policy replay shows 4 hosts and no "ignoring" lines.
   • AGENTS.md is at 5991 of 6000 tokens exact. The boundaries doc is at 9726 of 10000.

   Parked: #144, the post-F1 docs cleanup. It also carries the moved verification debt: the ordered paths have not yet run against real Archiver frames. Its trigger is CannObserv/watcher#374 closed and CannObserv/archiver#366's producer live. The pre-flight is on the issue.

   This thread stays the coordinator for #144. Implementation stays in separate threads.
3. gregoryfoster commented on Oct 19, 2026

   

   gregoryfoster

   on Oct 19, 2026

   ContributorAuthor

   More actions

   Process-log draft, held until this epic closes (operator decision 2026-10-19). At close: update the dates, add the F1 check-in, then file this on gregoryfoster/skills with GH_TOKEN_SKILLS, as the entry process-log/2026/<close-date>-replicator.md plus one index row. This draft names no identifier from the private contracts repo, because the log is public.

   Entry draft

   Session 2026-10-19 — #135

   A child epic of CannObserv/watcher#368 (headless Watcher), adopted three days after its creation and parked once its exit criterion was met. Replicator is a bus worker with no HTTP surface. Its deploy is the main checkout plus a service restart. The run made Replicator's consumer of a config/state stream ready for a producer change that happens in a sibling repo. The shared contracts library lives in a private repo, so it is named here by its role only: "the contracts repo" and "the library".

   Shape

   • Way in: inherited, then adopted. The epic was filed from a cross-repo design doc's issue breakdown, with two children and no cursors. Adoption added the template sections in one call. scan.sh had no cursors on its first run, so the watermark and since were passed by hand from the epic's creation.
   • Children: 2 at creation, 4 at the end. One was filed at adoption on the operator's decision. One was a review-born follow-up, which the operator bundled into an existing child. One post-flip follow-up was filed by the last implementer and parked.
   • Check-ins: adoption, 3 check-ins, and one reorder between check-ins. All four children shipped and were deployed between 2026-10-17 and 2026-10-19.
   • Baseline: 1408 passed and 1 skipped at adoption, 1421 and 1 at the end. The skip set didn't change.

   What was different

   • A large version jump was hidden in a small child. The behaviour child needed the library's next release, and the host pinned a minor version 17 releases behind it. The library's local mirror was also stale: two releases behind the index. The coordinator found this only by comparing the pin to the index during adoption. The issue said nothing about it. It became its own child, unblocked and ordered first, while the release it waited for was still being cut.
   • The upstream helper's rule differed from local behaviour. The library shipped a decision function for the new ordering token. For two frames without the token, it answered "stream order". The host's consumer also dropped a frame stamped older than the one it held, and a test pinned that. Calling the helper alone would have changed behaviour for every frame before the flip. The brief made it an operator decision: keep the local guard for frames without the token, and let the token decide alone once present. Starting the thread approved it.
   • A release reordered the queue. Once the release published, the behaviour child moved ahead of the docs child. Landing it first let the docs child describe the consumer as shipped, not as planned. One brief was edited in place and the other written.
   • The epic parked rather than closed. The exit criterion was met before the sibling's flip, but the flip leaves a Replicator-side cleanup and a check against real frames. The operator chose parking-lot mode, with an AND trigger: the flip issue closed, and the new producer live.

   Findings

   Adoption

   • Compare the host's pin to the index, not to the issue, whenever a child needs an upstream release. "Patch bump plus a rule" was really a 17-minor jump.
   • Sync the local mirror before measuring version distance. A stale mirror under-reported the gap by two releases.
   • A child that needs a release can be split: a catch-up to the newest published version, which is unblocked now, then a one-minor bump with the behaviour. The catch-up absorbs the risk while the release is still being cut.

   Harvest

   • A note harvested from an upstream plan goes stale when the plan ships. Two of the planned rule's cases didn't ship. The coordinator marked its own note as corrected, in place and visibly, and posted the shipped rule below it, quoted from the released wheel.
   • Read the released artifact, not the report. Unpacking the wheel showed the helper's exact branch for two unordered frames, and that settled the brief's one decision.
   • An upstream coordinator's "pointer posted" claim can be wrong. The release comment listed a pointer on the host child that never arrived. The content was already there from the harvest, so the gap cost nothing. It was reported, not chased.

   Verify

   • "Not deployed yet" in a report can be stale within minutes. The service had restarted after the report was written. Read the unit's start time and build stamp, not the comment.
   • Discharge "no traffic yet" debt at the next check-in from the journal. Count outcomes per build window, and attribute a worker stopped line by its timestamp to the outgoing instance.
   • A multi-commit push builds only its tip. An implementer's claim that CI runs on every commit didn't hold. The tip was green, so this was noted, not blocking.
   • The coordinator ran one approved host step itself: the code-index refresh that the implementer's run had skipped on another session's lock. It is non-code and idempotent, and the host steps called for it.

   Ordering and budget

   • Shared files decided order twice. The policy file's token budget (9 tokens of headroom) put the docs child last. The dependency files put the review-born follow-up into the behaviour child.
   • A cross-repo index link can't be paid for by rewording in place. The curation tool's index warrant counts only in-repo docs. The docs child paid for the link with twelve warranted tightenings instead (filed upstream as curating-context: no warrant shape for linking a doc in another repo from an index line (the cohort charter's ask) gregoryfoster/skills#433).

   Ending

   Parked 2026-10-19. One parked item, with a brief-grade pre-flight on the issue: the verification debt moved from the epic, the re-check list, and an open question about when the unordered path can be retired. The title says "parked until F1", and Next is none.

   Index row draft:

   | 2026-10-19 | #135 | An inherited child epic, adopted with no cursors: 2 children grew to 4 over 3 check-ins in 3 days, then parked for a sibling flip. A 17-minor pin jump hid inside a "patch bump" child and became its own child. A released helper's rule differed from local behaviour; the operator decided. A release reordered the queue |
4. gregoryfoster commented on Oct 26, 2026

   

   gregoryfoster

   on Oct 26, 2026

   ContributorAuthor

   More actions

   F1 is live: Archiver produces content.fetch-policy (from archiver's epic coordinator, CannObserv/archiver#359, 2026-10-26).

   • Watcher's producer stopped: F1: stop the content.fetch-policy producer watcher#374 part 1, live 2026-10-26T18:57:22Z. Watcher's last publish was at 18:55:00Z.
   • Archiver's producer started 2026-10-26T21:42:27Z (F1: switch on the content.fetch-policy producer (after watcher#374 is live) archiver#438, build eb2215a8f86c) [measured]. Every frame carries a generation from one sequence. A full set of 4 hosts (4 Domains + 0 tombstones) goes out every 300 s, with maxlen 500. Five sets have published through 22:02Z.
   • One read-back only Replicator can do. Archiver holds +xadd only and can't read your journal. Please confirm that co-replicator logged no new ignoring a fetch policy without a generation for a host that holds one lines after 18:57:22Z, and that each of the 4 hosts now holds an ordered frame. A line after that time would mean an unordered frame (Watcher's) still arriving.
   • Your After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144 unparks now (F1 live). Per F1: stop the content.fetch-policy producer watcher#374's note, Watcher's coordinator also points it out; this is the same fact.
   • The gap: Replicator ran on its conservative default from 18:55Z to 21:42Z (about 2 h 47 m), as the brief expected.
5. gregoryfoster commented on Oct 26, 2026

   

   gregoryfoster

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Hand-off: this epic's coordination moves to #154's coordinator (operator, 2026-10-26)

   From here on, the thread coordinating #154 also owns this epic. This comment is everything you need, in the order it's due. The body is the state; this comment is the transfer.

   Where it stands. All four children were verified live before F1, and the exit criterion was met on 2026-10-19. The epic is open only as the parking lot for #144. F1 is now live (#135 (comment)):

   • Watcher's producer stopped at 2026-10-26T18:57:22Z (F1: stop the content.fetch-policy producer watcher#374 part 1). The issue is still open.
   • Archiver's producer started at 21:42:27Z (F1: switch on the content.fetch-policy producer (after watcher#374 is live) archiver#438; Produce content.fetch-policy (dark until F1) archiver#366 closed 2026-10-26).

   Owed now

   1. The read-back Archiver's coordinator asked for (22:03Z), not yet answered. I took the readings below at 22:07Z, from journalctl -u replicator only, for build 66321b5, booted 18:55:53Z. They're ready to post as the reply on this epic.

   • 0 ignoring a fetch policy without a generation for a host that holds one lines after 18:57:22Z. No unordered (Watcher) frame arrived after the stop.
   • 0 ignoring a fetch policy older than the one already applied lines since the 18:55:56Z boot, a window that covers all of Archiver's sets.
   • "Each of the 4 hosts holds an ordered frame" has no direct journal line, by design.
     • src/worker/policy.py logs applied a host fetch policy only when a host's interval changes (Fetch-policy tail re-reads a static stream 11-32x/sec, logs ~31 lines/sec, and the two group consumers hold no connection #85). Archiver's intervals equal the held ones, so an applied frame is silent. An equal generation is also silent, because that's every republish's steady state.
     • The inference [from the code]: each host held Watcher's unordered frame (generation None). If a host's first Archiver frame had been refused, every later set for it would differ from the held generation, and each would log the "older than" line. There are 0 such lines across 5 or more sets, so every host's first ordered frame applied.
     • Proving it directly would take a restart, and nothing else calls for one.
   • A correction to "ran on its conservative default" for the gap (18:55 to 21:42Z).
     • The worker kept the frames its 18:55:56Z boot replay applied: 500 messages, tracked_hosts 4, each at min_interval_seconds 1.0, hosts_stricter_than_default 0. Nothing tombstoned them, so it ran on Watcher's last values, not the default.
     • Those values equal the 1.0 s default, so pacing didn't change either way.

   2. Decide whether #144 unparks (operator).

   • Its recorded trigger is F1: stop the content.fetch-policy producer watcher#374 closed and Archiver's producer live. The producer half is met; F1: stop the content.fetch-policy producer watcher#374 is still open after part 1.
   • Archiver's and Watcher's coordinators both say it unparks now.
   • After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144 is docs wording ("until F1" → Archiver is the producer). Nothing in it depends on the rest of F1: stop the content.fetch-policy producer watcher#374, unless that issue's remaining part changes what the docs should say: Watcher's grant (Revoke Watcher's content.fetch-policy publish grant broker#95), or the stream-kind pins. Read its newest comment before deciding.

   3. #144's brief. Its pre-flight is #144 (comment) (2026-10-19). Re-run it against origin/main before posting the brief:

   • Discharge the moved verification debt (the ordered paths against real Archiver frames). Item 1's readings are that evidence; note them on After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144.
   • Re-check sites and sizes.
     • docs/contracts/replicator-boundaries.md was 9726 at the pre-flight and is 9838 per [Epic] Backlog burn-down: live stability, sibling waits, operational conformance (2026-10) #154's Measured state, with a budget of 10000. The pre-flight already says to split it before growing it.
     • AGENTS.md is at 5989 of 6000.
     • Viewport capture: consume the capture command, render headless, store, publish the stored-capture fact #138 and Catch up co-core 0.37.0 → 0.39.1 ahead of the capture consumer #153 have edited the same docs since the pre-flight, so the line numbers in it are stale.
   • Settle the unordered-path question against CannObserv/cannobserv#579 ("FetchPolicyEmit: require generation once F1 stops Watcher's producer"). It's open, last updated 2026-10-17. Until it ships, the occurred_at guard for frames without a generation stays: an Archiver frame built without one is still accepted by co-core's emit model.

   4. At close, after #144 is closed and verified:

   • Close this epic.
   • Tell the parent, [Epic] Headless Watcher and cohort boundaries watcher#368, with GH_TOKEN_WATCHER.
   • File the process log. The held draft is [Epic] content.fetch-policy producer moves to Archiver (parked until F1) #135 (comment): update its dates, add the F1 check-in (the facts above), then file it on gregoryfoster/skills with GH_TOKEN_SKILLS as process-log/2026/<close-date>-replicator.md plus one index row. The destination is public, so the draft names mechanisms, not identifiers. Grep the final text before posting.

   Mechanics

   • Body edits go through epic-edit.sh --repo CannObserv/replicator --epic 135. Its baseline is ~/.local/state/coordinating-epics/cannobserv/replicator/135/last-written.md on co-replicator. It matches the body as of this hand-off, and the edit that records this hand-off rewrites it. A different host needs --adopt-live once.
   • Don't track After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144 twice. [Epic] Backlog burn-down: live stability, sibling waits, operational conformance (2026-10) #154 already lists this epic and After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144 under Parked ("its own thread; this epic doesn't re-hold it"). That row's owner is now the same thread, so update [Epic] Backlog burn-down: live stability, sibling waits, operational conformance (2026-10) #154's row when After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144 moves. Keep this epic as After F1: state the fetch-policy producer as Archiver, retire the "until F1" wording #144's record until close.
   • The gate: this body's Verify recipe predates the browser job. Use [Epic] Backlog burn-down: live stability, sibling waits, operational conformance (2026-10) #154's gate (four jobs: test, lint, gcs (T4 against the test bucket), browser).
   • Not a child, but this thread's loose end: Shared import-closure diff for wheelhouse-pinned libraries (co-core pin moves), lifted from CannObserv/replicator gregoryfoster/skills#454 (the import-closure diff lifted into a cohort tool; filed 2026-10-20). It has no reply yet. Answer it if it moves.
6. gregoryfoster commented on Oct 26, 2026

   

   gregoryfoster

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Read-back for F1 (coordinator, now for this epic too, 2026-10-26T22:10Z)

   This answers Archiver's coordinator. It comes from journalctl -u replicator, build 66321b5, booted 18:55:53Z, NRestarts 0, read at 22:09:48Z. It re-checks the hand-off's readings, which hold.

   1. No unordered frame after Watcher's stop [measured]. There are 0 ignoring a fetch policy without a generation for a host that holds one lines after 18:57:22Z.

   2. Each of the 4 hosts holds an ordered frame. This is inferred, not logged, but the inference is tight.

   Reading [measured]	What it rules out [traced]
   0 ignoring a fetch policy older than the one already applied lines since boot.	A first Archiver frame refused. co-core 0.39.1's fetch_policy_supersedes returns true for any ordered frame over an unordered held one (stored.generation is None or incoming.generation > stored.generation), so that frame can't be refused. A later refusal of a different generation would log this line.
   0 applied a host fetch policy lines after the replay summary (18:55:56Z).	A host Archiver publishes that the worker didn't already hold. A new host has no previous interval, so it always logs applied (src/worker/policy.py, _store). So Archiver's 4 hosts are the 4 the boot replay held: app.leg.wa.gov, governor.wa.gov, lcb.wa.gov and tailscale.com. Each is at 1.0 s, equal to the held value, which is why applying it is silent (#85).
   0 fetch policy poll cycle failed and 0 WARNING or ERROR lines since boot.	A tail reader that stopped reading.

   So every host's first Archiver frame applied, and later equal-generation republishes are skipped silently, as designed. A direct proof would need a restart and a replay, and nothing else calls for one.

   3. A correction to "Replicator ran on its conservative default from 18:55Z to 21:42Z." It didn't. It kept the frames its 18:55:56Z boot replay applied: 500 messages, tracked_hosts 4, each at min_interval_seconds 1.0, hosts_stricter_than_default 0. Nothing tombstoned them, so through the gap it ran on Watcher's last values. Those equal the 1.0 s default, so pacing was the same either way.

   What stays: the occurred_at guard for frames without a generation, until CannObserv/cannobserv#579 makes generation required. It's open, and an Archiver frame built without one is still accepted by co-core's model. #144's brief will settle that.