[kriszyp]

⊙ Problem

A scalar subscription cursor can skip a transaction committed later at a lower key (harper#2928), and cannot resume on another node. Origin-closed floors exist in seq rows but subscriptions do not consume them.

🤔 Is a per-origin checkpoint the right requirement, and does it belong in one slice?

Yes: the owner confirmed harper-pro#922 items 3–4. Issuance, admission and MQTT share one completeness invariant, so they stay in one PR.

💡 Solution

A reportProgress subscription on RocksDB issues a compact per-origin checkpoint. subscribe({ resume }) admits it on any node after checking membership, coverage, revocation and retention, then replays each member's log from its boundary. Live delivery skips the key gate, so a lower-key late commit arrives. Boundaries come from floors certified after a drained pass, advertised only once on disk (syncWal); without it no token is issued. MQTT keeps the token beside the scalar position.

{ v: 1, scope, time, overlap, exceptions: { laggingOrigin: key }, members: '<64 hex>' }

🤔 Compact form with a membership digest rather than an explicit per-origin vector

An explicit vector grows with the cluster, which #922 exists to avoid; the owner rejected it. The digest binds membership without naming healthy origins. Cost: up to 60 s of replay per healthy origin on resume.

⚖️ Alternatives

🤔 The MQTT scalar never claims more than the token

A downgrade or 409/400 fallback replays from the scalar, but live progress() follows the database-wide dispatch mark. The session caps the scalar just below the token's lowest boundary, frozen while certificates lapse. A dead origin holds it back: longer replay, never a skip. Public progress() is unchanged.

🤔 Pre-CONNACK admission checks database, table and table generation only

An admitted token skips the scalar retention check, since the cap can hold the scalar below retention. A resource re-registered under another prefix passes preflight, then resets after CONNACK: no loss, and rare.

Framing-Verdict: chosen-approach-sound

Approaches weighed

• Per-transport progress (MQTT, SSE, WS each reading floors): three re-implementations of the capture-before-drain proof; the only code that sees both replay and live is Table.subscribe. Rejected.
• A scalar position bounded by the minimum floor over all origins: one stale origin pins every subscriber's resume to a retention window of every other origin's history, the "quiet subscription rescans" the issue opens with. Rejected.
• A server-held session handle instead of a token: each subscriber needs persistence and expiry, plus replication to serve another node. MQTT's record already does this. Rejected for the JS API.
• Advertise only flushed floors on a 60 s cadence instead of syncWal: one SST per idle database per minute. Rejected.
• A producer-side progress() bounded by origin floors instead of the MQTT scalar cap: it changes a public contract for every reportProgress consumer. Rejected.
• Planning review: three --mode plan rounds returned better-alternative-exists, each naming corrections inside the same architecture (floor-only replication resume, a durable publication boundary, origin generation identity, sticky revocation, per-origin QoS 2 receipts, exact record matching); all adopted. One point (one durability barrier over log files and metadata) was overruled: Harper's commit contract is WAL-durable across a crash, not power-loss durable, so a lost unsynced log entry is a durability loss absent everywhere, not this slice's.

🤔 Open: generation selection orders by the highest floor a generation's rows saw

A restored origin whose clock runs behind its old generation's floors keeps the old generation selected, so its tokens skip restored writes. It needs a clock-behind restore plus stale rows; carried as harper#3357. Your call whether it blocks.

🔧 Changes

• resources/subscriptionCheckpoint.ts (new): the token type, membership digest, encoding policy (MAX_OVERLAP_MS), seq-row state reader, strict capture, validation, bounded membership search and admission.
• resources/transactionBroadcast.ts: per-origin floor capture on a timer, certified after a drained pass; certifiedFloors(); the ungated flag on the live key gate.
• resources/Table.ts: subscribe({ resume, checkpointInterval, filterKey }), scope identity, admission and the inclusive per-log replay with epoch rechecks, checkpoint(holds), in-band checkpoint events, origin tagging of events; the seq-row merge takes level-2 qualifiers, revocations and route marks and bumps originStateEpoch.
• resources/auditStore.ts, resources/originClosedFloor.ts: persist → syncWal → publish; originFloorsDurable.
• server/DurableSubscriptionsSession.ts: the token in the durable entry, origin-aware holds, receipts kept and compacted under the token, resume preference and scalar fallback, the scalar cap, the before-CONNACK admission check.
• resources/ResourceInterface.ts: the public SubscriptionRequest and Subscription additions.
• resources/DESIGN.md, server/DESIGN.md, DESIGN.md: the invariants and their proofs.

Product and architecture tour

Product and architecture tour

The checkpoint a consumer holds

What does a client keep to resume on any node without losing a delayed transaction? What does it reveal?

Before	After
A scalar log position from one node's generation; a transaction committed later under a lower key is skipped, and another node cannot check the position.	A compact per-origin checkpoint: one common boundary, exceptions for lagging origins, a digest binding the origin set; any node admits it and a late lower-key transaction is delivered.

A boundary B for origin O means: every entry of O with a key below B that this subscription would deliver has been taken by the consumer, or was appended before the subscription began.
Lowering any boundary is always safe (duplicates only); the token never names a healthy origin.

Example: a healthy three-node cluster

C's subscription has taken everything A, B and C wrote up to their certified floors, which lie within 60 s of one another.

Outcome: { v: 1, scope, time: 1792509223593, overlap: 4100, exceptions: {}, members: '7f…' } — nothing about A, B or C appears beyond the digest.

Example: a lagging origin

B's route to A is a minute behind, so A's boundary is far below the common one.

Outcome: exceptions: { 'A': 1792509160000 } — only the lagging origin is named, at its own boundary; the digest still binds all three.

Where a boundary comes from

How does a node prove that everything below a boundary was delivered?

Floors are certified by the broadcaster, per database

1. Capture — every 5 s the broadcaster reads the qualifying floors: its own published floor (synced to disk first) and every seq-row origin with a level-2 floor, the origin's generation, full coverage, a route start, no hole and no revocation.
2. Pass — a notify pass runs; every entry below a captured floor was in the log before the capture, so the drained pass dispatched it to every registered subscription.
3. Publish — once the pass drains without a failed read, the capture becomes the certified floors; a yielded or failed pass publishes nothing.
4. Bound — an origin's subscription boundary is the minimum of its floor, its oldest queued event not yet taken, and the consumer's holds (MQTT: unacknowledged deliveries).
5. Encode — time = max B, S = max(min B, time − 60 s), exceptions below S, digest over every member's name and generation.

Strict availability: one known origin without a qualifying floor (an LMDB or legacy peer, a partial route, an origin two hops away) means no token at all; MQTT keeps today's scalar.

Resuming on another node

What must the target node check before it serves a checkpoint, and how does it replay?

Admission and replay on the target node

A checkpoint issued on C is resumed on B, which holds A's history through its own route.

sequenceDiagram
    participant Client
    participant B as Node B
    participant Rows as B's seq rows
    Client->>B: subscribe({ resume: token })
    B->>B: scope matches? (400 / 409)
    B->>Rows: read origin states
    B->>B: digest matches own candidates? (late joiner excluded, else 410)
    B->>B: each member retained, not revoked, whole route from at or below its boundary (410)
    B->>B: seek each member's log inclusively at its boundary
    B-->>Client: replayed entries, then live, ungated

Loading

Once live, a certified subscription takes every live entry. The broadcaster's key gate would drop a late lower-key commit that the next floor certifies as delivered.
At every replay yield, admission facts are rechecked against a shared epoch, bumped after the seq-row merge commits.

Two generations of one origin (a restored peer) are ordered by the highest floor their rows saw. Floors sample the origin's clock. A revoked row keeps its floor at revocation; a removed peer's retained row freezes at its death. Only current-generation rows speak for the origin.

MQTT durable sessions

How does a durable session carry the token without breaking an older node that reads the same record?

Before	After
{ qos, topic, startTime, databaseGeneration }; receipts dropped on another node's generation.	The same scalar shadow plus checkpoint; receipts name origin keys and travel with the token; a 410 resets the session, a 409 falls back to the scalar.

• Frozen once established. While floors are unavailable the token stops advancing; it is never replaced by the scalar.
• The scalar stays under the token (just below its lowest boundary), so a downgrade or fallback replay never skips a lagging origin's commit.
• Receipts under the token are compacted and pruned by the checkpoint's lowest boundary, never by the scalar watermark an origin can commit below; one boundary build per PUBREC (about 9 µs at 20 origins with 1,000 queued events).
• Before the first token the session keeps main's scalar receipt semantics, so a QoS 2 message PUBREC'd in that window can be redelivered once at the upgrade (accepted; see the deferred findings).

✅ Verification

On main 45581f85e: test:unit:resources passes (4754), and the session and subscription suites pass. Three new regressions fail on the pre-cap code.

Executed evidence

• Review scope: all open PRs and 50 of 51 recently merged PRs were scanned; one older merged PR was not scanned.

• unitTests/resources/subscribeCheckpoint.test.js: digest and encoding, membership resolution, validation; on RocksDB: issuance names no healthy origin, resume delivers only what follows, an open transaction holds the boundary and is delivered on resume, a late lower-key commit is delivered live, a queued event and a consumer hold lower the boundary, in-band checkpoint events, 400/409/410 refusals, a late joiner admitted only from its first retained entry, strict availability and revocation.

• unitTests/server/durableSessionCheckpoint.test.js: the token beside the scalar and a reconnect through it, a late lower-key transaction redelivered after reconnect, per-origin holds, QoS 2 receipts under the token, a 410 reset before CONNACK, a 409 fallback to the scalar.

• unitTests/resources/replicationSeqCursor.test.js: a sequence update carrying only route marks or a revocation reaches the seq row; the qualifier lifetimes above.

• Original feature gates (before the rebase): test:unit:resources 4420 passing, test:unit:server 1422 passing (plus the existing subscribeProgress and durableSessionResume suites).

• End to end: harper-pro integrationTests/cluster/subscriptionCheckpointResume.test.mjs (A–B–C plus a capability-off E), 4/4: level-2 certificates and a floor-only resume after restart; a checkpoint issued on C resumes on B and delivers a transaction A held open across it, a foreign member set is refused with 410; an MQTT durable session fails over from C to B the same way; a peer that stops certifying at level 2 is revoked on its receiver and one hop beyond, and no checkpoint is issued while it is.

• Not observable here: power loss (the syncWal ordering is pinned by the rocksdb-js ticker test and the sync-then-publish order in auditStore.ts); a two-hop relay (strict availability keeps such a node on the scalar by construction).

PR QA: skipped @ f989077 — dispatch QA trial not available in this session.

_{🤖 Generated by Claude Fable 5.1, OpenAI Codex and Claude Opus 5.5; posted via @kriszyp.}

Release target: main — new API surface (subscription checkpoints) that depends on rocksdb-js syncWal() (HarperFast/rocksdb-js#949; the floor stays level 1 and no token is issued until it is pinned)
Related PRs: #3137 overlaps (broadcaster delivery), #3307 overlaps (subscription attribute filtering), #3248 overlaps (origin-floor timestamp admission), 37 others independent
Complexity: complicated

Deferred findings

• 95e3957b5092a007 minor data-integrity — resources/Table.ts — scalar-resumed subscription: commit below the start before goLive is lost and certified — NOT FILED
• 2cef88f782de4812 minor correctness — resources/Table.ts — goLive ungating redelivers entries the replay or scan already read (scalar and LMDB) — NOT FILED
• 3e23575dcd5989fc minor data-integrity — server/DurableSubscriptionsSession.ts — bound scalar never moves backward; late lower-key commit delivered live and unacked at disconnect is lost on scalar resume — NOT FILED
• 56ed4b88bf5ddc09 nit reliability — resources/Table.ts — replay retention check against a fixed lowest start over-rejects — NOT FILED
• 8ab283c116d7f6e4 nit test-coverage — unitTests/resources/subscribeCheckpoint.test.js — syncWal stub leaves the auditStore persist-sync-publish tick and two-origin cases untested — NOT FILED
• 43b97769b55b8fff nit correctness — server/DurableSubscriptionsSession.ts — a record with a scalar above its token's cap is loaded unchanged and a 409/400 fallback replays from it — NOT FILED

Late Grok findings

2 surviving (highest minor) — Grok reviewed 1062a65

• minor — resources/Table.ts:7649 — the new checkpoint replay calls eventFromAudit without the live flag. Scenario (code trace, not run): with full records and no includeSuperseded, origin X's older write A at key La is merged into origin Y's newer write B at key Lb, so the stored row keeps B's version. The subscriber has taken
• minor — server/DurableSubscriptionsSession.ts:1421 — a scalar position bound to a database generation never moves backward. A late lower-key commit L, which is now delivered live, can therefore be lost on resume if it is still unacknowledged at disconnect.

Not standing: 1 accepted-known. Grok ran 17m (17m from launch to report); the fold adjudication (cursor-grok@1062a65abece-4e3a5855, claude) took 2m.

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; declined=cursor-grok,muse; rounds=15; full=7 @ 03ebc14}

_{Review-Attention: deep ~85m (critical: Table.ts, auditStore.ts +1; hot: DurableSubscriptionsSession.ts; decisions: do-less-alternative, membership-digest, strict-availability, ungate-all-progress-subscriptions, token-410-resets-session, syncwal-gated-own-floor, public-checkpoint-interval-api, release-target-unclear) @ 03ebc14}

[gemini-code-assist]

Code Review

This pull request implements per-origin subscription checkpoints for RocksDB-backed tables, enabling compact checkpoints that bind origin sets with a digest. It introduces the subscriptionCheckpoint.ts module to handle checkpoint validation, encoding, and admission planning, and integrates this mechanism into Table.ts, transactionBroadcast.ts, and DurableSubscriptionsSession.ts to support checkpoint-resumed MQTT durable sessions. Additionally, WAL syncing is added to ensure origin-closed floors are power-loss durable before being advertised. Comprehensive unit tests have been added to verify these changes. I have no feedback to provide as there are no review comments.