Skip to content

Retain Autobahn lane last block across prune - #4199

Merged
wen-coding merged 3 commits into
mainfrom
wen/autobahn-parent-retention
Sep 16, 2026
Merged

wen-coding merged 3 commits into
mainfrom
wen/autobahn-parent-retention

Conversation

@wen-coding

@wen-coding wen-coding commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • After a CommitQC drains [first, next), keep the last local lane proposal so the next produced/pushed block still names a real parent instead of the zero hash (which stalled the lane).
  • Persist that last block when it has left the active range and has not been flushed, restore it from the WAL on restart, and never rewind nextBlockToPersist.
  • Jump-prune (Anchor Next past the local tip / empty-range zero LastHash) is left as a TODO; this change only covers last-seen local production.

Test plan

  • GOWORK=off go test ./sei-tendermint/internal/autobahn/avail/ -count=1

Made with Cursor

@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedSep 16, 2026, 2:06 PM

@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.57%. Comparing base (c45517d) to head (f8a3863).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4199      +/-   ##
==========================================
- Coverage   66.70%   65.57%   -1.14%     
==========================================
  Files        2204     2083     -121     
  Lines      169825   157898   -11927     
==========================================
- Hits       113279   103534    -9745     
+ Misses      56405    54223    -2182     
  Partials      141      141              
Flag Coverage Δ
sei-chain-pr 88.12% <100.00%> (?)
sei-db 74.50% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-tendermint/internal/autobahn/avail/inner.go 96.99% <100.00%> (+1.30%) ⬆️
sei-tendermint/internal/autobahn/avail/state.go 86.23% <100.00%> (+0.15%) ⬆️

... and 121 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ff8b76c. Configure here.

Comment thread sei-tendermint/internal/autobahn/avail/inner.go Outdated
seidroid[bot]
seidroid Bot previously requested changes Sep 16, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retaining the last lane proposal across a prune correctly fixes the zero-parent stall and the setNextBlockToPersist rewind, and the last/retentionFloor/unpersistedLast invariants hold up (last is always either the queue tip or exactly first-1, so the persist batch stays contiguous). However, enforcing the parent-hash check against the retained last removes the previously deliberate recovery path at the prune boundary, so a node holding an equivocating predecessor can no longer resync onto the certified chain.

Findings: 1 blocking | 0 non-blocking | 1 posted inline

Blockers

  • None at the file/PR level.
  • 1 blocking issue(s) flagged inline on specific lines.

Non-blocking

  • None at the file/PR level.

Comment thread sei-tendermint/internal/autobahn/avail/state.go Outdated
@masih

masih commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Relates to #4189

@wen-coding
wen-coding dismissed seidroid[bot]’s stale review September 16, 2026 12:45

The proposed fix is wrong, local last block and qc block might be on different block number.

We plan to do reconcilation of local lasthash and qc block hash in the future.

@wen-coding

Copy link
Copy Markdown
Contributor Author

@seidroid re-review please

@seidroid

seidroid Bot commented Sep 16, 2026

Copy link
Copy Markdown

Claude finished @wen-coding's task in 6m 8s —— View job


Re-review — a54a6ef

  • Gather context (full diff, queue/persist/subscriptions call sites)
  • Re-check the blocker flagged on ff8b76c
  • Trace the last / retentionFloor / cursor invariants end-to-end
  • Review tests
  • Post findings

Previous blocker is resolved. a54a6ef reverts both parent checks (state.go:472, inner.go:147) to q.first < q.next, so the retained last is used for production only and never to reject an incoming block. That closes both the divergence stall and the restoreBlocks crash-loop, and it also subsumes Bugbot's finding. TestPushBlockRecoversWhenCertifiedLastDiffers and the leftover below first is not parent-checked subtest pin exactly those two paths. The rejecting test I'd worried about (TestPushBlockRejectsBadRetainedParentHash) is gone.

Findings: 0 blocking | 5 non-blocking


What I verified holds

I traced the new state machine rather than re-reading the diff in isolation; these are the properties the change silently depends on, and they all check out:

  • last is never below first-1. pushBack puts it at next-1; prune(newFirst ≤ next) leaves it, and next-1 ≥ newFirst-1; prune(newFirst > next) clears it; restore sets it only at first-1. This is load-bearing — see finding 3.
  • Persist-batch tail is WAL-contiguous. When unpersistedLast fires, its number is first-1 and the loop starts at max(cursor, first) = first, so laneWALState.persistBlock's nextBlockNum sequence check (blocks.go:52) is satisfied. When it doesn't fire, bb.first may be first-1 while the tail starts higher — harmless, truncateForAnchor bumps nextBlockNum over the gap (blocks.go:79).
  • No persist busy-loop. The wait predicate cursor < q.next goes true exactly once after prune parks the cursor at first-1, and setNextBlockToPersist(first) clears it. The restore path sets cursor = q.next, so a WAL-restored last is never rewritten.
  • No duplicate votes. RecvBatch (subscriptions.go:81-85) now sees a cursor that can sit below bq.first, and it assigns r.next[lane] = upperBound unconditionally. That's safe only because upperBound = min(bq.next, cursor) is monotonic in both terms — it is, since prune raises next to first and setNextBlockToPersist now refuses to rewind.
  • The setNextBlockToPersist guard is genuinely needed, not defensive: runEvict can prune to a higher floor while a batch is on disk, and the old unconditional write rewound the cursor below q.first.
  • Retained-last write races resolve. Collect → runEvict prunes past it → PruneAndPersist still appends it → next batch's retentionFloor prunes it back off. Transient, and a crash in that window restores cleanly (stale record at < first-1 fails b.Number+1 == q.first and is ignored).

Non-blocking

1. A conflicting local predecessor survives prune and is now durable. inner.go:299-302

prune keeps last whenever lr.Next() <= q.next, without comparing it to lr.LastHash(). If this node holds X@N-1 in its own lane while the network certified Y@N-1, ProduceLocalBlock (state.go:663-666) parents block N to X, and peers whose prune lags still have Y in range and reject it — the same stall this PR exists to fix.

This is not a regression — on main the zero parent was rejected by those same lagging peers — and the TODO explicitly names it ("a later QC cannot replace a conflicting local predecessor"). What is new is that unpersistedLast now writes X to the WAL and restoreBlocks restores it, so the wedged state survives a restart where before it was lost. The reconciliation is one comparison at a site that already has lr in hand:

if lr.Len() > 0 {
    if p, ok := bq.last.Get(); ok && p.Msg().Block().Header().Hash() != lr.LastHash() {
        bq.last = utils.None[*types.Signed[*types.LaneProposal]]()
    }
}

Guarding on lr.Len() > 0 keeps the empty-range zero-LastHash case (the other half of the TODO) out of scope. I'd rather see this land here than as a follow-up, since it is strictly narrower than the jump-prune seeding the TODO defers. Fix this →

2. Restore adopts the first-1 record unvalidated. inner.go:135-140

Same family as 1, and the fix is equally local: restoreInner calls i.prune(anchor) at state.go:167 immediately before restoreBlocks, so anchor.CommitQC.LaneRange(lane).LastHash() is available to decide whether the WAL record at first-1 is actually the certified predecessor or a leftover from a divergence before the crash. Fixing 1 at the prune site does not cover this, because restore seeds last after prune ran.

3. The last invariant is undocumented but load-bearing. inner.go:16-18

The field doc describes last as "the last proposal this node pushed". The thing collectPersistBatch actually relies on is narrower: last is None, or sits at index ≥ first-1. If it could ever sit at first-2, bb.tail would have a hole and persistBlock would fail the batch with block .../N out of sequence. That is currently true of every mutation path, but nothing says so, and a future prune variant could break it with no local signal. Per the AGENTS.md rule about relocating rather than dropping load-bearing invariants, this belongs on the field.

4. The four-line TODO sits inline in the prune loop. inner.go:299-302

AGENTS.md: a long comment inline in a flow means the step was never named. The body it describes is bq.prune(lr.Next()), so the natural home is blockQueue.prune's doc comment (which already explains the clearing rule) rather than the call site.

5. last pins a full payload past prune. inner.go:18

last holds the whole *Signed[*LaneProposal], so one block's payload per lane stays reachable after prune — indefinitely for a lane that stops producing. It's bounded (+1 against BlocksPerLane = 3 * MaxLaneRangeInProposal) and the full proposal is genuinely required until unpersistedLast has been flushed, so this is an observation, not a request. Narrowing to a header after the flush would need a second field; probably not worth it.


Test coverage

Good, and the new tests pin behaviour rather than implementation. Two gaps worth a line each:

  • No test that the cursor can sit below q.first without RecvBatch re-emitting already-voted headers. That's the non-obvious consequence of parking the cursor at retentionFloor, and it is currently only true by accident of monotonicity.
  • No test for two prunes with no flush in between (prune(N) retaining last, then prune(M > q.next) clearing it) — that's the path where setNextBlockToPersist's new rewind guard fires.

Note

I could not execute go test / gofmt in this environment (sandbox denied both), so the analysis above is static. Please confirm GOWORK=off go test ./sei-tendermint/internal/autobahn/avail/ -count=1 and make fmtcheck are green locally.
· branch wen/autobahn-parent-retention

@wen-coding wen-coding added the backport giga-1 Backport to experimental Giga 1 branch label Sep 16, 2026
@masih

masih commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

@seidroid review

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retaining the last lane proposal across a prune correctly fixes the zero-parent-hash stall, and the reworked check now skips the parent comparison outside [first, next) so the prune-boundary recovery path from the previous review is restored. The new retentionFloor/unpersistedLast/no-rewind invariants hold up under the orderings I traced; remaining notes are about deleted rationale and maintaining last for lanes that never read it.

Findings: 0 blocking | 2 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

// NOTE: after pruning (q.first >= q.next), we cannot verify the parent
// hash because the previous block is gone. This is safe because
// headers() never follows the first block's parentHash in a LaneRange.
// Parent is checked only while the predecessor is still in [first, next).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The replacement comment states the scope of the check but drops the invariant that made the gap safe: headers() seeds want from lr.LastHash() and walks backwards (state.go:541), and FullCommitQC.Verify does the same (types/commit_qc.go:99-103), so the first block's parentHash in a LaneRange is never followed. That is the reason an unverified parent at the prune boundary cannot deadlock header reconstruction, and it is now recorded nowhere.

I confirmed the invariant still holds on the current tree, so this is documentation rather than behaviour — but per AGENTS.md ("Relocating a load-bearing invariant is the move, never deleting one to tidy up") it should be carried over rather than removed. Without it, the next reader has no way to tell whether if q.first < q.next is a deliberate, safe narrowing or an oversight — which is exactly the question the previous round of review had to re-derive from scratch.

The same applies to the sibling edit at inner.go:146, which dropped the note that the WAL prunes lazily and can therefore hand restoreBlocks blocks below the anchor. That reason is now more load-bearing than before, since this PR deliberately retains one such block via retentionFloor.

// blockQueue is a per-lane block queue.
type blockQueue struct {
queue[types.BlockNumber, *types.Signed[*types.LaneProposal]]
// last is None, or this node's last pushed proposal at height >= first-1.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] last is only ever read for local block production (ProduceLocalBlock, state.go:664) — PushBlock deliberately does not consult it — yet pushBack sets it for every lane, so retentionFloor and unpersistedLast keep and may flush one extra signed proposal (full payload) per foreign lane, in memory and in the WAL, with no consumer. On a large committee that is one retained proposal per validator held past the anchor for the life of the process.

Related: the doc comment says "this node's last pushed proposal", but PushBlock sets it from a remote producer's block too. A reader could reasonably conclude foreign lanes don't retain anything here.

If gating is wanted, the choke point is addLane/newBlockQueue (which is where the local key could be compared against lane.Validator) rather than a condition at each pushBack caller. Otherwise, at minimum reword the comment to match what the code does.

@wen-coding
wen-coding added this pull request to the merge queue Sep 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 16, 2026
@wen-coding
wen-coding added this pull request to the merge queue Sep 16, 2026
Merged via the queue into main with commit 36bcce7 Sep 16, 2026
92 of 94 checks passed
@wen-coding
wen-coding deleted the wen/autobahn-parent-retention branch September 16, 2026 15:31
@seidroid

seidroid Bot commented Sep 16, 2026

Copy link
Copy Markdown

Successfully created backport PR for giga-1:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport giga-1 Backport to experimental Giga 1 branch non-app-hash-breaking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants