Summary
Give evmOnlyApplication a durable execution cursor so an evmOnly Autobahn node survives a restart. Today the cursor (committedHeight, appHash, parentHash) lives only in evmOnlyState, so every restart after block 1 reports Info().LastBlockHeight == 0, re-runs InitChain, and replays block 1 against a state store that already recovered to height N → nonce too low panic, forever.
Source: sei-chain#4169 (Linear mirror: SEI-10400). Blocks restart/chaos testing of Autobahn + Giga on harbor.
Root cause (verified against main @ 9435e24)
| Piece of state |
Where it lives |
Durable? |
| EVM state (nonce/balance/code/storage) |
flatkv via StateDB.CommitStateChanges, called inside FinalizeBlock (giga/evmonly/giga_store.go) |
yes — WAL + OpenDBWithRecovery converges state/receipts/block store to one height on open |
| receipts |
receipt store, same call |
yes |
committedHeight, nextHeight |
evmOnlyState struct fields, advanced in Commit |
no |
appHash (chained sha256 of prev ‖ height ‖ blockHash ‖ gasUsed ‖ changeset) |
evmOnlyState.appHash |
no |
parentHash (Autobahn header hash fed to the EVM as ParentHash) |
evmOnlyState.parentHash |
no |
gasLimit, executor |
set only in InitChain from ConsensusParams.Block.MaxGas |
no — never rebuilt on restart |
giga_router_common.go:runExecute trusts app.Info(): last == 0 → InitChain + start at GenDoc.InitialHeight. Its comment ("re-entering on restart is safe — nothing was committed") assumes Info() is durable, which holds for BaseApp (cms.LastCommitID()) but not here. The seedInitialStateVersion guard returns early when initialHeight == 1, so the "state already at height %d" refusal never fires at the default initial height.
Note the ordering quirk that shapes the design: state is committed in FinalizeBlock, before commitAppHashToVault and Commit. So "durable height" for this app is last finalized-and-state-committed block, and a cursor written only in Commit would still leave a crash window (FinalizeBlock(N) done, Commit(N) not) that replays N against state N.
Decision to record first (short ADR, docs/rfc/rfc-002-evmonly-durable-cursor.md, from rfc-template.md)
One question: where does the cursor live so it can never disagree with flatkv's committedVersion?
Recommended (A) — the cursor is part of the block's changeset.
- Height: no new storage.
committedHeight := storage.SC().GetLatestVersion() after OpenDBWithRecovery. The storage manager already converges block store / state WAL / receipts to one target, so this is the converged height.
appHash, parentHash: written as a second proto.NamedChangeSet (e.g. Name: "evmonly", keys apphash, parenthash) appended to the EVM changeset in the same CommitStateChanges(blockNumber, …) call. flatkv routes non-evm named changesets to miscDB under "<module>/…" (ktype docs), so this is supported today. Atomic with state by construction; rolled back/replayed by recovery together with state; deterministic across validators so it is safe to fold into the lthash root.- Restore path: on construction (or a new
Restore() step called before Info()), if GetLatestVersion() > 0 read the two keys back via SC().Get(...) and populate evmOnlyState; build the executor; Info() then reports the real height and the router takes its normal restart branch (InitLastHeader + re-PushAppHash) and never calls InitChain.
gasLimit: InitChain is not called on restart, so the app needs the genesis consensus params another way. Recommended: NewEVMOnlyApplication takes the RequestInitChain (node already has GenDoc.ToRequestInitChain()), and InitChain becomes idempotent w.r.t. it. Alternative: persist gasLimit in the same changeset. Pick one in the ADR.seedInitialStateVersion: drop the initialHeight == 1 early-return; latest != 0 must always refuse InitChain (the router will not call it once Info() is durable, so this becomes a true invariant guard, not a hole).
Alternative (B), to be rejected or chosen explicitly: replace the chained sha256 appHash with flatkv's lthash root (StateDB.RegisterHashListener returns the most recent BlockHash) and recover parentHash from storage.BlockStore().ReadBlockByNumber(last). Cleaner semantically (app hash == state root) but changes what the app hash commits to (drops gasUsed/blockHash), touches hashEVMOnlyResult/TestEVMOnlyApplicationProducesDeterministicRoot, and is a consensus behaviour change. Not needed to fix the bug.
Also record in the ADR: after this change a node that crashes between FinalizeBlock(N) and Commit(N) restarts reporting height N (state-committed) rather than N-1. This is correct for Autobahn — block N is already in the durable ledger and the app hash is recomputed identically — and matches the hashvault's idempotent re-commit contract.
Scope
sei-tendermint/internal/evmonlyapp/app.go: cursor changeset, restore on open, Info()/LastBlockHeight() from durable cursor, executor + gasLimit available without InitChain, guard fix.giga/evmonly/flatkv_changeset.go (or a sibling): encode/decode helpers for the cursor keys. Keep the EVM changeset untouched.sei-tendermint/node/public.go: wiring for the constructor change, if (A)'s RequestInitChain route is chosen.giga/evmonly/README.md + the router comment at giga_router_common.go ("nothing was committed"): update to state the durable-Info() requirement any ABCI app in giga mode must meet.- Out of scope: per-tx failure channel (
evmOnlyABCIResults hardcoding CodeTypeOK) — separate ticket; evmonly block-commit logging / sei_chain_evm_block_base_fee for evmOnly (optional follow-up).
Acceptance criteria
- Restart regression test in
evmonlyapp that crosses a storage boundary (the current app_test.go only asserts Info() in-process, which passes while the bug exists): execute blocks 1..N with real txs, storage.Close(), reopen bootstrap.NewGigaStorageManager on the same homePath, construct a fresh app, and assert
Info().LastBlockHeight == N and Info().LastBlockAppHash equals the hash returned by FinalizeBlock(N);FinalizeBlock(N+1) with the sender's next nonce succeeds and its AppHash equals the one an uninterrupted app produces for the same block (determinism across restart);InitChain on the reopened storage returns an error (guard fires at initialHeight == 1).
- Crash-window test:
FinalizeBlock(N) without Commit(N), close/reopen → Info() reports N and FinalizeBlock(N+1) succeeds (no replay of N).
- Fresh-genesis path unchanged:
TestEVMOnlyApplicationExecutesRawEthereumBlock and TestEVMOnlyApplicationProducesDeterministicRoot pass unmodified.
- Cluster check on harbor:
Autobahn + evmOnly: true, 1 validator, allowEmptyBlocks: false; send one EIP-1559 transfer with tipCap >= 1 gwei; wait for sei_chain_flatkv_current_version >= 1; SIGKILL seid → pod comes back, height keeps advancing, no executeBlock(1) panic. Repeat with 4 validators after 400+ blocks.
- ADR merged (can be the same PR, or a docs PR that lands first).
Suggested order
- ADR PR (small; unblocks the design question).
- Implementation PR: failing restart test first, then cursor + restore, then guard fix.
go test ./sei-tendermint/internal/evmonlyapp/ ./giga/evmonly/... via scripts/ramtest.sh (opens stores). make fmtcheck.
- Harbor verification per AC 4; attach the Loki/Prometheus evidence.
References
Summary
Give
evmOnlyApplicationa durable execution cursor so anevmOnlyAutobahn node survives a restart. Today the cursor (committedHeight,appHash,parentHash) lives only inevmOnlyState, so every restart after block 1 reportsInfo().LastBlockHeight == 0, re-runsInitChain, and replays block 1 against a state store that already recovered to height N →nonce too lowpanic, forever.Source: sei-chain#4169 (Linear mirror: SEI-10400). Blocks restart/chaos testing of Autobahn + Giga on harbor.
Root cause (verified against
main@9435e24)StateDB.CommitStateChanges, called insideFinalizeBlock(giga/evmonly/giga_store.go)OpenDBWithRecoveryconverges state/receipts/block store to one height on opencommittedHeight,nextHeightevmOnlyStatestruct fields, advanced inCommitappHash(chained sha256 ofprev ‖ height ‖ blockHash ‖ gasUsed ‖ changeset)evmOnlyState.appHashparentHash(Autobahn header hash fed to the EVM asParentHash)evmOnlyState.parentHashgasLimit,executorInitChainfromConsensusParams.Block.MaxGasgiga_router_common.go:runExecutetrustsapp.Info():last == 0→InitChain+ start atGenDoc.InitialHeight. Its comment ("re-entering on restart is safe — nothing was committed") assumesInfo()is durable, which holds forBaseApp(cms.LastCommitID()) but not here. TheseedInitialStateVersionguard returns early wheninitialHeight == 1, so the "state already at height %d" refusal never fires at the default initial height.Note the ordering quirk that shapes the design: state is committed in
FinalizeBlock, beforecommitAppHashToVaultandCommit. So "durable height" for this app is last finalized-and-state-committed block, and a cursor written only inCommitwould still leave a crash window (FinalizeBlock(N)done,Commit(N)not) that replays N against state N.Decision to record first (short ADR,
docs/rfc/rfc-002-evmonly-durable-cursor.md, fromrfc-template.md)One question: where does the cursor live so it can never disagree with flatkv's
committedVersion?Recommended (A) — the cursor is part of the block's changeset.
committedHeight := storage.SC().GetLatestVersion()afterOpenDBWithRecovery. The storage manager already converges block store / state WAL / receipts to one target, so this is the converged height.appHash,parentHash: written as a secondproto.NamedChangeSet(e.g.Name: "evmonly", keysapphash,parenthash) appended to the EVM changeset in the sameCommitStateChanges(blockNumber, …)call. flatkv routes non-evmnamed changesets to miscDB under"<module>/…"(ktypedocs), so this is supported today. Atomic with state by construction; rolled back/replayed by recovery together with state; deterministic across validators so it is safe to fold into the lthash root.Restore()step called beforeInfo()), ifGetLatestVersion() > 0read the two keys back viaSC().Get(...)and populateevmOnlyState; build the executor;Info()then reports the real height and the router takes its normal restart branch (InitLastHeader+ re-PushAppHash) and never callsInitChain.gasLimit:InitChainis not called on restart, so the app needs the genesis consensus params another way. Recommended:NewEVMOnlyApplicationtakes theRequestInitChain(node already hasGenDoc.ToRequestInitChain()), andInitChainbecomes idempotent w.r.t. it. Alternative: persistgasLimitin the same changeset. Pick one in the ADR.seedInitialStateVersion: drop theinitialHeight == 1early-return;latest != 0must always refuseInitChain(the router will not call it onceInfo()is durable, so this becomes a true invariant guard, not a hole).Alternative (B), to be rejected or chosen explicitly: replace the chained sha256
appHashwith flatkv's lthash root (StateDB.RegisterHashListenerreturns the most recentBlockHash) and recoverparentHashfromstorage.BlockStore().ReadBlockByNumber(last). Cleaner semantically (app hash == state root) but changes what the app hash commits to (dropsgasUsed/blockHash), toucheshashEVMOnlyResult/TestEVMOnlyApplicationProducesDeterministicRoot, and is a consensus behaviour change. Not needed to fix the bug.Also record in the ADR: after this change a node that crashes between
FinalizeBlock(N)andCommit(N)restarts reporting height N (state-committed) rather than N-1. This is correct for Autobahn — block N is already in the durable ledger and the app hash is recomputed identically — and matches the hashvault's idempotent re-commit contract.Scope
sei-tendermint/internal/evmonlyapp/app.go: cursor changeset, restore on open,Info()/LastBlockHeight()from durable cursor, executor + gasLimit available withoutInitChain, guard fix.giga/evmonly/flatkv_changeset.go(or a sibling): encode/decode helpers for the cursor keys. Keep the EVM changeset untouched.sei-tendermint/node/public.go: wiring for the constructor change, if (A)'sRequestInitChainroute is chosen.giga/evmonly/README.md+ the router comment atgiga_router_common.go("nothing was committed"): update to state the durable-Info()requirement any ABCI app in giga mode must meet.evmOnlyABCIResultshardcodingCodeTypeOK) — separate ticket; evmonly block-commit logging /sei_chain_evm_block_base_feefor evmOnly (optional follow-up).Acceptance criteria
evmonlyappthat crosses a storage boundary (the currentapp_test.goonly assertsInfo()in-process, which passes while the bug exists): execute blocks 1..N with real txs,storage.Close(), reopenbootstrap.NewGigaStorageManageron the samehomePath, construct a fresh app, and assertInfo().LastBlockHeight == NandInfo().LastBlockAppHashequals the hash returned byFinalizeBlock(N);FinalizeBlock(N+1)with the sender's next nonce succeeds and itsAppHashequals the one an uninterrupted app produces for the same block (determinism across restart);InitChainon the reopened storage returns an error (guard fires atinitialHeight == 1).FinalizeBlock(N)withoutCommit(N), close/reopen →Info()reports N andFinalizeBlock(N+1)succeeds (no replay of N).TestEVMOnlyApplicationExecutesRawEthereumBlockandTestEVMOnlyApplicationProducesDeterministicRootpass unmodified.Autobahn+evmOnly: true, 1 validator,allowEmptyBlocks: false; send one EIP-1559 transfer withtipCap >= 1 gwei; wait forsei_chain_flatkv_current_version >= 1;SIGKILLseid → pod comes back, height keeps advancing, noexecuteBlock(1)panic. Repeat with 4 validators after 400+ blocks.Suggested order
go test ./sei-tendermint/internal/evmonlyapp/ ./giga/evmonly/...viascripts/ramtest.sh(opens stores).make fmtcheck.References
sei-db/bootstrap/recovery.go(OpenDBWithRecovery,recoveryTarget)sei-db/state_db/sc/flatkv/ktype/ktype.gosei-tendermint/internal/p2p/giga_router_common.go(runExecute,executeBlock)