dev(mock_block_validation): node-local SC migration rate override + arctic-1 FlatKV/ingress findings

[alexander-sei]

Describe your changes and provide context

Draft, stacked on feat/ingress-node-mode. Adds a dev-only way for a single node to walk the FlatKV migration (memiavl_only → migrate_evm → … → flatkv_only) against a chain whose migration/NumKeysToMigratePerBlock gov param is still 0, so FlatKV-only and --ingress bootstrapping can be exercised against live network state without the network migrating.

Why this is needed

The migration is consensus-relevant: from the first migration commit the AppHash composition changes (evm_lattice appended, memiavl infos dropped at V3), and the per-block rate comes exclusively from chain state (applyMigrationBatchSize). A node cannot advance on its own without (a) a node-local rate and (b) tolerating the resulting AppHash mismatch. (b) already exists: the mock_block_validation consensus policy swallows ErrAppHash at block validation, the ABCI handshake, and state-sync verifyApp, and CI already publishes that image. This PR adds (a), in that build only.

What changes

• app/unsafe_migration_batch_size_mock.go (//go:build mock_block_validation): reads SEI_UNSAFE_MIGRATION_BATCH_SIZE once at process start and substitutes it for the on-chain rate. A malformed value panics every subcommand rather than leaving the migration silently paused. Logs one UNSAFE error line on first use.
• app/unsafe_migration_batch_size.go (!mock_block_validation): identity function. Production builds are unchanged.
• app/abci.go: applyMigrationBatchSize applies the override between the gov-param read and the MaxNumKeysToMigratePerBlock clamp, so the override cannot exceed the max.

Only the V0→V1 kick-off is automatic (rootmulti.SetMigrationBatchSize); V1→V2 and V2→V3 have no live trigger and are entered by pinning sc-write-mode with sc-write-mode-enable-auto = false, which NewMigrationManager accepts as long as the persisted version is the mode's start or target version.

What it was used for (arctic-1, MacBook M5 / 32 GiB)

Anchor: state-synced onto arctic-1 with the strict build first (followed ~1,000 blocks with AppHash verification on, i.e. main is consensus-compatible with v6.7.0-rc1 at V0), then switched to the mock build with SEI_UNSAFE_MIGRATION_BATCH_SIZE=100000:

phase	batches	keys	time
V0→V1 migrate_evm	68	6,755,267	29.7 s
V1→V2 migrate_all_but_bank	14	1,328,695	~10 s
V2→V3 migrate_bank	8	775,452	~2 s

The node kept executing arctic-1 blocks throughout. arctic-1's whole state is ~2 GB (memiavl 2.3 GB → FlatKV 1.3 GB). RSS: 2.5 GiB at V0 (memiavl), 6.2 GiB mid-migration (both backends open), 0.7 GiB right after the flatkv_only restart, 2.7 GiB an hour later as the default caches fill.

Ingress: a second local node (seid start --ingress, same mock build) state-synced from the anchor's format-2 FlatKV snapshot (41 chunks, 390 MB), passed validateIngressProfile, and followed the chain. Anchor and ingress report byte-identical last_block_app_hash at the same height, so a store migrated live by the MigrationManager and a store restored from its snapshot agree. Contract code, nonce and storage also match the public arctic-1 RPC. Ingress RSS 0.66–0.77 GiB after 1 h (heap ~0.4 GiB, Pebble caches ~134 MiB, view cache 57 MB of a 960 MiB ceiling).

Findings for feat/ingress-node-mode (not addressed here)

1. SS-off breaks the block APIs. eth_getBlockByNumber, eth_getBlockReceipts, eth_feeHistory, eth_getBlockTransactionCountByNumber fail 100% with -32603 method handler crashed. latest resolves to the receipt-watermark height (≤ committed−1), EncodeTmBlock needs a state ctx there, and CacheMultiStoreWithVersion refuses any non-live version without SS (rootmulti/store.go:421-426). The ingress profile forces SS off, so this lands on every ingress node; the panic is also surfaced as "handler crashed" instead of a clean JSON-RPC error.
2. latest state reads race with commits on SS-off nodes: 6/40 eth_getBalance(latest) calls failed with evm module does not exist on height N. CacheMultiStoreWithVersion(latest) returns the live SC stores and CheckVersion's VersionExists(N) is N == tree.Version(), which flips once the next block commits. Neither issue is FlatKV- or migration-specific.
3. State sync requires two peers even when one node is the only snapshot source (statesync/reactor.go:383); a validator works as the second.
4. configureIngressProfile hard-codes rpc.laddr = 127.0.0.1:26657.
5. /genesis from the RPC serialises vote_extensions_enable_height as a string; GenesisDocFromFile rejects it and seid start nil-derefs on the ignored error (sei-cosmos/server/start.go:185).

Footprint observations (FlatKV / ingress)

• The FlatKV directory holds the store ~3×: working/ (561 MB), the last snapshot as a full independent copy (663 MB, link count 1), and the changelog/ since that snapshot (738 MB after ~9k blocks; sc-snapshot-interval is 10,000). Hard-linked checkpoints plus a shorter interval for ingress would bring this to ~1.1–1.3× logical.
• Code is keyed by address (ktype.go:54). Dumping arctic-1's code bucket: 24,625 entries, 9,514 unique bytecodes; 61% of entries are duplicates and 42.6% of code bytes would go away with code-hash keying (one 11 KB bytecode is deployed 1,363 times).
• chunkQueue.Next never deletes applied chunk files, so the whole snapshot sits in statesync.temp-dir until the restore ends (~14 GiB for atlantic-2-sized state; atlantic-2's snapshot is 1,116 chunks vs arctic-1's 51).
• The 960 MiB view cache lives in the Go heap; HeapSys reached 1.58 GiB against 0.41 GiB live after the restore. A debug.SetMemoryLimit derived from the cache ceilings would make ingress RSS predictable.
• Already fine: zstd on every Pebble level, no bloom filter on L6, receipts pruned by min-retain-blocks.

Extrapolated to atlantic-2 (21.9× the compressed state): ingress ~1.8–2.2 GiB RSS / ~30–40 GB disk; a default-profile full node ~9–10 GiB as the 8 GiB of caches actually fill.

This override must never run on a validator or a publicly reachable node. The build it lives in also swallows DataHash mismatches and masks any divergence, not just the migration's.

Testing performed to validate your change

• go test ./app/ -run TestUnsafeMigrationBatchSizeOverride and go test -tags mock_block_validation ./app/ -run 'TestReadUnsafeMigrationBatchSize|TestUnsafeMigrationBatchSizeOverride' pass; go vet clean under both tags; gofmt -s / goimports clean.
• Binary smoke: SEI_UNSAFE_MIGRATION_BATCH_SIZE=lots seid-mock version panics with the parse error; the strict build ignores the variable.
• End-to-end on arctic-1 as described above: V0→V3 on one node, FlatKV snapshot production, --ingress state sync from it, AppHash agreement between anchor and ingress, EVM state agreement with the public RPC.

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 14, 2026, 1:06 PM