Skip to content

chore: Concurrent state load - #2466

Open
sergerad wants to merge 4 commits into
nextfrom
sergerad-concurrent-load
Open

chore: Concurrent state load#2466
sergerad wants to merge 4 commits into
nextfrom
sergerad-concurrent-load

Conversation

@sergerad

@sergerad sergerad commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Relates to #1697.

Why:

  • Store startup on testnet is ~16s. The largest instrumented cost is
    verify_account_state_forest_consistency (6.9s, ~42%), which checked every public account
    serially. The remaining structures (MMR, account tree, nullifier tree, forest) also loaded
    strictly one after another despite living in independent storages, and the three blocking
    RocksDB opens ran directly on the async runtime.
  • Operators get a faster node startup; there is no functional or API change.

How:

  • Parallelized the forest consistency scan: pages are still fetched via the existing cursor,
    but each page's per-account checks (storage-header decode + forest lookups) now run through
    rayon. The forest parameter gained a Sync bound.
  • Overlapped the independent loads: all loader functions now take &Db instead of &mut Db
    (every underlying query is &self; the &mut threading was an API artifact), the Db is
    Arc-wrapped up front, and the MMR, account tree, nullifier tree, and forest (+ its verify)
    load in four spawned tasks joined with try_join!. Spawned tasks rather than joined futures
    because loading has long synchronous sections (RocksDB opens, MMR hashing, SMT top
    reconstruction) that would serialize if polled from a single task.
  • The three RocksDB opens now run in spawn_blocking_in_current_span, so they execute
    concurrently, off the async workers, and remain traced. Panics from load tasks are resumed
    on the caller so they propagate as panics rather than being masked as errors.
  • The chain tip is now read from a cheap latest-header query instead of the loaded MMR; this
    is safe because load_mmr's existing consistency check pins the MMR to that same header.
  • Tradeoff: CPU-bound sections inside the async loaders block runtime workers while they run.
    This is fine during startup (the write worker and gRPC servers are not up yet); if loading
    ever runs concurrently with live traffic, those sections should move to the blocking pool.
  • Follow-up: flushing RocksDB WALs on graceful shutdown (to avoid WAL recovery on the next
    open) needs a flush API upstream in miden-crypto, which does not currently expose one.

Results:

Local benchmark (benchmark-store load-state, toy-scale store): the three RocksDB opens now
start simultaneously (~22ms of summed open time → ~7.5ms wall clock), warm loads dropped from
~18–20ms to ~12–13ms.

Iteration 0: state loaded in 23.321459ms   ← cold: WAL recovery + cold page cache
Iteration 1: state loaded in 13.185792ms
Iteration 2: state loaded in 11.870416ms

Warm-iteration trace — note the identical start timestamps on the three RocksDB opens,
which previously ran strictly one after another:

load [ 27.1ms ]
┕━ load_with_database_options [ 27.1ms ]
   ┝━ load_with_pool_size [ 2.96ms | 10.93% ]
   ┝━ load_mmr [ 70.4µs | 0.26% ]
   ┝━ open_tree_storage [ 7.09ms | 26.16% ] path: "accounttree"        ← t+3.5ms
   ┝━ open_tree_storage [ 7.36ms | 27.15% ] path: "nullifiertree"      ← t+3.5ms
   ┝━ open_forest_storage [ 7.52ms | 27.74% ] path: "accountstateforest" ← t+3.5ms
   ┝━ load_account_state_forest [ 9.88µs | 0.04% ]
   ┝━ load_account_tree [ 914µs | 3.37% ]
   ┝━ load_nullifier_tree [ 608µs | 2.24% ]
   ┝━ verify_account_state_forest_consistency [ 517µs | 1.91% ]
   ┕━ verify_tree_consistency [ 54.3µs | 0.20% ]

Against the serial baseline on the same machine: warm loads dropped from ~18–20ms to
~12–13ms, with ~22ms of summed open time (7.1 + 7.4 + 7.5ms) collapsing to ~7.5ms of wall
clock. The tree/MMR loads and the forest verify also overlap with each other now. The
absolute numbers are toy-scale (200 accounts); on testnet data the same structure applies to
the 6.9s forest scan (rayon-parallel per page) and the three multi-second opens.

Changelog

[[entry]]
scope       = "node"
impact      = "changed"
description = "Improved node startup time by loading and verifying Merkle tree structures in parallel."

@sergerad
sergerad requested review from Mirko-von-Leipzig and kkovaacs and removed request for kkovaacs August 11, 2026 03:27
Comment thread crates/store/src/state/lifecycle.rs Outdated
Comment thread crates/store/src/state/lifecycle.rs Outdated
Comment on lines +205 to +207
// because loading has long synchronous sections (RocksDB opens, MMR hashing, SMT top
// reconstruction) that would serialize if polled from a single task.
let (blockchain, account_tree, nullifier_tree, forest) = tokio::try_join!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not use our task set for this? This is really difficult to read.

@sergerad sergerad Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Tasks struct is designed around long-running service tasks. It should probably be renamed to reflect its specific use case.

The main issue is that Tasks only allows for futures returning anyhow::Result<()> which means we can't coalesce results via join. Even if it was Result<T> it wouldn't help here because every future returns a different type.

@Mirko-von-Leipzig

Mirko-von-Leipzig commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Could we also focus on getting the changelog entries more user friendly? At the moment they're written as for us - overly technical and specific.

"Store startup now opens its storage backends and loads the chain MMR, 
account tree, nullifier tree, and account state forest concurrently, and 
verifies the account state forest in parallel, reducing node startup time."

No operator/user knows (or cares) about the store, forests, trees, or concurrency etc; as an example this could just be:

Improved node startup time by loading and verifying Merkle tree structures in parallel

Base automatically changed from sergerad-bench-load to next August 17, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants