Skip to content

Prevent ChannelMonitor persistence cancellation during shutdown - #1229

Draft
comwanga wants to merge 1 commit into
lightningdevkit:mainfrom
comwanga:2026-11-monitor-persistence-shutdown
Draft

comwanga wants to merge 1 commit into
lightningdevkit:mainfrom
comwanga:2026-11-monitor-persistence-shutdown

Conversation

@comwanga

@comwanga comwanga commented Nov 10, 2026 •

Copy link
Copy Markdown

Refs #1081.

Problem

The async ChannelMonitor persister spawns its tasks through RuntimeSpawner, so they are ordinary cancellable background tasks. Node::stop() can therefore lose the completion of a monitor write in two ways:

  • A write that is still in flight is cancelled together with the other cancellable tasks.
  • The background processor flushes pending monitor writes once more after its final ChannelManager persist. By then the task tracker is already closed, so the spawn is refused (since 10ac933).

Either way the ChainMonitor is never told that the write completed. After stop() and start() on the same Node, the channel is still waiting on that update and can't make progress until the node is rebuilt.

What changed

Monitor persistence now uses its own MonitorPersistenceSpawner. Its tasks are neither cancelled nor refused by stop(). They are cancelled once the Runtime is dropped, which also holds when we run on a runtime handle supplied by the user. RuntimeSpawner is now only used for gossip verification, and all other cancellable tasks behave as before.

stop() does not wait for outstanding monitor writes. A write may complete after stop() returns, while any unfinished persistence tasks are cancelled when the runtime is dropped. This intentionally exempts monitor persistence from shutdown cancellation and from the late-spawn rejection introduced in 10ac933.

Testing

  • tests/integration_tests_monitor_persistence.rs adds four tests that stop a node with a monitor update write in flight, not yet issued, or only flushed during shutdown, and with the initial monitor write in flight. Each then restarts the same Node and expects payments to go through. All four fail without the src/ changes and pass with them.
  • Two unit tests in runtime.rs check that the tasks survive stop and are cancelled when the runtime is dropped, for both an owned runtime and an external handle.
  • Locally I ran the lib tests, cargo fmt --check, the CI clippy invocation, and cargo check with uniffi-default and with chain-electrum,storage-postgres (i.e. without chain-bitcoind).
  • I could not get integration_tests_rust through in one uninterrupted run on my machine (memory), so I ran it in batches with CI's --cfg cycle_tests --cfg tokio_unstable. Each of the 89 tests on the base I tested (79e4d92) passed at least once. Three failed intermittently: a_close_is_classified_for_a_channel_whose_facts_were_never_recorded fails the same way without this change, and two others hit the block-wait timeout in the test helpers once each and did not reproduce with or without it.

I also ran the doc tests, reorg_test, and an MSRV build (Rust 1.85.1, with the dependency pins the CI MSRV job uses). Under uniffi-default I ran the lib tests and the new regression tests; the rest of the UniFFI suite I left to CI.

Related recovery problem

The scenario as written in #1081 (stop, drop, rebuild from the store) is still stuck with this change, because of a separate problem in rust-lightning: if the initial ChannelMonitor was persisted but the last saved ChannelManager still has it pending, the reloaded manager keeps waiting for it. This PR does not address that path.

I have a small fix for it locally. With it applied to the pinned rust-lightning revision (d7a9806) together with this PR, the rebuild scenarios I tested pass. It is not merged or backported; I've described the problem in lightningdevkit/rust-lightning#4765.

One side effect to be aware of: if the initial monitor write of an inbound channel had not been issued yet at stop() and the node is then rebuilt, the write used to be lost and the channel was dropped on reload. Now the write can complete after stop(), so the rebuilt node runs into the reload problem above instead. Stores that issue writes eagerly, like SQLite, already end up there on main.


This change was prepared with the assistance of AI tools (Claude Code).

Our async `ChannelMonitor` persister spawned its tasks via
`RuntimeSpawner`, i.e., as cancellable background tasks. `Node::stop`
could therefore lose the completion of a monitor write in two ways: a
write still in flight was cancelled along with all other cancellable
tasks, and a write the background processor only flushed after its
final `ChannelManager` persist was refused, as the task tracker had
already been closed by then. In both cases the `ChainMonitor` never
learns that the write completed, and the channel keeps waiting on it
after the same `Node` is started again.

Here, we give the persister a dedicated `MonitorPersistenceSpawner`.
Its tasks are neither cancelled nor refused by `stop`, and are only
cancelled once the `Runtime` is dropped, which also holds when we run
on an externally supplied runtime handle. The spawner holds a runtime
handle rather than an `Arc<Runtime>`, as the spawned tasks own the
persister and must not keep the runtime alive. `RuntimeSpawner` is now
only used for gossip verification, and all other cancellable tasks
behave as before.

Note that this does not fix rebuilding a node whose initial monitor
write completed without the `ChannelManager` being persisted
afterwards. That needs a change to how LDK reloads such a channel.

This change was prepared with the assistance of AI tools (Claude Code).
@ldk-reviews-bot

Copy link
Copy Markdown

👋 Hi! I see this is a draft PR.
I'll wait to assign reviewers until you mark it as ready for review.
Just convert it out of draft status when you're ready for review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants