Repository navigation
Conversation
Our async `ChannelMonitor` persister spawned its tasks via `RuntimeSpawner`, i.e., as cancellable background tasks. `Node::stop` could therefore lose the completion of a monitor write in two ways: a write still in flight was cancelled along with all other cancellable tasks, and a write the background processor only flushed after its final `ChannelManager` persist was refused, as the task tracker had already been closed by then. In both cases the `ChainMonitor` never learns that the write completed, and the channel keeps waiting on it after the same `Node` is started again. Here, we give the persister a dedicated `MonitorPersistenceSpawner`. Its tasks are neither cancelled nor refused by `stop`, and are only cancelled once the `Runtime` is dropped, which also holds when we run on an externally supplied runtime handle. The spawner holds a runtime handle rather than an `Arc<Runtime>`, as the spawned tasks own the persister and must not keep the runtime alive. `RuntimeSpawner` is now only used for gossip verification, and all other cancellable tasks behave as before. Note that this does not fix rebuilding a node whose initial monitor write completed without the `ChannelManager` being persisted afterwards. That needs a change to how LDK reloads such a channel. This change was prepared with the assistance of AI tools (Claude Code).
|
👋 Hi! I see this is a draft PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1081.
Problem
The async
ChannelMonitorpersister spawns its tasks throughRuntimeSpawner, so they are ordinary cancellable background tasks.Node::stop()can therefore lose the completion of a monitor write in two ways:ChannelManagerpersist. By then the task tracker is already closed, so the spawn is refused (since 10ac933).Either way the
ChainMonitoris never told that the write completed. Afterstop()andstart()on the sameNode, the channel is still waiting on that update and can't make progress until the node is rebuilt.What changed
Monitor persistence now uses its own
MonitorPersistenceSpawner. Its tasks are neither cancelled nor refused bystop(). They are cancelled once theRuntimeis dropped, which also holds when we run on a runtime handle supplied by the user.RuntimeSpawneris now only used for gossip verification, and all other cancellable tasks behave as before.stop()does not wait for outstanding monitor writes. A write may complete afterstop()returns, while any unfinished persistence tasks are cancelled when the runtime is dropped. This intentionally exempts monitor persistence from shutdown cancellation and from the late-spawn rejection introduced in10ac933.Testing
tests/integration_tests_monitor_persistence.rsadds four tests that stop a node with a monitor update write in flight, not yet issued, or only flushed during shutdown, and with the initial monitor write in flight. Each then restarts the sameNodeand expects payments to go through. All four fail without thesrc/changes and pass with them.runtime.rscheck that the tasks survivestopand are cancelled when the runtime is dropped, for both an owned runtime and an external handle.cargo fmt --check, the CI clippy invocation, andcargo checkwithuniffi-defaultand withchain-electrum,storage-postgres(i.e. withoutchain-bitcoind).integration_tests_rustthrough in one uninterrupted run on my machine (memory), so I ran it in batches with CI's--cfg cycle_tests --cfg tokio_unstable. Each of the 89 tests on the base I tested (79e4d92) passed at least once. Three failed intermittently:a_close_is_classified_for_a_channel_whose_facts_were_never_recordedfails the same way without this change, and two others hit the block-wait timeout in the test helpers once each and did not reproduce with or without it.I also ran the doc tests,
reorg_test, and an MSRV build (Rust 1.85.1, with the dependency pins the CI MSRV job uses). Underuniffi-defaultI ran the lib tests and the new regression tests; the rest of the UniFFI suite I left to CI.Related recovery problem
The scenario as written in #1081 (stop, drop, rebuild from the store) is still stuck with this change, because of a separate problem in rust-lightning: if the initial
ChannelMonitorwas persisted but the last savedChannelManagerstill has it pending, the reloaded manager keeps waiting for it. This PR does not address that path.I have a small fix for it locally. With it applied to the pinned rust-lightning revision (d7a9806) together with this PR, the rebuild scenarios I tested pass. It is not merged or backported; I've described the problem in lightningdevkit/rust-lightning#4765.
One side effect to be aware of: if the initial monitor write of an inbound channel had not been issued yet at
stop()and the node is then rebuilt, the write used to be lost and the channel was dropped on reload. Now the write can complete afterstop(), so the rebuilt node runs into the reload problem above instead. Stores that issue writes eagerly, like SQLite, already end up there onmain.This change was prepared with the assistance of AI tools (Claude Code).