Proposal: Cross-runtime correctness validation for asyncband primitives

[jiengup]

Motivation

Asyncband advertises itself as runtime-agnostic, but our test suite is effectively Tokio-only: all tests run under #[tokio::test] (mostly the single-threaded current-thread runtime), and benchmarks poll futures directly without a real executor. This leaves waker registration, cancellation, and wake-up races under real multi-threaded scheduling unverified, which undermines the core promise of the crate.

This proposal adds a cross-runtime correctness validation suite that exercises the library's primitives under the commonly used Rust async runtimes and gates it in CI.

Target runtimes

Primary matrix (all platforms, active maintenance):

• tokio – de facto standard; multi-threaded work-stealing runtime. Required baseline.
• async-std – still one of the mainstream runtimes listed in the official async book.
• smol – the lightweight, composable runtime; representative of the async-executor/async-lock ecosystem.
• compio - one thread per core async runtime

Secondary matrix (Linux-only):

• glommio – io_uring-based thread-per-core runtime (no Mutex, tasks are !Send).
• monoio – io_uring-based thread-per-core runtime (limited sync primitives).

Scope

• Build a small runtime-agnostic test harness (spawn / join / yield / timeout adapters) and run a shared semantic test suite for each primitive on every runtime in the primary matrix.
• Validate guaranteed properties only (no lost wakeups, correct cancellation, close/disconnect semantics, single-flight/once dedup, no deadlock on single-threaded executors) — not implementation-specific fairness guarantees.
• CI: run the primary matrix on Ubuntu, macOS, and Windows (stable toolchain only); run the secondary matrix on Ubuntu as a Linux-only job.
• Keep the existing Tokio test suite and microbenchmarks unchanged.

Open questions

• Should the secondary matrix (glommio, monoio) be required or informational in CI?
• Do we want benchmark comparisons against runtime-native primitives(e.g. tokio::sync::Mutex) in this proposal, or a follow-up?
• Where should the harness live: a separate non-workspace test crate to keep heavy runtime dependencies out of regular cargo test?
• Is there any project that we can leverage? e.g. loom

[tisonkun]

Thanks for starting this thread @jiengup!

I agree that test coverage would help in developing async primitives like this.

In a very early stage I opened #21 but since the implementation often self-proven and it takes a lot of time to build reasonable testkit without deeply hacking the primitive internals I don't spend too much time on it.

But nowadays AI agents can eliminate a lot of implement burden while you have the proper intention, so I agree that it's time to investigate cross-runtime correctness validation methods.

My main focus would be, "try your best to avoid hacking primitives internals only for testing".

[jiengup]

"try your best to avoid hacking primitives internals only for testing".

@tisonkun I agree on this point. Some thoughts after diving into some related projects:

For the current stage, I propose a test suite structure as:

• A separate, non-workspace test crate, keeping runtime dependencies out of regular cargo test.
• Shared black-box contract tests using only asyncband’s public API.
• Small runtime capability adapters for spawn, yield_now, timeout, and join among the common runtimes.
• Separate local/single-threaded and multi-threaded suites.

Demo code:

trait Runtime {
    type Join<T>: Future<Output = Result<T, TaskError>>;

    fn spawn<F, T>(future: F) -> Self::Join<T>
    where
        F: Future<Output = T> + Send + 'static,
        T: Send + 'static;

    fn yield_now() -> impl Future<Output = ()>;

    fn timeout<F>(
        duration: Duration,
        future: F,
    ) -> impl Future<Output = Result<F::Output, TimedOut>>
    where
        F: Future;
}

async fn mutex_contract<R: Runtime>() {
    let mutex = Arc::new(asyncband::Mutex::new(0));

    let tasks = (0..10).map(|_| {
        let mutex = mutex.clone();
        R::spawn(async move {
            for _ in 0..100 {
                *mutex.lock().await += 1;
                R::yield_now().await;
            }
        })
    });

    for task in tasks {
        R::timeout(Duration::from_secs(5), task)
            .await
            .expect("possible deadlock")
            .expect("task failed");
    }

    assert_eq!(*mutex.lock().await, 1000);
}

Cancellation should not depend on runtime-specific task handles because, for example, dropping a Tokio JoinHandle detaches the task while dropping a smol Task cancels it. Instead, the primitive future can be cancelled inside the task:

let outcome = select(primitive_future, cancel_signal).await;
// If cancel_signal wins, primitive_future is dropped here.

A small wrapper can signal when the primitive future first returns Pending. The actor then performs notify, send, release, or cancellation, ensuring the test exercises a real executor waker without using sleeps or inspecting internal state.

This should cover public guarantees such as:

• lost wakeups
• cancellation recovery
• close/disconnect semantics
• deduplication
• and single-threaded deadlocks.

Existing white-box tests that deliberately pause an internal state machine should remain as unit tests; a black-box cross-runtime suite cannot deterministically replace those narrow race tests. Newly added primitives also need to include the scheduling windows that cannot be covered by the test suite in the internal white-box testing.

In the long run, based on the "Runtime Agnostic" principle of this project, should we build something similar to loom that can enumerate various polling scenarios (such as the mapping between features and OS threads, and whether tasks can be stolen, etc.) to make the implementation of primitives robust? Of course, this is very complex and requires more abstraction and discussion...

[tisonkun]

Sure, I think the overall approach is fine. You can go ahead and try implementing your idea with a few primitives—things like semaphores or mutexes—or wherever you think is best to start.

Honestly, rather than spending more time on the architectural design, I'd rather see a working demo so I can get a feel for whether it's on the right track. Right now, adding an extra trait feels a bit unnecessary, and I'm not entirely clear on the reasoning behind it. Plus, are you planning to use Loom, or build our own custom equivalent? It's hard to make a call until I see something running.

It might be best to take some time to think it through, pick a primitive or two, and put together a basic test demo so we can review it together.

[orthur2]

I like the direction. For a runtime-agnostic crate, having some black-box contract tests outside Tokio definitely makes sense.

That said, adding more runtimes is only one side of the story. Asyncband does not interact with a runtime directly; it only talks to the executor through poll and Waker. The exact poll interleavings still matter a lot.For example, what happens when a waker is registered or replaced, a wake arrives at an awkward moment, or a pending future is dropped. You can already see this style of testing in some of the existing test cases, where futures are polled manually. If the goal is correctness validation, I do not think this side should be overlooked.

As a side note, I noticed that async-std’s latest commit was last year, and async-std now describes itself as deprecated in favor of smol, so I would probably keep it as an optional compatibility job rather than a required primary runtime.

I would also clarify the contract per primitive before defining the shared suite. The current Semaphore and RwLock documentation already promises fair/FIFO behavior, so those properties are public guarantees rather than merely implementation details. They should either be covered by the suite or intentionally revised.

I agree with @tisonkun that a small running prototype will answer some of the harness questions faster than more upfront design.

Thanks again for putting this proposal together. I’d be happy to help if needed.

[jiengup]

I like the direction. For a runtime-agnostic crate, having some black-box contract tests outside Tokio definitely makes sense.

That said, adding more runtimes is only one side of the story. Asyncband does not interact with a runtime directly; it only talks to the executor through poll and Waker. The exact poll interleavings still matter a lot.For example, what happens when a waker is registered or replaced, a wake arrives at an awkward moment, or a pending future is dropped. You can already see this style of testing in some of the existing test cases, where futures are polled manually. If the goal is correctness validation, I do not think this side should be overlooked.

As a side note, I noticed that async-std’s latest commit was last year, and async-std now describes itself as deprecated in favor of smol, so I would probably keep it as an optional compatibility job rather than a required primary runtime.

I would also clarify the contract per primitive before defining the shared suite. The current Semaphore and RwLock documentation already promises fair/FIFO behavior, so those properties are public guarantees rather than merely implementation details. They should either be covered by the suite or intentionally revised.

I agree with @tisonkun that a small running prototype will answer some of the harness questions faster than more upfront design.

Thanks again for putting this proposal together. I’d be happy to help if needed.

Thanks for your feedback! I did some research and prototypes these days and came up with some solutions. I will push a patch soon so that we can discuss more.