Deadline-bounded fan-out: is there a case for Async when the constraint is downstream CPU rather than socket waiting?

[aemadrid-twx]

What we're doing

We're building the same feature four times — Ruby/Rails, Elixir/Phoenix, Crystal/Marten, Go — to find out what a deadline-bounded concurrent fan-out actually costs in each runtime. It's a comparison exercise, not a product under deadline pressure, so we have room to do the Ruby side properly.

The Ruby track now serves an identical contract on either Puma or Falcon, selected at boot by one environment variable, with the same specs and the same fixtures on both. The next step would be rewriting our concurrency layer on Async rather than threads.

Before we do that, we'd like a sanity check, because our own measurements point away from the rewrite and we're not confident we're reading them correctly.

The shape of the work

One HTTP request, one 300 ms absolute deadline anchored at controller entry — a monotonic instant, not a duration.

Inside it we fan out N small predictors ("encoders"), each returning one category or nil. They come in three classes with genuinely different cost shapes. Our realistic flow has eleven:

class	what it does	count
pure	arithmetic over request fields	6
local-io	a batched identity read against our own Postgres	3
remote-io	an HTTP call to a third party	2

Four rules shape the design, and they're the reason we can't just reach for a stock executor:

• Whatever hasn't returned at the deadline contributes zero, and we answer anyway. A partial answer on time beats a complete answer late.
• Passing the deadline downgrades work; it does not cancel it. Every unit is in one of two service classes — inside its deadline (high) claims the whole pool, past it (low) gets only the slots high hasn't taken. Low is allowed to starve. Nothing is ever shed.
• Refuse-to-dispatch, not start-and-abandon. An encoder whose declared p99 budget doesn't fit the time remaining is never started, and we record that as a different fact from "started and hadn't returned when we answered".
• Exactly two writes. One synchronous, inside the budget, before the response goes out. One after enrichment finishes, off the request path. A per-request latch orders them, because the second can become ready before the first has returned.

What we have today, in threads

Three fixed thread pools, one per encoder class, process-wide. The pool is about eighty lines:

class Pool
  def initialize(size:)
    @high, @low = [], []
    @mutex, @available = Mutex.new, ConditionVariable.new
    @workers = Array.new(size) { Thread.new { work_loop } }
  end

  def submit(service_class = :high, &task)
    @mutex.synchronize do
      (service_class == :low ? @low : @high) << task
      @available.signal
    end
  end

  private

  def next_task
    @mutex.synchronize do
      loop do
        return @high.shift if @high.any?
        return @low.shift  if @low.any?
        @available.wait(@mutex)
      end
    end
  end
end

Dispatch is tiered — an encoder that consumes another's output runs in a later tier — and the request thread parks on a timed condition variable until the deadline, minus a reserve for its own serialization and first write:

ready, declined = tier.partition { |spec| fits?(spec, deadline) }
ready.each do |spec|
  @pools[spec.encoder_class].submit(:high) { collection.record(run_encoder(spec, ...)) }
end
collection.await(deadline - @post_snapshot_cost_ms, @clock)

def fits?(spec, deadline)
  remaining = deadline - @clock.now_ms - @post_snapshot_cost_ms
  remaining >= spec.expected_budget_ms
end

Post-response continuation is a process-wide single-worker executor with no defined lifetime: @executor.submit { work.call }.

What we measured

We ran a spike to answer one question: do the clients this app actually uses yield to a fiber scheduler while they wait? A client that blocks the thread inside a fiber stalls the whole reactor, which would be strictly worse than our threads while looking fine until load.

Eight concurrent calls, every arm scored against a threads reference on the same machine and the same workload:

workload	fibers + raw pg	fibers + Active Record	threads + AR (reference)
pg_sleep(0.5)	7.1x of 8	7.1x of 8	8.3x of 8
our real identity read	2.1x of 8	2.4x of 8	2.7x of 8
remote call (Net::HTTP)	—	8.0x of 8	7.9x of 8

The good half: pg, Active Record (with isolation_level = :fiber) and Net::HTTP all hand control back while they wait. The failure we were afraid of isn't there.

The half that's stopping us: look at the middle row. Our real identity read overlaps only 2.7x for the threads reference too, on a box that just managed 8x twice with that same substrate. EXPLAIN (ANALYZE) says why — it's a Parallel Seq Scan with two extra workers, roughly a million rows touched, 333,322 rows removed by filter per worker, 257–284 ms. The table carries no index on the identity columns. Every identity read occupies three Postgres backends doing CPU work.

That matches what our load ladder had already shown and we couldn't explain: about 22 req/s at three Puma threads and at sixteen; going 3 → 16 threads moved p50 from 713 ms to 1229 ms with no throughput gain; sixteen threads with a pool of twenty made service time about 50% worse, again with no gain. Meanwhile a control run with the database and the remote taken out stays flat to 50 req/s at a 10 ms p50. Three different client-side concurrency knobs, one saturated resource behind all of them.

So our current read is: a fiber scheduler would add concurrency in exactly the place those sixteen threads did, and the resource it arrives at is doing work, not waiting. That's an argument for adding an index, not for changing substrate. We've since split one pool into three — one per encoder class — on the same reasoning, because forty concurrent identity reads is up to a hundred and twenty Postgres backends.

How much to trust the numbers above

Being straight about this, since it bears on how much of your time the question deserves:

• The probe numbers are from a laptop and are directional only. What we think transfers is the structure — which workloads overlap and which don't — plus the query plan, which is identical on the laptop and on our dev box.
• The ladder figures clear our own quotability bars but came from throwaway runs, so they're indicative rather than published.
• Every performance artifact we have published is currently withdrawn. Our fixture corpus aged out of its own 90-day window, so the one real identity read matched zero rows while still reporting complete. We're rebuilding and re-measuring.
• The biggest hole we know about: our fake remote currently answers with no declared latency, so the two remote calls per request are about a millisecond of JSON each. We have therefore never run this flow against a dependency that genuinely waits — which is precisely the regime a fiber scheduler is aimed at. "Go measure that first" is a completely fair answer.

What we'd like guidance on

1. Is our read right? If the constraint is downstream CPU saturation rather than connection-bound waiting, is there a case for Async here at all — or is "fix the index, raise the remote latency, then ask again" the correct answer? We'd genuinely rather hear that than build the thing.

2. The post-response continuation. Ours is a process-wide executor with no defined lifetime, which structured concurrency clearly has an opinion about. What's the right Async shape for work that outlives the response but must still complete an ordered second write? We deploy on Cloud Run, where the SIGTERM → SIGKILL window is a non-configurable 10 s, and we haven't decided between a transient task (dies instantly on SIGTERM, never delays a deploy) and a non-transient one (holds the drain open, completes more of the work).

3. The deadline. We use a monotonic instant plus an explicit fits? check, so an encoder that can't finish is never started. Does Async's timeout machinery preserve refuse-to-dispatch, or does composing with it push us back toward start-and-abandon?

4. The two service classes. High is inside its deadline and claims everything; low is past its deadline, takes leftovers, may starve, and is never shed. Is that expressible with Barrier and Semaphore, or are we describing something Async deliberately doesn't want to do?

5. falcon-limiter — is it aimed at our regime at all? Its own definition of a long task is "1+ sec and not CPU-bound", and start_delay defaults to 0.1 s, which is a third of our entire budget before the swap takes effect. The gem is in our Gemfile wired to nothing, and we'd rather remove it than wire it wrong.

Happy to share the probe script, the flow definitions, or a reproducible case if any of this is worth a closer look.

[ioquatix]

Please share the full benchmark client and server + instructions to run if it's not obvious.

[ioquatix]

Thanks for the detailed write-up. I think your interpretation is mostly right, with one distinction I’d make: whether Async improves the performance of the saturated database workload is a separate question from whether Async is a useful substrate for expressing the concurrency and lifecycle of the request.

On the database side, your measurements make sense. If the real identity query is consuming substantial CPU in Postgres, increasing the number of Ruby tasks waiting for it does not create more database capacity. Fibers can make waiting cheaper, but they cannot make the downstream resource execute more work in parallel. The 3-vs-16 Puma result is a particularly strong indication of this: throughput stays flat while latency increases, which is what I would expect once the constrained resource is saturated.

So I would definitely fix the index and repeat the experiment with realistic remote latency before drawing conclusions about the final shape. pg_sleep and the Net::HTTP test already demonstrate the regime where the scheduler helps: the Ruby task is genuinely waiting and another task can use the thread.

That said, I would not necessarily frame Async as "adding more database concurrency". Logical concurrency and resource concurrency should be treated separately. You can have many Async tasks while putting an explicit Semaphore around a particular downstream resource:

database_limit = Async::Semaphore.new(3)

database_limit.async do
  run_identity_query
end

The appropriate limit there should come from the capacity of the database/resource, not from the number of fibers or server threads.

Your fits? rule is also distinct from timeout handling, and I would keep it. A timeout normally constrains work which has already started. What you are doing is admission control: if the operation's expected budget does not fit inside the remaining request budget, don't dispatch it at all.

Those compose quite naturally:

if fits?(spec, deadline)
  # Start the task.
else
  # Record that it was declined.
end

You can still use timeout/deadline machinery around the surrounding operation if useful, but Async does not require you to start work and then abandon it.

For the high/low service classes, Barrier and Semaphore don't themselves provide the policy you describe. Barrier is primarily about grouping and waiting for tasks, and Semaphore constrains concurrency. A strict policy of:

high work always takes available capacity
low work only runs in unused capacity
low work may starve

is a scheduling policy on top of those primitives. You could implement it with Async, but I would not try to make Barrier or Semaphore implicitly mean that.

There is one aspect of that design I would examine carefully: if low-priority work is never shed and is allowed to starve indefinitely, then it is effectively a backlog. On Cloud Run, an in-memory backlog cannot also have a strong "nothing is ever lost" guarantee because eventually the process can be terminated. If completion is genuinely required, I would consider whether that portion belongs in a durable job system rather than in the lifetime of the server process.

The post-response continuation is where structured concurrency is probably most relevant. I would avoid an unowned process-global executor. Instead, make the ownership explicit: request-critical tasks belong to the request; work which is intentionally allowed to outlive the response should belong to a longer-lived task associated with the server/process.

Conceptually:

server task
├── request task
│   └── work needed before response
└── enrichment task
    └── ordered second write

Then shutdown policy becomes an explicit property of that task. If enrichment is transient, it can be stopped during shutdown. If it is non-transient, it can participate in graceful shutdown. Whether the latter is useful with Cloud Run's 10-second termination window depends on how valuable partial completion during drain is to you.

The ordering between the two writes is orthogonal to this. Keeping an explicit latch/event for "first write completed" is reasonable if the enrichment can race ahead of it.

I don't think falcon-limiter is particularly well matched to the problem as described. It is intended to handle tasks which unexpectedly remain active for relatively long periods, especially I/O-bound ones. You already know the resources you need to protect, you have a 300 ms application deadline, and you have explicit admission and concurrency policies. Direct resource limits are both simpler and more predictable here.

So my short version would be:

• Fix the database query first; Async won't solve downstream CPU saturation.
• Add realistic latency to the remote dependency and rerun the test.
• Keep the absolute monotonic deadline and fits? admission check.
• Treat Async task concurrency separately from downstream resource concurrency; use explicit limits where appropriate.
• Your high/low policy is implementable, but it is application scheduling policy rather than something Barrier/Semaphore provide directly.
• Give post-response work an explicit owner and shutdown policy.
• If "never shed" really means "must eventually complete", consider durable work rather than an in-process queue.

Once the indexed DB workload and realistic remote latency are in place, I think the measurements will give you a much better answer about whether replacing the thread-based orchestration is worthwhile. I wouldn't rewrite it purely to obtain more concurrency against the current query.

[aemadrid-twx]

Thanks. We fixed the index and it changes the picture; details below.

The index

Same host, same corpus, same identity, the query shape the app issues:

arm	plan	buffers	rows removed by filter	execution
before	Parallel Seq Scan, 2 workers	39,603	333,323 per worker	172.79 ms
after	Index Scan	35	0	1.06 ms

The index is (client_id, <identity column>, observed_at DESC), so the two equalities lead and the timestamp serves both the window and the ordering. The scan stops at the limit instead of sorting the whole match set.

Two things we hit on the way, both of which would have quietly invalidated the re-run.

Our probe queried a different column than the application, and had since it was written, while its own comments claimed it mirrored the app. Harmless while neither column was indexed and both cost the same scan. Once only one of them was indexed, re-running unchanged would have reproduced a seq scan the app no longer performs, and confirmed the old verdict for the wrong reason.

Our declared per-encoder budgets are now wrong in the safe direction. The identity encoders still declare 60 ms, which in our design is the threshold below which the scheduler declines to start an encoder at all, not a timeout. Against a read that now costs about a millisecond, that threshold is roughly sixty times too high, so near the deadline it would decline work that would have fit. We have not measured how often that happens, only that the number is wrong. Before re-running the ladder we have to decide whether that flow keeps the old declaration as an audit of the unindexed case or gets a new one, and a new one needs a load run to pick, since the declared value is meant to be a high percentile rather than a typical cost.

The benchmark you asked for

Coming, and it should have come three days ago. It will be a small standalone repo rather than our harness, because the harness reads a schema copied from a table we cannot publish.

It will hold a generator for a synthetic observation table (about a million rows, an identity key, a timestamp spread over 180 days), the fiber-scheduler probe rewritten to run without our application, the fake remote with its configurable hold, and a README with the exact commands. Both arms will be a flag, since indexed versus unindexed is a rebuild rather than a switch in the probe.

I will post the link in this thread when it is up.

What your answer changed

Recording these because the next person reading the thread should see which way each landed.

Admission control stays, and stays separate from timeouts. We had half-assumed we would express the fits? check through whatever timeout machinery Async offered. Your framing confirms the distinction we built on, and it means the deadline logic survives a substrate change untouched. That was the risk we were most worried about.

We will stop expecting the service classes for free. We had Barrier and Semaphore down as a possible drop-in for the high/low policy. Taking them as grouping and concurrency primitives, with the policy as ours to write on top, makes the rewrite's cost clearer: we would be reimplementing the one part of our current pool that is actually load-bearing.

The post-response continuation gets an owner. The explicit tree, with a server task owning both the request task and the longer-lived enrichment task, is what we were missing. Ours is a process-global executor with no owner other than the process, which we knew was wrong without being able to say what right looked like.

One correction to our own post while we are here: we described the separate downstream limits as something we would adopt. We already do that, and had it written down before we asked; each of our three pools is sized from the resource behind it rather than from a worker count. Your answer confirmed the rule rather than changing it, and the post gave the wrong impression.

The backlog point

You are right, and it is the part we had not admitted to ourselves. Work not started at the deadline is going to a durable queue, which narrows what a process death can lose to the work already in flight and makes the rest a queue's problem rather than a promise we cannot keep. The ordered second write becomes a cross-process question, which is ours to solve and separate from the substrate.

We will do your step two first. The fake remote's hold has only ever been exercised by the probe, at 200 ms; our load ladder has always run it at zero, so the flow has never met a dependency that genuinely waits.