Roadmap after 3.0: developer experience, telemetry and feature improvements (review findings) #51

Findings from a three-part review of the code as it stands in 3.1 (#50): developer experience, telemetry and operations, and features compared with Azure Functions queue triggers, the Service Bus processor, MassTransit, Wolverine, Rebus and SQS libraries. Every "missing" or "wrong" claim was checked against the code. Sizes: S is under a day, M is a few days, L is a week or more. Everything is additive (fits a 3.x release) unless marked breaking.

Fix before releasing 3.1 (cheap now, a compatibility problem later)

• Future message formats would be misread by 3.1 apps (S). The trace wrapper reader (Telemetry/TraceContextEnvelope.cs) treats an unknown version or an extra field as "not a wrapper" and hands the raw JSON to the serializer. System.Text.Json ignores unknown fields, so a 3.1 app reading a 3.2 message gets an object with empty fields instead of an error. Fix: if the marker is present but the version is unknown, set Error so the message is dead-lettered and can be replayed after upgrading. Allow optional fields to be added later, and make traceparent optional so the wrapper can carry other data (headers, a dead-letter reason, a blob pointer). Several items below depend on this.
• Poison-queue messages are deleted after 7 days (S). StorageQueue.DeadLetterAsync sends with the default time-to-live, which is the service's 7-day maximum. A poison message nobody looks at for a week is silently gone. Add PoisonTimeToLive, defaulting to never expire.
• Processing-duration percentiles are meaningless (S, bug). queue_message_processing_duration_seconds has no bucket advice or unit, so OTel's default millisecond-scale buckets put almost every message in the 0–5 s bucket. Add InstrumentAdvice with second-scale buckets and unit: "s". queue_message_size_bytes has the same problem: its default buckets stop at 10,000 bytes, but messages go up to 64 KiB. Changing buckets later splits stored history, so it's better done before 3.1 ships.
• IntelliSense shows none of the doc comments (S). The csproj has no GenerateDocumentationFile, so the package ships without its XML docs. Turning it on will also flag undocumented public members (CS1591).

Quick wins (S, additive)

• Configure the storage account once. An account URI plus QueueName is rejected today ("EndpointUri with no queue name"). Accept it, and add queues.UseStorageAccount(connectionStringOrUri) next to UseTokenCredential. That fits how Aspire hands out ConnectionStrings:<name>.
• Processor options from configuration. Add an AddProcessor<T, H>(IConfiguration) overload. Options are stored under a hidden internal name today, so MaxConcurrentMessages can't vary by environment without a code change.
• Clear error when injecting an unregistered client. Injecting IQueueSender with only named clients registered, or with a mistyped key, gives the generic DI error rather than the factory's message listing the registered names. Fix: a KeyedService.AnyKey registration plus an unkeyed fallback, both routing to the factory.
• More context on each message. Add QueueMessage<T>.QueueName (one handler can serve two queues) and IsLastAttempt, so a handler can alert or compensate on its final try.
• Less noisy, better-pitched logs. A failed poll logs the error and stack trace twice (StorageQueue and the processor), once per worker per backoff. Every healthy message logs at Information twice (move both to Debug). A handler failure logs at Error on every retry (use Warning while retries remain, Error on the last attempt).
• Log scope per delivery. BeginScope with queue name, message id and dequeue count in DeliverAsync, so handler logs correlate in sinks that don't use trace ids.
• Time-in-queue metric. Record now minus InsertedOn when delivery starts. It's the best "are we keeping up" signal and costs no storage calls.
• Saturation and cost metrics. A gauge of running handlers, a counter of receive requests by result (empty, message or error; idle polls are billable and invisible today), counters for lease renewals and renewal failures, and a dead-letter counter with a bounded reason (max_dequeue, deserialization, handler). Plus span events for renew, complete, abandon and dead-letter.
• Hung handlers. Renewal keeps a deadlocked handler's message hidden forever, with no signal. Add an oldest-running-handler age gauge, a one-time warning past a threshold, and an optional MaxRenewalDuration, as Service Bus has.
• Send guardrails. A clear error before sending a message over the 64 KiB limit; Base64 and the trace wrapper both make messages bigger, and today the service just returns a bare 413. Add a Delay alias for SendOptions.VisibilityTimeout, and check the delay against the time-to-live and the 7-day limit.
• Wake idle workers on a burst. Each worker backs off on its own, so after an idle spell a burst can wait up to 30 s for sleeping workers. This is the interim fix ahead of the batched receiver below.
• In-memory queues emit the same telemetry as the real queue, so sent and received counts appear in tests.
• Pause, resume and rate limiting. A controller to pause a processor during a downstream outage, plus an optional RateLimiter before each receive.
• Compression. A .UseCompression() serializer wrapper. The wrapper's body64 already carries non-JSON output.
• Docs. Alert recipes (time-in-queue p95, poison depth, failed/received ratio, empty-poll rate, renewal failures), exemplar setup, KEDA notes (the approximate count includes hidden messages), client-side encryption through SpecializedQueueClientOptions, the "ClientOptions": { "MessageEncoding": "Base64" } config binding, and [JsonDerivedType] as a way to put several message types on one queue today.

Bigger features (M/L, additive)

• One receiver feeding the workers (M). A single loop receives up to as many messages as there are idle workers (max 32), starts lease renewal on each immediately, and hands them to workers through a channel. This keeps the design's guarantees (design doc F4: no message waits behind another's handler, and the processor never holds a message no worker is free for). It speeds up the response to bursts, matches Functions' batchSize, and cuts receive calls to about 1/32. The per-message delete stays. Batch handlers (IQueueBatchHandler<T>) become a follow-on.
• Poison-queue replay and inspection (M; needs the wrapper fix). ReplayAsync(max, filter) moves messages back to the source queue. Keep the dead-letter reason, original id and dequeue count in the wrapper, and strip them on replay.
• Exception policy (M). options.DeadLetterOn<TException>() or an OnHandlerError callback returning retry, dead-letter or complete, so a ValidationException doesn't need a try/catch in every handler.
• Handler middleware (M). One pipeline around HandleAsync, which covers idempotency, scopes, rate limits, timing and circuit breaking without separate features. This is the MassTransit/Rebus model.
• Headers and routing by message type (M; needs the wrapper fix). SendOptions.Headers and QueueMessage.Headers (correlation id, message type, content type), plus a routed processor mapping each type to a handler. Today two processors on one queue take each other's messages and dead-letter them. Also routing by CloudEvent type.
• Health checks and queue-depth gauges (M; probably a small separate package for the health-checks dependency). Reachability, last successful receive per processor, and consecutive failures. Opt-in gauges for main and poison queue depth, refreshed at a configurable interval since each refresh is billable. "Poison depth > 0" is the alert most teams want.
• Lambda handlers (M). For example AddProcessor<Order>(async (Order order, OrdersDb db, CancellationToken ct) => ...), which also removes the message.Body! every handler writes today.
• ReadAllAsync<T>() returning IAsyncEnumerable (S–M). It would replace the roughly 30-line backoff loop in the README's "own loop" sample.
• Standard-named metrics as an opt-in (M). messaging.client.sent.messages, messaging.client.consumed.messages, messaging.process.duration and messaging.client.operation.duration on a second meter, leaving the 3.0 names untouched.
• AOT and trimming readiness (M). IsAotCompatible, a UseJsonSerializer(IJsonTypeInfoResolver) overload, and the config-binding source generator. Check first that the Azure SDK is trim-clean.
• Large messages via Blob storage (L, separate package). Bodies over 64 KiB go to Blob storage, with a pointer on the queue. This is the biggest functional gap. It needs the send path's encode step to become async and pluggable, and the download to happen inside delivery after renewal starts. The blob is deleted on completion and kept on dead-letter. Orphaned blobs need a documented lifecycle rule.

Behaviour changes to decide on

• camelCase JSON silently becomes empty objects. The default serializer reads case-sensitively, so messages from JavaScript, Functions or Event Grid produce empty fields, not an Error, and get completed. Making reads case-insensitive is low-risk but still a behaviour change. Switching the default output to camelCase would be breaking (4.0).

Rough edges (verified)

• Calling AddProcessor<T, H> twice for the same client and handler starts two processors sharing merged options, silently doubling the workers.
• ReceiveAsync(visibilityTimeout:) and AbandonAsync(delay) aren't checked before the call, so a bad value becomes a 400 from the service. The processor options and client settings are checked.
• QueueClientSettings is [ExcludeFromCodeCoverage] but now holds all the validation logic.
• docs/v3-design.md's "Nothing escapes" section is out of date. It promises a "set CreateIfNotExists" hint for a missing queue, which doesn't exist, and says a failed dead-letter uses the maximum retry delay, which is not what the code does.

Remaining for 3.1 (moved here from "4.0 candidates" on 2026-10-16: 3.0 shipped the same day with negligible uptake, so these breaking changes go into 3.1)

• Replace the SDK's SendReceipt and QueueProperties in the public interfaces with the library's own records.
• Rename RenewLockAsync to RenewVisibilityAsync.
• Rename QueueClientSettingsBuilder.
• Default to camelCase JSON output.
• Batch handlers (IQueueBatchHandler<T>), the follow-on to the single receiver.
• UseBlobStorageForLargeMessages option for a container created in advance (no create permission needed).

4.0 candidates (breaking) — superseded by the section above

• Replace the SDK's SendReceipt and QueueProperties in the public interfaces with the library's own records, so fakes don't need QueuesModelFactory.
• Rename RenewLockAsync to RenewVisibilityAsync, to match the RenewVisibility option.
• Rename QueueClientSettingsBuilder, which sits awkwardly beside IQueueClientBuilder.
• Default to camelCase JSON (see above).

Considered and skipped

• Outbox: needs database transaction integration; that's Wolverine/MassTransit territory.
• Encryption hooks: the SDK's client-side encryption already works and survives the 3.1 options copy; documenting it is enough.
• Batch send: the service has no batch-send call.
• Scheduled send: SendOptions already delays up to 7 days.
• Multi-queue fan-out: one line of user code.
• Access-policy management: the SDK covers it.
• First-class idempotency store: do it through middleware.
• Priority and sharded queues: niche until the single receiver exists, then cheap.

Suggested order

1. The four "before 3.1" items, then release 3.1.
2. The quick wins and the exception policy: a "production operations" 3.2.
3. The single receiver, poison replay, health checks and middleware.
4. Headers and routing, lambda handlers, AOT.
5. Large messages, as their own package.