Feature/h3 experimental engine - #888
Open
MDA2AV wants to merge 16 commits into
Open
Conversation
GenHTTP models HTTP/3 already - HttpProtocol.Http3 exists and endpoints carry
an EnableQuic flag - but only the Kestrel engine acts on it. On the Internal
and Ioxide engines the flag does nothing. This adds an engine that serves it.
A separate engine rather than a mode of the Internal one, because the Internal
engine's endpoint model is TCP by construction: the base EndPoint creates a
SocketType.Stream socket, accepts Sockets, and hands each connection over as a
single Stream. QUIC has none of those. A QuicEndPoint deriving from it would
override everything but the constructor.
Glyph3 does HTTP/3, System.Net.Quic does QUIC, and H3Connection is the bridge.
Three things it has to reconcile:
- Glyph3 is one state machine and MsQuic reads streams concurrently, so
everything funnels through one channel and one consumer.
- GenHTTP handlers do real I/O and must not run on that consumer, so they are
started with Task.Run and their responses come back through a
SynchronizationContext that posts to the pump. The context is installed only
around calls into Glyph3: leaving it set made the pump post its own
continuation to the queue only it drains, and wait for itself.
- OpenUniStream is synchronous and OpenOutboundStreamAsync is not, so the
unidirectional streams are opened before the connection starts.
Two HTTP/1.1 assumptions had to be translated. HTTP/3 has no Host header, so
:authority is surfaced as one (RFC 9114 4.3.1), without which GenHTTP's
compliance check rejects every request. And :path carries the query string, so
it is stripped before routing sees it.
AltSvc.To(port) advertises the endpoint from a TCP server, which is the only
way a browser ever reaches it.
Verified against .NET's own HTTP/3 client:
GET /hello -> HTTP/3.0 200 Hello from GenHTTP over HTTP/3!
GET /json -> HTTP/3.0 200 {"ok":true}
GET /missing -> HTTP/3.0 404
GET /hello?a=1&b=2 -> HTTP/3.0 200 (query no longer breaks routing)
GET /hello -> HTTP/3.0 200 (connection reused)
Both engines bind 8443, since TCP and UDP are different sockets, and the HTTP/1.1 host advertises the HTTP/3 one with Alt-Svc. That advertisement is the only way a browser ever reaches HTTP/3, and nothing verifies the port in it matches what the other host bound. Verified from one process: TCP :8443 HTTP/1.1 200 OK, alt-svc: h3=":8443"; ma=86400 UDP :8443 HTTP/3.0 200, same handler, same body
Two bugs a load test found, both in the MsQuic bridge. A finished request stream was never disposed, so its stream credit was never returned to the peer. Every connection stalled after MaxInboundBidirectionalStreams requests: exactly 800 completed and 256 failed across 8 connections, run after run, which is a limit rather than a measurement. Disposing it then had to happen off the writer loop. Awaiting DisposeAsync there serialised every other request on the connection behind one stream's teardown: 612 req/s with the await, 235,704 without. Unidirectional streams are exempt: those are the control and QPACK streams and live as long as the connection. Also renames the playground's port constant, which said H3Port while the HTTP/1.1 host bound it too. Both hosts binding 8443 is correct - TCP and UDP are different sockets - but the name read like a bug.
Dispatch started every request with Task.Run and waited for the response to come back through the pump's SynchronizationContext, so each one paid two thread transitions. Under load that also starved the pool: requests failed outright, around 100 per ten-second run. Starting the handler inline costs nothing when the chain completes synchronously, which is the common case, and Glyph3 submits without ever leaving the pump. A handler that genuinely suspends still returns a Task at its first incomplete await and resumes through PumpContext, so the pump is only ever held for the synchronous prefix - the same contract any event loop offers, and the same one ioxide's reactor has. 31,144 -> 218,409 req/s, and failures went from ~100 per run to zero.
A console line per request is enough to dominate a throughput measurement, and it was the first thing that had to be switched off to benchmark the engine. The comment says how to get it back.
Glyph3 calls OpenUniStream exactly once, for its control stream. HTTP/3 also defines QPACK encoder and decoder streams, but Glyph3 advertises a dynamic table capacity of 0, so there is nothing to insert and nothing to acknowledge. The other two were opened, never written, and held until the connection ended. Named the count so the reason is written down rather than rediscovered.
…layground Host.Create takes a QPACK dynamic-table capacity now, plumbed through to Glyph3. Zero stays the default and switches the mechanism off entirely. A nonzero value also needs two more unidirectional streams - QPACK decoder, and encoder once the client advertises a table of its own - so the bridge opens three rather than one. Requires Glyph3 0.3.0, which is where both directions of the table landed. The playground now serves a wwwroot over both protocols instead of one string: an index, three images and a script, deliberately several subresources so a browser makes repeated requests on one connection. That is the traffic a dynamic table exists to compress, and no client tested so far uses one - curl 8.21 advertises a capacity of 0, .NET's HTTP/3 client is static-table only, and h2o inserts nothing. Browsers are the remaining candidate, which is what this page is for: it reports the protocol it arrived over. A browser will not speak HTTP/3 to an untrusted certificate, so startup prints the certificate's SPKI hash for Chrome's --ignore-certificate-errors-spki-list, alongside --origin-to-force-quic-on. Verified over both protocols with curl: index, image and script all 200.
Host.Create(QpackCapacity) does not say what the number is; the named form does, and the playground's other knobs are all self-describing.
…er got it Connection options are case-insensitive tokens (RFC 9110 7.6.1), but the value was compared with SequenceEqual against "Keep-Alive". Browsers send "keep-alive" in lower case, so the match failed and every browser request was answered with Connection: close - a new TCP connection, and a new TLS handshake, per request. curl hid it by sending no Connection header at all, which falls through to the protocol default and keeps the connection alive.
Glyph3 now exposes the peer's advertised QPACK capacity and the per-direction insert counts, so a connection can be asked whether the dynamic table was used rather than guessed at. Logged at Debug when a connection closes, separating "the peer advertised 0" from "SETTINGS never arrived" - both leave the counters at 0 while meaning different things. Measured with it: Chrome advertises 65536 B and decodes our dynamic references, so the outbound path pays off. curl, h3x and .NET's client all advertise 0, which makes the table inert - and nothing tested inserts into our decode table at all. The playground binds IPv6Any rather than Loopback. A browser resolves "localhost" itself and prefers ::1, so an IPv4-only listener never sees a packet from one; TCP hides this by falling back to IPv4, QUIC does not, and it surfaces as ERR_QUIC_PROTOCOL_ERROR rather than as too narrow a bind. It also takes a certificate through PLAYGROUND_CERT, since a browser needs one from a CA it trusts before it will speak HTTP/3 at all.
Headers were collected into a list and then copied item by item into the response's own list, so every response allocated a second list and its backing array for nothing. They are now written straight into the response. Lowercasing allocated as well. HTTP/3 requires lowercase field names, and the names GenHTTP emits - Server, Date, Content-Type - all arrive with capitals, so each one copied itself into a fresh array on every response. Known names now resolve to a shared pre-lowercased array, leaving the allocation for names not in the table. Worth about 3% of allocation on a small-response benchmark. It does not move throughput, which is bounded elsewhere: the engine allocates less per second than Kestrel does while serving fewer requests, so this path is not GC-bound.
Glyph3 now encodes response headers against the QPACK static table rather than using it for :status alone, so a field whose name it knows is never written out: name and value both in the table cost one byte, a known name costs an index plus the literal value. That makes the lowercasing here redundant twice over. Names it resolves are matched case-insensitively and never reach the wire, and names it has to write out it lowercases itself. Converting them here only duplicated the work and allocated an array per header to do it, so the table of pre-lowercased names and the function that used it are both gone. Bytes per response, not throughput: a header block carrying server, date, content-type, cache-control, etag, vary and accept-ranges drops from 172 to 67. The benchmark here is a single-header response over loopback, which is not bandwidth-bound and measures no difference at all.
The writer loop awaited each write before starting the next, so MsQuic only ever had one stream's data pending and could not fill a datagram from several streams at once. Every response became its own datagram. Writes are now issued for everything drained from the queue and awaited together, so a batch of responses is visible to MsQuic simultaneously. 272-286k to 291-310k req/s, and CPU for the same work falls from 14.7 cores to 10.6. Ordering within a stream still holds: a second write to a stream already in flight drains first, since QuicStream is not safe for concurrent writes. Buffers go back to the pool and finished streams are released only after their write completes - releasing early strands the stream's credit and the peer stalls after MaxInboundBidirectionalStreams requests. Handlers stay inline on the pump deliberately. Dispatching them with Task.Run instead collapses this to 75k req/s and leaves streams unanswered, because the pump then waits on a pool the offloaded work is competing for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First idea:
Both h1 an h3 servers, can't really merge them into one because internal engine is TCP/Socket bounded.
h3 adds a alt-svc advertisement on the h1 server.