Cut parked per-connection memory 59% by shrinking each connection's task

[kriszyp]

⊙ Problem

Each accepted TLS connection is one tokio task, and the task's future stays allocated for the connection's whole life. On 1.3.x that future is 42.8 KB, so 100k idle connections spend 4.3 GB on it. Most of it is state that most connections never use: the HTTP/1 header rewriter (17.4 KB), a second copy of the copy future, and four dead copies of the 1.2 KB TLS stream.

🤔 Is this the right fix to make now?

Yes, as specified. It is the largest per-connection cost that #60's benchmark found. No config, default or behavior changes.

💡 Solution

• The per-connection task drops from 43,016 B (two allocations) to one 3,456 B allocation. Parked RSS per connection drops 59%, from 66.8 KB to 27.2 KB, at 30k connections.
• Routes that rewrite HTTP/1 headers pay one extra heap allocation per connection, for the rewriter. Other routes pay nothing extra.
• A unit test fails if handle's future grows past 4 KiB.

⚖️ Alternatives

• Only Add an out-of-process memory benchmark for active, stalled and churning connections #60's patch (box the rewriter, pin copy once) — 8,192 B. It leaves four dead copies of the TLS stream.
• Move the TLS handshake into proxy_via_tls — measured 3,824 B, worse than passing the stream by reference, with more code moved.
• Shrink the rewriter's buffers instead of boxing it — changes four framing functions in http_proxy.rs and saves nothing on header-rewriting connections.

🤔 Is 4 KiB the right budget?

It leaves about 700 B over today's 3,376 B (cargo test profile). Passing the TLS stream by value again (+1.2 KiB) or inlining the rewriter (+17 KiB) fails it. A small future stored twice still fits. It guards handle only, not the short-lived HTTP :80 task. Raise it deliberately if a dependency bump trips it.

🔧 Changes

• src/proxy_conn.rs
  • forward awaits proxy_http1_rewriting through Box::pin, so its 17 KiB lives on the heap only on header-rewriting routes.
  • copy is pinned where it is created, and the half-close select! polls that pinned future instead of moving and re-pinning it.
  • handle matches the handshake result by reference and proxy_via_tls takes &mut TlsStream, so the stream has one slot in the future. Look hardest here: the client stream now drops in handle, right after proxy_via_tls returns. That is after the upstream, in the same poll, before any metric is recorded.
  • the_per_connection_future_stays_within_budget reads the future's size from its type, so it builds no socket, runtime or ConnContext.
• src/listener.rs, src/http_listener.rs: spawn handle(..) and handle_http(..) directly. The async move wrapper held their arguments a second time.
• __test__/liveness.spec.ts: two cases for the half-close bound on a TLS-terminated route that rewrites headers, with the bound on and off. Until now the bound was tested only on passthrough.
• CLAUDE.md: a gotcha entry on keeping the per-connection future small, naming the guard and how to find what grew.

✅ Verification

• Guard test fails on origin/main (42,872 B) and on Add an out-of-process memory benchmark for active, stalled and churning connections #60's patch alone (8,256 B), and passes here (3,376 B).
• cargo test: 154 pass. npm test: 155 pass, including liveness.spec.ts and copy-buffers.spec.ts. The two new liveness cases also pass on main's addon. Clippy and cargo fmt --check are clean.
• Add an out-of-process memory benchmark for active, stalled and churning connections #60's out-of-process harness, 30k connections, glibc, 3 repeats per arm. Parked RSS per connection: main 66,788 B, Add an out-of-process memory benchmark for active, stalled and churning connections #60's patch 32,211 B, this PR 27,226 B. The two drops (34,577 B and 4,985 B) match the type-layout changes (34,568 B and 4,992 B) within 10 B. Active and slow-reader memory is unchanged.

Benchmark: 30k connections, 3 repeats

Parked marginal is the RSS slope between 15k and 30k parked connections. Cells: mean (min–max) over 3 runs, bytes per connection.

arm	parked total	parked marginal	active Δ over parked	slow Δ per stalled conn	delivered %	symphony CPU % (active)
main	68,933 (68,930–68,935)	66,788 (66,783–66,796)	10 (8–14)	74,021 (73,971–74,086)	99.7	121 (117–129)
#60's patch	34,300 (34,292–34,306)	32,211 (32,187–32,237)	7 (5–8)	74,021 (73,993–74,042)	99.8	128 (124–129)
this PR	29,316 (29,310–29,325)	27,226 (27,222–27,229)	9 (3–17)	73,927 (73,855–74,045)	99.8	112 (100–123)

N=30000, 3 client procs, 4 UDS upstreams (ipAffinity true), PROXY v1, default 8 KiB copy buffers, msg 256 B at 1/s/conn, slow fraction 0.1. Each arm is a release .node built from its commit with the same toolchain, swapped into one checkout's dist/ via --addon-arm; arms run in shuffled order per repeat, under bench-run (CPU capped at 2 GHz). Run record: ~/dev/tmp/bench-runs/symphony-connfuture-30k-20261013-210938/meta.txt on the bench host. The harness is not committed here; it lands with #60.

Future sizes, and the other per-connection futures

Release build, RUSTC_BOOTSTRAP=1 RUSTFLAGS=-Zprint-type-sizes cargo build --release --lib, rustc 1.94.1, tokio 1.53.3, rustls 0.23.45 (resolved locally; Cargo.lock is not tracked).

future	origin/main	#60's patch	this PR
task spawned per TLS connection	128 B cell + 42,888 B boxed future	8,448 B cell	3,456 B cell
proxy_conn::handle	42,808	8,192	3,344
proxy_conn::proxy_via_tls	39,664	5,048	1,416
proxy_conn::proxy_raw (passthrough)	35,992	1,376	1,376
proxy_conn::forward<..>	35,528	912	912
http_proxy::proxy_http1_rewriting<..>	17,408 inline	17,408 boxed	17,408 boxed
task spawned per HTTP :80 connection	4,864 B cell	4,864 B cell	4,736 B cell
Timeout<Accept<TcpStream>> (TLS handshake)	1,328	1,328	1,328
sni::peek	440	440	440

tokio's spawn boxes any future over 16 KiB in release builds, which is why main's task is two allocations.

Test profile (cargo test), handle: main 42,872; box only 8,712; pin only 25,336; #60's patch 8,256; this PR 3,376.

• h2 dispatch has no future of its own. handle picks destination_h2 and runs the same forward.
• TLS handshake: its 1,328 B sits in a different suspend state from proxying, so it overlaps and adds nothing.
• Passthrough and TLS now cost the same. handle's handshake, TLS and passthrough states are each 3.3 KB. Going lower means trimming locals they all hold (route, peek info, source forwarding), which is not mechanical.
• HTTP :80 listener: 4.6 KB, nearly all a 4 KiB read buffer in read_header_block. These connections live for one header read and one response, so it is not a parked cost.
• Header-rewriting connections still allocate the 17.4 KB rewriter: each of its two pumps holds copy_exact's 8 KiB array inline, plus read_header_block's 4 KiB array in another state. Follow-up: heap buffers sized on demand.

_{🤖 Generated by Claude (Opus 5.5); posted via @kriszyp.}

🤖 Generated with Claude Code

https://claude.ai/code/session_01DxQT3d8CgJNgAjYp8Pc5ww

Related PRs: #60 overlaps (source of the first commit; both edit src/proxy_conn.rs)
Complexity: medium

Origin — the dispatch brief this PR was written from

Shrink symphony's per-connection task future, the 'future-size patch' described in draft PR #60 (see the collapsed section 'The future-size patch (the futfix arm; not in this PR)' in its body). In src/proxy_conn.rs, handle's future is ~42.8 KB per connection on 1.3.x; 17.9 KB of that is new in 1.3.0. Two changes take it to ~8.2 KB: (1) Box::pin the crate::http_proxy::proxy_http1_rewriting(...) future so its large state machine lives on the heap only on routes that rewrite HTTP headers, instead of inline in every connection's future; (2) pin the copy future once with std::pin::pin! where it is created, and drop the later tokio::pin!(copy) inside body, so the future is not stored twice. That measured -34.6 KB resident per parked connection, with no behavior change. This is the dominant per-connection cost on fan-out nodes with 100k+ long-lived TLS connections.

Deliverables:

1. Apply the patch on a branch off origin/main (not off the Add an out-of-process memory benchmark for active, stalled and churning connections #60 branch).
2. Add a regression guard that fails if the per-connection future grows back. For example, a unit test that constructs the handle future (or the closest public-in-crate entry point spawned per accepted connection) without polling it, and asserts std::mem::size_of_val(&fut) is under a budget (e.g. 12 KiB), with a comment saying why the budget exists. If constructing it in a test needs heavy scaffolding, a smaller test around the copy/body composition is fine, as long as it would fail on today's main. Prove the guard fails without the fix: run it against main's proxy_conn.rs and watch it go red.
3. Survey the other per-connection futures (the h2 dispatch path, the HTTP listener path, the TLS accept/handshake path) with -Zprint-type-sizes on nightly, or with size_of_val probes. Report the top few sizes in the PR body. Fix any with the same pattern (an inline large rarely-used branch, or a future stored twice) only if the fix is equally mechanical; otherwise list them as follow-ups.
4. Measure: run the out-of-process harness from the Add an out-of-process memory benchmark for active, stalled and churning connections #60 branch (npm run benchmark:memory-oop, parked phase at least 30k connections, glibc default) against this branch vs main, and put before/after B/conn in the PR body. Do not commit the harness here; it lands with Add an out-of-process memory benchmark for active, stalled and churning connections #60.

Acceptance

• Patch applied; cargo test and npm test pass; CI green.
• A regression guard test that fails on origin/main and passes on the branch (show both in the PR body).
• PR body: before/after per-connection RSS from the Add an out-of-process memory benchmark for active, stalled and churning connections #60 harness (parked, 30k+ conns), future sizes before/after, and the survey of other per-connection futures.
• No changes to defaults, config schema, or the reaping/half-close semantics. No version bump or release.
• Draft PR.

_{Dispatch: task symphony-shrink-connection-future · queued by kris-session · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse,muse; rounds=2; full=1 @ 312469f}

_{Review-Attention: read ~3m (decisions: box-rewriter-vs-slim-framing, layout-budget-as-guard, guard-covers-tls-only) @ 312469f}

[gemini-code-assist]

Code Review

This pull request optimizes the memory footprint of per-connection futures to ensure they stay within a 4 KiB budget. Key changes include spawning the handle and handle_http futures directly instead of wrapping them in async move blocks, passing the TLS stream by reference rather than by value, pinning the copy future in place, and boxing the large HTTP/1 header rewriter future. Additionally, a unit test was added to assert the future size budget, and integration tests were introduced to verify half-closed connection reclamation on HTTP-rewriting routes. No review comments were provided, so there is no additional feedback to address.