perf(rust): coalesce intercepted HTTP response chunks

[mohamedmansour]

Problem

When ClientOptions.request_handler intercepts HTTP traffic, each small response fragment waits for a separate runtime acknowledgement. Bursty responses generate too many sequential RPCs. The default inference path is unaffected.

Solution

Read ahead while the current write awaits acknowledgement, combining available HTTP bytes into chunks up to 32 KiB. Flush without waiting to fill a chunk, preserving byte order, cancellation, and error handling. WebSocket messages remain unchanged.

Two reusable buffers cap raw forwarding storage at 64 KiB per exchange, excluding the current source frame, source internals, and RPC serialization. No new tasks, dependencies, or parallel data RPCs.

Performance

Median results through the real SDK and local HTTP, comparing 0dd9d43 with e5f719f:

Workload	Before	After	Data RPCs
1 MiB burst, 1 KiB fragments	53.486 ms	5.794 ms (9.2x faster)	1,031 → 33
1 MiB burst, 32 KiB fragments	12.483 ms	11.392 ms	39 → 33
Sparse 64 KiB, 1 KiB every 10 ms	806.473 ms	800.563 ms	64 → 64
1 MiB/1 KiB with artificial 2 ms ACK delay	4,180.140 ms	159.508 ms	1,031 → 33

For the 1 KiB burst, allocation operations fell 61,189 → 5,295. Tradeoffs: first-byte latency 1.040 → 1.121 ms and peak live heap 2.27 → 2.37 MB. Heap/allocation measurements include SDK and synthetic peers, not RSS.

Five alternating pairs on Apple M4 Pro, macOS 26.6.2, Rust 1.94.0 release. Timing binaries were uninstrumented; allocations measured separately. Sparse streaming is essentially unchanged, and large-fragment timing ranges overlap. Shared-host results are workload-specific, not model-generation speedups; artificial ACK timers exceeded the requested 2 ms.

[Copilot]

🟡 Changes recommended

Fix the compile-blocking response-body ownership error before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Improves Rust HTTP response forwarding by coalescing small chunks with bounded read-ahead.

Changes:

• Adds bounded response buffering.
• Integrates coalesced forwarding with cancellation handling.
• Adds forwarding and regression-test coverage.

File summaries

File	Summary
rust/tests/http_response_forwarding_test.rs	Tests buffering, failures, cancellation, and byte preservation.
rust/src/copilot_request_handler/http_response_reader.rs	Implements bounded response buffering and error handling.
rust/src/copilot_request_handler.rs	Integrates coalesced forwarding; contains a critical ownership error that prevents compilation.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mohamedmansour]

@stephentoub can you please look at this perf improvement please, found by profiling