perf(query): serialize compatible responses without per-row Value maps

[VibeSWE]

Query responses currently encode Arrow batches, parse the JSON into per-row Value maps, then serialize those maps again for HTTP. This change writes compatible responses directly to bytes, removing those intermediate maps and copies. Both ordinary JSON and NDJSON use the new path; the existing public QueryResponse::to_json() remains unchanged.

Flat rows use Arrow scalar encoders with serde conversion where needed to preserve floating-point and timestamp formatting. Schema guards retain the legacy path for incompatible cases. Missing fields, field order, null handling, escaping, error responses and NDJSON framing retain their existing behavior. Only src/response.rs and src/handlers/http/query.rs change; dependencies, allocator, ingestion and query execution are unchanged.

Validation

Tested head: 56790e6668b58295f005cc02eabdd069f3c30446, against main 9747e2ba7a6dc18e2965b24fc84dd0632dadd87e.

• Exact-head CI passed Rust tests, clippy, UI build and standalone/distributed integration smoke checks (eight successful checks; two skipped).
• Two ABBA blocks per campaign, eight fresh lifetimes each: flat JSON median latency fell 56–58%; direct JSON and both NDJSON options fell 60–64% on 10,000-row responses, with lower request CPU and sampled query RSS.
• The output-path campaign independently verified 2,592 response receipts and 256 compatibility cases. The 64 inherited NDJSON null-omission expected failures remain unchanged.
• A same-executable component comparison of the production serializers matched exact retained response bytes. Count/sum, grouped, and single-row grouped fixtures used 35%, 63%, and 45% less thread CPU, respectively, with fewer allocations and substantially lower peak live requested bytes.
• A controlled idle A/B/B/A check with independent binary copies and zero initial executable-cache residency did not reproduce the earlier extra mapped-memory observation: mean executable RSS was 46.543 MiB on main vs 46.428 MiB here.
• Independent final source/API/resource review found no concrete merge blocker on the tested paths. No small-result fallback or additional code change was warranted.

Evidence limits

Native server measurements used 4 CPU / 4 GiB diagnostic scopes on a shared host. They do not qualify maximum deployment capacity at 12 GiB, sustained ingestion throughput, or every query workload. The component fixtures use inferred Arrow types and one batch; allocation bytes are not RSS.

Earlier whole-request small-query CPU means and ingest RSS point estimates were adverse, so the original all-workload point gate remains recorded as NOT MET in the frozen campaign. Subsequent controlled memory and serializer experiments weaken attribution of those observations to this change; they do not identify every CPU burst or prove universal parity. The merge assessment rests on the combined evidence and source review, not on rewriting that historical verdict. Shared-library cache and host variation remain limitations. Future serde_json feature changes must continue to pass the exact-byte differential tests.

Graphs, raw-result summaries, regressions and follow-up diagnoses are in the Mayfly benchmark report. Local evidence is retained under pr-qualification-20261019/{pr49-comparison-blocks,pr49-missing-output-paths,pr49-controlled-residency,pr49-small-serializer-isolation-v2,pr49-final-merge-review} in the existing experiment archive. Earlier failed attempts and historical overlay measurements remain preserved separately.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}