Skip to content

A route that fails to rebuild no longer keeps its old client-certificate policy - #65

Merged
kriszyp merged 1 commit into
mainfrom
fix/route-client-auth-carry-forward
Oct 15, 2026
Merged

kriszyp merged 1 commit into
mainfrom
fix/route-client-auth-carry-forward

Conversation

@Devin-Holland

@Devin-Holland Devin-Holland commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

When a hot reload couldn't build a route, symphony kept serving the whole previous route, including its client verifier. A route updated to require client certificates kept admitting clients without one until a later reload succeeded. Reproduced on main through symphony-server (#62).

🤔 Is a policy-aware carry-forward the right size for this?

It closes the hole for every consumer (napi and symphony-server) and keeps mid-rotation cert/key failures serving. host-manager#254 doesn't depend on it. The "trust nothing" route from #62 item 3 is proposed on the issue, not built here.

💡 Solution

⚖️ Alternatives

  • Drop on every failed rebuild — breaks cert/key rotation, which writes two files non-atomically; that window is why carry-forward exists.
  • Keep the policy on Route — every connection clones Route into its task; measured +32 B per connection (3384 → 3416 B).
  • Fix only in server.ts — embedded napi consumers keep the hole, and TS can't predict which rustls builds fail.
  • Reuse the old cert under the new verifier — serves a patched route instead of refusing, and needs new session-cache identity.

🤔 Should an unreadable CA file drop the route?

It does: the requested policy is unknown, and keeping the route could keep trusting a removed CA. A CA file caught mid-write leaves the route offline until the next reconcile. Rename-based rotation (what host-manager does) never hits this window.

🔧 Changes

🤔 What about a listener-level CA?

Out of scope here. An unreadable or unbuildable listener mtls CA still leaves the old proxy, and its old policy, running. Failing closed there takes every route on the port down.

✅ Verification

Live probe output (before → after)
A2 reloaded to required mtls with empty CA, certless client     CONNECTED → REFUSED (ECONNRESET)
B2 key file missing, policy unchanged, client with cert          CONNECTED → CONNECTED
B2 key file missing, policy unchanged, certless client           REFUSED   → REFUSED (certificate required)
C2 CA file missing, client with cert                             CONNECTED → REFUSED (ECONNRESET)
D2 key file missing, no mtls, certless client                    CONNECTED → CONNECTED

Fixes #62

🤖 Generated by Claude Opus 5.5 via Claude Code (devain); posted via @Devin-Holland.

🤖 Generated with Claude Code

Release target: main — symphony ships from main; it has no release branches

Related PRs: none found
Complexity: complicated

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-muse; blocked=domain(auth); declined=cursor-grok,cursor-kimi,muse; rounds=2; full=1 @ 5c33811

Review-Attention: read ~6m (raised: degraded review) @ 5c33811

A hot reload that can't build a route carried the previous route forward
whole, client verifier included. A route republished to require client
certificates kept admitting certless clients, and one whose CA bundle
changed kept trusting the old CAs, until a later reload built (#62).

build_route_table now carries a failing route forward only while the new
spec asks for the client-certificate policy the last-good route enforces:
passthrough, no client cert, or the same effective CA bundle and
requireClientCert. Otherwise the SNI is tombstoned so it can't fall
through to a wildcard. Mid-rotation cert/key failures under an unchanged
policy still keep last-good. The policy lives on RouteTable, not Route,
so the per-connection task doesn't grow.

One spec decides each configured SNI even when it is rejected: only the
last exact duplicate is built, and the first wildcard reserves its suffix
before it is validated, so a duplicate can't serve a tombstoned SNI under
another policy.

symphony-server's resolveRoute keeps the requested mtls when a cert file
can't be read, and substitutes an unbuildable empty CA bundle when the CA
file itself can't be read, so an unknown policy on a terminating route is
dropped, never kept.

Fixes #62

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that when a route fails to rebuild during a hot-swap, its last-good version is only carried forward if the requested client-certificate policy remains unchanged. If the policy has changed, the route is dropped (tombstoned) to prevent an old verifier from outliving its replacement. This is implemented by introducing a ClientAuth enum and tracking client-certificate policies in the RouteTable. Additionally, the TypeScript server wrapper is updated to handle client CA resolution failures by returning an unresolvable mTLS configuration, and comprehensive integration and unit tests are added to verify these behaviors. I have no further feedback to provide as there are no review comments.

@kriszyp
kriszyp merged commit 0ccd71b into main Oct 15, 2026
11 checks passed
kriszyp added a commit that referenced this pull request Oct 15, 2026
Sync package + platform + crate versions to 1.4.0.

Since 1.3.0, in the order a consumer meets them:

Less memory per connection (#61). Each connection's task future shrank
from 42.8 KB to 3.3 KB: the HTTP/1 header-rewriting path is boxed, the
copy future is pinned once, and the accepted TLS stream is passed by
reference instead of being copied into every state. Parked connections
measured 66.8 KB -> 27.2 KB each (-59%). A test fails if the future grows
past 4 KiB.

Memory is returned after connection peaks on Linux (#66). The addon's Rust
allocations now use jemalloc on Linux (glibc and musl, x64 and arm64);
macOS keeps the system allocator, and Node's own allocations are
unchanged. Under glibc, symphony kept ~100% of its peak memory after the
connection count fell, and the peak ratcheted up across cycles. With
jemalloc, troughs hold ~28% of peak and stay flat across cycles, at +1.9%
memory per parked connection. Build with --no-default-features for the
system allocator. arm64 builds use 64 KiB jemalloc pages so one binary
runs on 4, 16 and 64 KiB-page kernels.

A route that fails to rebuild no longer keeps its old client-certificate
policy (#65). A hot reload now carries a failing route forward only when
the new spec asks for the same client-cert policy; otherwise the SNI is
tombstoned rather than admitting clients under the old verifier.

TLS session resumption works for resolved suspended routes, and expired
IP-affinity entries are reclaimed (#57).

forwardFingerprint accepts a list, so a route can forward both JA3 and
JA4 (#49).

Overlapping upgrades keep status.json pointing at the new process: an
outgoing process no longer repoints or unlinks it, and a process that
yields to a newer peer keeps watching.

Benchmarking: `npm run benchmark:memory-oop` measures symphony's own
memory out of process under parked, active, stalled-reader and
churn-then-drop load (#60). SYMPHONY_TLS_BUFFER_LIMIT and
SYMPHONY_MALLOC_TRIM_INTERVAL_MS are benchmark hooks, off by default and
not a supported configuration surface.

Known open measurements on #66: CPU for 4 KiB messages is still noisy
(256 B messages show no difference), and trough retention with 64 KiB
pages has been measured on x86 only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A route that fails to rebuild on reload keeps its old client-certificate policy

2 participants