fix(device-auth): trust one client-cert header, and its trusted end (audit H7) - #431
Conversation
In Envoy's `x-forwarded-client-cert` convention each trusted proxy *appends* its element, so the element describing the immediate TLS client of the terminating proxy is the last one. Everything before it describes hops further upstream, and the head of the list is whatever the original client chose to send. The parser split on `,` — so it anticipated multiple elements — and then took `.next()`. Where the chain has two hops (or XFCC is in an append-forward mode), a caller can prepend `Cert=<PEM of a victim device>` and be authenticated as that device: the proxy appends its own element, and canopy derives the SPKI from the attacker's. Device certificates are public, so this is a device-auth bypass on every mTLS-authenticated public-server endpoint and on enrollment's `resolve_spki`. It's only safe if the proxy replaces the header outright, which code that parses a list can't assume. Element selection now takes the last element, and the split is quote-aware — Envoy quotes any value containing a separator, and mis-slicing the list is precisely what decides which hop is trusted. A last element with no `Cert=` yields nothing rather than searching backwards, since an earlier element is the untrusted input this is meant to ignore. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
|
Fortunately the envoy ingress path isn't yet live. |
Reading the trusted end of XFCC fixes which element is believed, but not *whether the header should be believed at all*. `x-forwarded-client-cert` was read unconditionally, with no check that the request came through Envoy — and preferred over `mtls-certificate`, so it overrode the header the live nginx ingress actually sets. That matters because `resolve` authenticates via `Device::from_key`: a lookup on the certificate's public key, with no proof of possession. Sound when a proxy terminates mTLS and vouches for the peer, but a public key is not a secret. Envoy isn't deployed and nginx doesn't use XFCC — so nginx has no reason to strip it, and any caller able to set the header could present an enrolled device's certificate and be resolved as that device. Each path now has its own switch: - `CANOPY_DEVICE_AUTH_MTLS_HEADER` (default **on**) — `mtls-certificate` and `ssl-client-cert`, what the live ingress sets. On by default so prod is unchanged. - `CANOPY_DEVICE_AUTH_XFCC` (default **off**) — `x-forwarded-client-cert`. Off until the Envoy path is live, at which point flipping the pair swaps ingress without a code change. An unrecognised value keeps the default rather than flipping a security switch on a typo, and both-on or both-off warns at startup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
The XFCC gate defaults off, which broke the public-server tests that authenticate over `x-forwarded-client-cert` — the test harness stands in for a trusted ingress, and the suite covers both transports. `force_trusted_cert_headers` sets the trusted set explicitly instead of reading the environment, so the harness is deterministic rather than racing a `LazyLock` against whichever test touches it first. Which headers a real deployment trusts stays configuration, unit-tested in `mtls`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
|
Actually instead of two gates, let's have one gate that spells out the header to trust, and default it to the current prod path. That renders the both on both off check unnecessary as the situation becomes impossible. |
Replaces the two booleans with a single `CANOPY_DEVICE_AUTH_CERT_HEADER` naming the header this deployment's ingress sets — `mtls` (default, the live nginx path) or `xfcc`. "Trust both" and "trust neither" stop being states the configuration can express, so the warnings guarding against them go too. It's also no longer a process global. `ClientCertHeader` lives in server state and reaches the extractor through `FromRef`, the idiom `AuthDevice` already uses for the pool and the tailnet directory. That's what lets a test choose, which a `LazyLock` couldn't. The suite now runs on XFCC. Production exercises the nginx header continuously; XFCC is the shape that goes live at the cutover and had no coverage outside unit tests. Testing the new shape means removing the old one later is a delete rather than a rewrite. `device_key_authentication_works_on_ the_nginx_header` keeps end-to-end coverage of the live path (and asserts XFCC is refused there), and goes at the cutover with the `Mtls` variant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
CI caught what a stale base hides: main has added tests using `mtls-certificate` (products/entitlements, billing tags, restore, names) and the suite now runs on XFCC, so they authenticated against the wrong header and 401'd. Merges cleanly and compiles — only running it finds this. Converts the 45 that arrived with main, and one in `device_key_auth.rs` that I over-excluded when protecting the deliberate nginx-path test in that file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
…ot' into claude/pr-370-fix-h7-xfcc-last
Rebasing the collision away rather than fixing it after each landing. This branch flips the suite to XFCC, so every test that arrives on the base using `mtls-certificate` breaks here — and does so only at runtime, since it merges cleanly and compiles. That has now happened twice, and #429 was queued up to make it three. Basing on #429 instead of main puts the two in a defined order and converts its `credentials_during_a_rotation_is_503` here, where the flip lives.
|
Stacked this onto #429 — base is now That retires the note at the bottom of the description. The recurring problem was that this branch flips the integration suite to XFCC, so any test arriving on the base with Order is fixed: #429 merges, GitHub retargets this to main, and it goes in behind it. Generated by Claude Code |
Fixes H7 (high) from the audit in #370, plus the larger hole review turned up underneath it.
1. Reading the wrong end of XFCC (the original H7)
In Envoy's
x-forwarded-client-certconvention each trusted proxy appends its element, so the element describing the immediate TLS client of the terminating proxy is the last one. Everything before it describes hops further upstream — and the head of the list is whatever the original client chose to send.extract_cert_pemsplit on,, clearly anticipating multiple elements, then took.next().xfcc_client_certnow takes the last element'sCert=. Two details worth review:By="spiffe://mesh/a,b"), and a naive split cuts that element in two, making the quoted tail look like the last element. Element slicing is exactly what decides which hop we trust.Cert=yieldsNonerather than searching backwards — an earlier element is precisely the untrusted input this exists to ignore.2. Trusting XFCC at all (found in review)
The bigger problem: XFCC was read unconditionally, with no check that the request came through Envoy, and preferred over
mtls-certificate. Envoy isn't deployed and nginx doesn't use XFCC — so nginx has no reason to strip it, and a client could set it themselves and override the header nginx actually verifies.That's exploitable rather than untidy because
resolveauthenticates viaDevice::from_key: a lookup on the certificate's public key, with no proof of possession. Sound when a proxy terminates mTLS and vouches for the peer; a public key is not a secret.So there's now one setting naming the header to trust:
One value, one header — "trust both" and "trust neither" aren't states the configuration can express. An unrecognised value keeps the live path rather than guessing.
It's not a process global:
ClientCertHeaderlives in server state and reaches the extractor throughFromRef, the idiomAuthDevicealready uses for the pool and the tailnet directory. Configuration belongs in state, and it's also what lets a test choose.3. The suite runs on XFCC
Production exercises the nginx header continuously; XFCC is the shape that goes live at the cutover and had no coverage outside unit tests. So the integration suite runs on XFCC — testing the new shape means removing the old one later is a delete, not a rewrite.
device_key_authentication_works_on_the_nginx_headerkeeps end-to-end coverage of the live path (and asserts XFCC is refused there). At the cutover, it goes along withClientCertHeader::Mtlsand themtls-certificatebranch.Tests
Twelve unit tests in
mtls.rs: the original six parser cases (includingchained_elements_use_the_last_cert, the attack shape, and the quoted-value splitting), plus selection in both directions —xfcc_is_ignored_on_the_nginx_path,xfcc_cannot_override_the_nginx_header,the_nginx_headers_are_ignored_on_the_envoy_path,xfcc_is_honoured_on_the_envoy_path— and env parsing.Note for whoever merges this
This branch is sensitive to landing late. Tests that arrive on
mainusingmtls-certificatefail here once merged, because the suite authenticates over XFCC — and they merge cleanly and compile, so only running them finds it. That's already happened twice (once with main's products/entitlements and billing-tag tests). #429 adds another (credentials_during_a_rotation_is_503), so if that lands first this needs one more conversion pass. Worth merging sooner rather than later.