Skip to content

fix(device-auth): trust one client-cert header, and its trusted end (audit H7) - #431

Merged
passcod merged 9 commits into
mainfrom
claude/pr-370-fix-h7-xfcc-last
Aug 1, 2026
Merged

fix(device-auth): trust one client-cert header, and its trusted end (audit H7)#431
passcod merged 9 commits into
mainfrom
claude/pr-370-fix-h7-xfcc-last

Conversation

@passcod

@passcod passcod commented Aug 1, 2026

Copy link
Copy Markdown
Member

Fixes H7 (high) from the audit in #370, plus the larger hole review turned up underneath it.

1. Reading the wrong end of XFCC (the original H7)

In Envoy's x-forwarded-client-cert convention each trusted proxy appends its element, so the element describing the immediate TLS client of the terminating proxy is the last one. Everything before it describes hops further upstream — and the head of the list is whatever the original client chose to send. extract_cert_pem split on ,, clearly anticipating multiple elements, then took .next().

xfcc_client_cert now takes the last element's Cert=. Two details worth review:

  • The split is quote-aware. Envoy quotes any value containing a separator (By="spiffe://mesh/a,b"), and a naive split cuts that element in two, making the quoted tail look like the last element. Element slicing is exactly what decides which hop we trust.
  • A last element with no Cert= yields None rather than searching backwards — an earlier element is precisely the untrusted input this exists to ignore.

2. Trusting XFCC at all (found in review)

The bigger problem: XFCC was read unconditionally, with no check that the request came through Envoy, and preferred over mtls-certificate. Envoy isn't deployed and nginx doesn't use XFCC — so nginx has no reason to strip it, and a client could set it themselves and override the header nginx actually verifies.

That's exploitable rather than untidy because resolve authenticates via Device::from_key: a lookup on the certificate's public key, with no proof of possession. Sound when a proxy terminates mTLS and vouches for the peer; a public key is not a secret.

So there's now one setting naming the header to trust:

CANOPY_DEVICE_AUTH_CERT_HEADER = mtls   # default: mtls-certificate / ssl-client-cert (live nginx path)
                               | xfcc   # x-forwarded-client-cert (Envoy, post-cutover)

One value, one header — "trust both" and "trust neither" aren't states the configuration can express. An unrecognised value keeps the live path rather than guessing.

It's not a process global: ClientCertHeader lives in server state and reaches the extractor through FromRef, the idiom AuthDevice already uses for the pool and the tailnet directory. Configuration belongs in state, and it's also what lets a test choose.

3. The suite runs on XFCC

Production exercises the nginx header continuously; XFCC is the shape that goes live at the cutover and had no coverage outside unit tests. So the integration suite runs on XFCC — testing the new shape means removing the old one later is a delete, not a rewrite.

device_key_authentication_works_on_the_nginx_header keeps end-to-end coverage of the live path (and asserts XFCC is refused there). At the cutover, it goes along with ClientCertHeader::Mtls and the mtls-certificate branch.

Tests

Twelve unit tests in mtls.rs: the original six parser cases (including chained_elements_use_the_last_cert, the attack shape, and the quoted-value splitting), plus selection in both directions — xfcc_is_ignored_on_the_nginx_path, xfcc_cannot_override_the_nginx_header, the_nginx_headers_are_ignored_on_the_envoy_path, xfcc_is_honoured_on_the_envoy_path — and env parsing.

Note for whoever merges this

This branch is sensitive to landing late. Tests that arrive on main using mtls-certificate fail here once merged, because the suite authenticates over XFCC — and they merge cleanly and compile, so only running them finds it. That's already happened twice (once with main's products/entitlements and billing-tag tests). #429 adds another (credentials_during_a_rotation_is_503), so if that lands first this needs one more conversion pass. Worth merging sooner rather than later.

In Envoy's `x-forwarded-client-cert` convention each trusted proxy *appends*
its element, so the element describing the immediate TLS client of the
terminating proxy is the last one. Everything before it describes hops
further upstream, and the head of the list is whatever the original client
chose to send. The parser split on `,` — so it anticipated multiple elements
— and then took `.next()`.

Where the chain has two hops (or XFCC is in an append-forward mode), a caller
can prepend `Cert=<PEM of a victim device>` and be authenticated as that
device: the proxy appends its own element, and canopy derives the SPKI from
the attacker's. Device certificates are public, so this is a device-auth
bypass on every mTLS-authenticated public-server endpoint and on enrollment's
`resolve_spki`. It's only safe if the proxy replaces the header outright,
which code that parses a list can't assume.

Element selection now takes the last element, and the split is
quote-aware — Envoy quotes any value containing a separator, and mis-slicing
the list is precisely what decides which hop is trusted. A last element with
no `Cert=` yields nothing rather than searching backwards, since an earlier
element is the untrusted input this is meant to ignore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
@passcod

passcod commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Fortunately the envoy ingress path isn't yet live.

@passcod
passcod marked this pull request as ready for review August 1, 2026 19:57
@passcod
passcod added this pull request to the merge queue Aug 1, 2026
@passcod
passcod removed this pull request from the merge queue due to a manual request Aug 1, 2026
Reading the trusted end of XFCC fixes which element is believed, but not
*whether the header should be believed at all*. `x-forwarded-client-cert` was
read unconditionally, with no check that the request came through Envoy — and
preferred over `mtls-certificate`, so it overrode the header the live nginx
ingress actually sets.

That matters because `resolve` authenticates via `Device::from_key`: a lookup
on the certificate's public key, with no proof of possession. Sound when a
proxy terminates mTLS and vouches for the peer, but a public key is not a
secret. Envoy isn't deployed and nginx doesn't use XFCC — so nginx has no
reason to strip it, and any caller able to set the header could present an
enrolled device's certificate and be resolved as that device.

Each path now has its own switch:

- `CANOPY_DEVICE_AUTH_MTLS_HEADER` (default **on**) — `mtls-certificate` and
  `ssl-client-cert`, what the live ingress sets. On by default so prod is
  unchanged.
- `CANOPY_DEVICE_AUTH_XFCC` (default **off**) — `x-forwarded-client-cert`.
  Off until the Envoy path is live, at which point flipping the pair swaps
  ingress without a code change.

An unrecognised value keeps the default rather than flipping a security
switch on a typo, and both-on or both-off warns at startup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
@passcod
passcod enabled auto-merge August 1, 2026 20:13
The XFCC gate defaults off, which broke the public-server tests that
authenticate over `x-forwarded-client-cert` — the test harness stands in for
a trusted ingress, and the suite covers both transports.

`force_trusted_cert_headers` sets the trusted set explicitly instead of
reading the environment, so the harness is deterministic rather than racing a
`LazyLock` against whichever test touches it first. Which headers a real
deployment trusts stays configuration, unit-tested in `mtls`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
@passcod
passcod disabled auto-merge August 1, 2026 20:22
@passcod

passcod commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Actually instead of two gates, let's have one gate that spells out the header to trust, and default it to the current prod path. That renders the both on both off check unnecessary as the situation becomes impossible.

claude added 3 commits August 1, 2026 21:02
Replaces the two booleans with a single `CANOPY_DEVICE_AUTH_CERT_HEADER`
naming the header this deployment's ingress sets — `mtls` (default, the live
nginx path) or `xfcc`. "Trust both" and "trust neither" stop being states the
configuration can express, so the warnings guarding against them go too.

It's also no longer a process global. `ClientCertHeader` lives in server
state and reaches the extractor through `FromRef`, the idiom `AuthDevice`
already uses for the pool and the tailnet directory. That's what lets a test
choose, which a `LazyLock` couldn't.

The suite now runs on XFCC. Production exercises the nginx header
continuously; XFCC is the shape that goes live at the cutover and had no
coverage outside unit tests. Testing the new shape means removing the old one
later is a delete rather than a rewrite. `device_key_authentication_works_on_
the_nginx_header` keeps end-to-end coverage of the live path (and asserts
XFCC is refused there), and goes at the cutover with the `Mtls` variant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
CI caught what a stale base hides: main has added tests using
`mtls-certificate` (products/entitlements, billing tags, restore, names) and
the suite now runs on XFCC, so they authenticated against the wrong header
and 401'd. Merges cleanly and compiles — only running it finds this.

Converts the 45 that arrived with main, and one in `device_key_auth.rs` that
I over-excluded when protecting the deliberate nginx-path test in that file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
@passcod
passcod enabled auto-merge August 1, 2026 21:23
@passcod
passcod disabled auto-merge August 1, 2026 21:24
@passcod passcod changed the title fix(device-auth): read the trusted end of the XFCC header (audit H7) fix(device-auth): trust one client-cert header, and its trusted end (audit H7) Aug 1, 2026
claude added 2 commits August 1, 2026 21:25
Rebasing the collision away rather than fixing it after each landing. This
branch flips the suite to XFCC, so every test that arrives on the base using
`mtls-certificate` breaks here — and does so only at runtime, since it merges
cleanly and compiles. That has now happened twice, and #429 was queued up to
make it three.

Basing on #429 instead of main puts the two in a defined order and converts
its `credentials_during_a_rotation_is_503` here, where the flip lives.
@passcod
passcod changed the base branch from main to claude/pr-370-fix-h5-rotation-slot August 1, 2026 21:29

passcod commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Stacked this onto #429 — base is now claude/pr-370-fix-h5-rotation-slot, so the diff above is just this PR's own 20 files.

That retires the note at the bottom of the description. The recurring problem was that this branch flips the integration suite to XFCC, so any test arriving on the base with mtls-certificate fails here — and merges cleanly and compiles, so only a test run finds it. It happened with main's products/entitlements and billing-tag tests, and #429's credentials_during_a_rotation_is_503 was lined up to be the third. That one is converted here now, where the flip lives.

Order is fixed: #429 merges, GitHub retargets this to main, and it goes in behind it.


Generated by Claude Code

Base automatically changed from claude/pr-370-fix-h5-rotation-slot to main August 1, 2026 21:37
@passcod
passcod enabled auto-merge August 1, 2026 21:41
@passcod
passcod added this pull request to the merge queue Aug 1, 2026
Merged via the queue into main with commit 2cbdd5b Aug 1, 2026
7 checks passed
@passcod
passcod deleted the claude/pr-370-fix-h7-xfcc-last branch August 1, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants