Encrypt VLAN/VXLAN tunnel traffic with per-tunnel AES-256 keys#117
Closed
antoncxx wants to merge 13 commits into
Closed
Encrypt VLAN/VXLAN tunnel traffic with per-tunnel AES-256 keys#117antoncxx wants to merge 13 commits into
antoncxx wants to merge 13 commits into
Conversation
Every VLAN/VXLAN link the server builds now gets its own random AES-256 key, sent to both endpoints alongside the existing VlanSetup/VxlanSetup control messages. VLAN traffic is encrypted/decrypted in nullnet-client's userspace forwarder (AES-256-GCM); VXLAN traffic is protected by a per-tunnel kernel IPsec (XFRM/ESP) SA, since that path is handled entirely by the kernel's native vxlan interface. Each VXLAN tunnel also gets its own UDP dstport (replacing the shared 4789 default) so XFRM policies can tell concurrent tunnels between the same host pair apart.
Previously, tunnels whose two endpoints happened to be colocated on the same physical host got no encryption at all: VXLAN's same-host branch used a plain veth pair with no IPsec, and VLAN traffic between two same-host access ports was switched directly by OVS without ever transiting the encrypting userspace forwarder. Neither is reachable by a network sniffer, but both are readable by anything else with sufficient privilege on that same host (a differently-privileged container, a host-level process) — which matters for a project whose whole premise is not trusting other things on the network by default. VXLAN: wrap the same-host veth pair in MACsec (802.1AE, GCM-AES-256), keyed with the same per-tunnel key already delivered for the cross-host IPsec path. Cascades away on teardown when the underlying veth is deleted; also deleted explicitly for clarity. VLAN: replace the single default-normal OVS flow with two more specific ones — traffic arriving from the trunk (already decrypted) is delivered by normal switching; traffic arriving from any access port always exits via the trunk, never directly to another access port. This forces every packet through the TAP and therefore through the existing encrypt/decrypt path in forward/send.rs and forward/receive.rs, with no changes needed there.
iproute2's macsec option parser is positional: `port` (part of this device's own SCI) must appear before `cipher`, not after. The previous order failed with "macsec: unknown command \"port\"?" every time, silently, because the command's stderr was suppressed to tolerate an unrelated race (the veth-pair creation a few lines earlier). None of the four macsec setup commands actually race against the sibling script invocation — each side only touches its own uniquely-named interface — so their stderr is no longer suppressed either, making a real failure here loud instead of silent next time. Found by live-testing on a real same-host deployment: `ip macsec show` was empty despite the veth pair (and its MTU bump) being created correctly, which pointed straight at the interface-creation line right after it.
Diagnostic aid for tracking down intermittent "connection reset" on proxied SSH-over-VLAN: every silent-drop point (missing key, decrypt/auth failure, malformed datagram) and every firewall verdict on both send and receive now logs to stderr with [DEBUG] prefix. In particular, receive.rs now logs before crafting a REJECT reply — that's the one code path that sends a real TCP RST back to the peer, so if a REJECT verdict is firing intermittently on legitimate traffic, this will show it directly instead of us guessing from symptoms alone. Intended to be reverted once the root cause is found — not a permanent logging addition.
The same-host defense-in-depth fix replaced OVS's single default flow
(priority=0,actions=normal) with a generic "everything from an access
port goes to the trunk" rule using a raw `output:<TAP>` action. Unlike
`actions=normal`, `output` does not re-add the 802.1Q tag that access
ports only carry internally (as an implicit port association, not part
of the packet's own bytes) — so every redirected frame arrived at
nullnet-client's TAP already stripped of its VLAN tag, and got dropped
by forward/send.rs as malformed ("Packet missing VLAN tag"), silently,
for every single packet on every VLAN tunnel that hit this path.
Replaced the one generic redirect rule with one precise rule per access
port (installed alongside the port itself in configure_access_port, torn
down alongside it in remove_vlan), each matching only its own in_port and
explicitly pushing the correct 802.1Q tag (push_vlan + mod_vlan_vid)
before sending to the trunk. The original priority=0 default-normal rule
is restored as a safety fallback for the narrow window before a fresh
access port's own rule lands.
Found via live testing: intermittent SSH-over-proxy connection resets
that traced back to a flood of "Packet missing VLAN tag" errors in
nullnet-client's own logs.
Same diagnostic purpose as the earlier debug commit — need to see what this frame actually was (IPv6 NDP/multicast noise vs. something that should have been carried) rather than just knowing the match fell through to the catch-all arm.
Same class of bug as the earlier macsec argument-order fix: `ip xfrm
policy add/delete` requires the full selector (src/dst/proto/dport) to
appear as one contiguous block, with `dir` only afterward — confirmed
against `ip xfrm policy help`'s own grammar (`SELECTOR dir DIR ...`).
The previous order put `dir out`/`dir in` in the middle of the selector,
between dst and proto, which the parser misreads as two separate `proto`
values ("duplicate \"unknown\": \"proto\" is the second value") and
rejects outright — so no policy was ever installed on either side of a
genuinely cross-host VXLAN tunnel, and traffic between the two hosts had
no route (the two sides' overlay subnets are meant to look like one flat
L2 domain over the VXLAN link, which never got its required IPsec policy
in place).
Found via live cross-host testing: an sshd service hosted on a second,
non-colocated node was unreachable ("No route to host") because its
side of the tunnel's bridge/VXLAN interface never got created at all —
traced back to this exact error in the client's own log output.
Same class of bug as the policy argument-order fix, one command over: `ip xfrm state add` requires ID [ALGO-LIST] [mode MODE] ... per its own help text — the algorithm clause (`aead ...`) has to come before `mode`, not after. The previous order (`mode transport aead ...`) is backwards and fails outright with a bare "RTNETLINK answers: Invalid argument", so no Security Association was ever installed on either side of a cross-host VXLAN tunnel — encryption never had a chance to run at all.
… bug The state-vs-policy argument-order fixes so far were based on reading `ip xfrm state help` / `ip xfrm policy help`'s grammar summaries, not confirmed against the actual parser behavior for every command — and the "RTNETLINK answers: Invalid argument" error is still occurring after both fixes, meaning at least one of those diagnoses was incomplete or wrong. Rather than keep guessing from documentation, trace every command with its real substituted values so the next failure shows the exact command line next to its exact error. Prints AEAD key material to stderr while enabled — treat logs as sensitive until this is reverted.
VXLAN IDs start at 101 (net_id_pool.rs's MIN_NET_ID), and the SPI was derived directly from the raw vxlan_id — landing squarely in SPI values 1-255, which RFC 4301 reserves for IANA and which the kernel's XFRM code rejects outright with a bare "RTNETLINK answers: Invalid argument". This was the actual cause of the state-add failures, not an argument-order issue (confirmed by testing with two genuinely real, non-loopback addresses and still hitting the identical error). Offset the SPI by a fixed +1000 so it's always comfortably clear of the reserved range, matching between vxlan-setup.sh (install) and vxlan-teardown.sh (delete, which must derive the identical SPI to remove the right SA).
The actual root cause behind every "RTNETLINK answers: Invalid argument" we chased across the SPI-range and argument-order fixes: `ip xfrm state add`'s key material (ALGO-KEYMAT) must be prefixed with "0x" — a bare hex string of the exact same value and length is silently rejected by the kernel. Confirmed by extensive live A/B testing on a real target host: identical algorithm, key length, and argument order succeeded the moment "0x" was added and failed identically without it, for both classic enc+auth (cbc(aes)+hmac(sha1)) and this script's actual aead (rfc4106(gcm(aes))) construction. The SPI-reserved-range fix (c7628f9) and the state/policy argument-order fixes (e0a0de8, 1e13ad9) were real, independently-necessary corrections found along the way — this was the last remaining piece keeping cross-host VXLAN encryption from ever actually installing an SA.
…ing xfrm bug" This reverts commit dba3542.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every VLAN/VXLAN link the server builds now gets its own random AES-256 key, sent to both endpoints alongside the existing VlanSetup/VxlanSetup control messages. VLAN traffic is encrypted/decrypted in nullnet-client's userspace forwarder (AES-256-GCM); VXLAN traffic is protected by a per-tunnel kernel IPsec (XFRM/ESP) SA, since that path is handled entirely by the kernel's native vxlan interface. Each VXLAN tunnel also gets its own UDP dstport (replacing the shared 4789 default) so XFRM policies can tell concurrent tunnels between the same host pair apart.