Encrypt VLAN/VXLAN tunnel traffic with per-tunnel AES-256 keys#118
Encrypt VLAN/VXLAN tunnel traffic with per-tunnel AES-256 keys#118antoncxx wants to merge 7 commits into
Conversation
997096f to
d8be059
Compare
Adds per-tunnel AES-256 encryption for all VLAN/VXLAN tunnel traffic: MACsec (802.1AE, GCM-AES-256) wraps same-host veth-pair tunnels, and IPsec (XFRM/ESP) encrypts cross-host VXLAN traffic. Also includes the eBPF firewall and MTU fixes needed to make the cross-host path work end-to-end: recognize per-tunnel VXLAN dstports in the data plane, allow ESP so IPsec-wrapped VXLAN isn't dropped, and shrink the overlay MTU (down to 1080) to leave room for cross-host ESP overhead. Also fixes a clippy -D warnings failure in nullnet-proxy from a deprecated openssl API (Asn1StringRef::as_utf8 -> to_string).
d8be059 to
5e7734f
Compare
The encryption feature's kernel-level overhead (VXLAN + ESP) needed the overlay MTU pushed down to 1080 in this environment, and that cost isn't worth paying everywhere. Add a server-side env var, ENCRYPTION_ENABLED (default: enabled, preserving current behavior), threaded through the existing per-edge setup path exactly like dstport already is - both regular chain edges and egress-gateway edges respect it. When disabled: the server generates no real key (placeholder zero bytes, which the client never parses), VLAN falls back to plain userspace forwarding via a new VlanCipher::Plaintext state (distinct from "unknown tunnel, drop"), and vxlan-setup.sh skips both the same-host MACsec wrap and the cross-host XFRM state/policy install, leaving a bare vxlan/veth link. Teardown needed no changes - its cleanup commands are already safe no-ops whether or not encryption was ever installed. Wire protocol gains one field per direction: VlanSetup.encrypted and VxlanSetup.encrypted, so each endpoint learns the tunnel's encryption state independently of the server's own env var.
VXLAN_PORTS and PEERS were two independent flat sets, checked with a plain AND - a packet just needed a known port AND a known peer, not necessarily the SAME tunnel's port and peer. With multiple concurrent tunnels to different peers, this meant tunnel B's peer IP could satisfy tunnel A's port. XFRM's own policy lookup masks this when encryption is on (it's scoped to the exact src/dst/port/SPI tuple and needs the right key), but with ENCRYPTION_ENABLED=false the firewall is the only gate, so the gap becomes real. VXLAN_PORTS now maps port -> the specific peer it was allocated to (was port -> marker), and data_plane() checks that pairing for a tunnel's dynamic dstport instead of independent set membership. The two fixed shared ports (4789 legacy, 9999 forward-socket) still check "any known peer" - they aren't tunnel-exclusive, so pairing doesn't apply there. This narrows cross-tunnel confusion at the firewall layer; it doesn't add cryptographic anti-spoofing (only XFRM/MACsec do that) - an attacker who can already forge a peer's real source IP can still hit that peer's own correctly-paired port.
Both were hand-written without running rustfmt at the time. No behavior change - cargo fmt --all --check now passes clean across the workspace, and the standalone ebpf crate was already clean. clippy -D warnings is clean on nullnet-client, nullnet-ebpf, nullnet-grpc-lib, and xtask. Could not run it on nullnet-server in this sandbox - its build.rs shells out to `npm` to build the UI, and npm isn't installed here (pre-existing gap, unrelated to this repo). Confirmed nullnet-server still builds and its 91 tests still pass.
|
I left a couple comments. |
…EADME PR NullNet-ai#118 review raised a real blocker: making dedicated dstport allocation conditional on encryption (previous commit) still capped total concurrent encrypted VXLAN tunnels at 40k system-wide, down from the ~2M the NetIdPool actually supports - the UdpPortPool was a single pool shared across every edge regardless of which two hosts it was between. XFRM policies already select on the full (src, dst, proto, dport) tuple (confirmed directly in vxlan-setup.sh's `ip xfrm policy add` calls) - so two different host pairs reusing the same port number never collide; only concurrent encrypted tunnels between the *same* two hosts actually need distinct ports. Orchestrator now keeps one UdpPortPool per (host_a, host_b) pair (order-independent key) instead of a single global one, so the 40k ceiling only applies per host pair - for any deployment with more than a handful of distinct host pairs, this removes the practical cap entirely. allocate_vxlan_port(net_id, host_a, host_b) replaces the old net_id-only signature; net_id_ports now remembers which pair's pool to free back into on teardown. Added direct unit tests for the pairing behavior (distinct within a pair, reusable across different pairs, order-independent, freed correctly on teardown). Also finishes the other review comment: documents ENCRYPTION_ENABLED in README.md (server env var section).
PR NullNet-ai#118 review raised a real blocker: making dedicated dstport allocation conditional on encryption (previous commit) still capped total concurrent encrypted VXLAN tunnels at 40k system-wide, down from the ~2M the NetIdPool actually supports - the UdpPortPool was a single pool shared across every edge regardless of which two hosts it was between. XFRM policies already select on the full (src, dst, proto, dport) tuple (confirmed directly in vxlan-setup.sh's ip xfrm policy add calls) - so two different host pairs reusing the same port number never collide; only concurrent encrypted tunnels between the same two hosts actually need distinct ports. Orchestrator now keeps one UdpPortPool per (host_a, host_b) pair (order-independent key) instead of a single global one, so the 40k ceiling only applies per host pair - for any deployment with more than a handful of distinct host pairs, this removes the practical cap entirely. allocate_vxlan_port(net_id, host_a, host_b) replaces the old net_id-only signature; net_id_ports now remembers which pair's pool to free back into on teardown. Added direct unit tests for the pairing behavior (distinct within a pair, reusable across different pairs, order-independent, freed correctly on teardown). Also finishes the other review comment: documents ENCRYPTION_ENABLED in README.md (server env var section).
VXLAN Encryption Scaling: Options ConsideredContext: encrypting VXLAN tunnels via XFRM needs a way to disambiguate concurrent 1. Per-host-pair port pool — SHIPPEDScope the 40k-port pool per (host_a, host_b) pair instead of one global pool. 2. Mark-based XFRM disambiguation — investigated, doesn't fit
3. XFRM interfaces (
|
clippy -D warnings flagged net_id_ports's nested tuple type (HashMap<u32, ((IpAddr, IpAddr), u16)>) as too complex. Factored into HostPair and AllocatedPort type aliases, also used for udp_port_pools and host_pair()'s return type.
2 participants
Every VLAN/VXLAN link the server builds now gets its own random AES-256 key, sent to both endpoints alongside the existing VlanSetup/VxlanSetup control messages. VLAN traffic is encrypted/decrypted in nullnet-client's userspace forwarder (AES-256-GCM); VXLAN traffic is protected by a per-tunnel kernel IPsec (XFRM/ESP) SA, since that path is handled entirely by the kernel's native vxlan interface. Each VXLAN tunnel also gets its own UDP dstport (replacing the shared 4789 default) so XFRM policies can tell concurrent tunnels between the same host pair apart.
Same-host tunnels are covered too, for defense-in-depth: the same-host VXLAN veth pair is wrapped in MACsec (802.1AE, GCM-AES-256) keyed with the same per-tunnel key as the cross-host IPsec path, and same-host VLAN traffic is forced through the encrypting userspace forwarder via precise per-access-port OVS flows instead of being switched directly.