Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -195,8 +195,7 @@ topology-data.json
# ============================================================
# Lab and local dev overrides
# ============================================================
lab/gvpc/clab-*/
lab/network/clab-*/
build/

# ============================================================
# Miscellaneous
Expand Down
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,9 @@ Under the hood, Galactic uses Segment Routing over IPv6 (SRv6) for efficient, de

## Getting Started

Two ContainerLab environments are available under [`lab/`](./lab/):
A ContainerLab environment is available under [`deploy/containerlab/`](./deploy/containerlab/):

- **[`lab/network/`](./lab/network/)** — Standalone SRv6 underlay lab (FRR + GoBGP, no Kubernetes). Good starting point for understanding the routing layer.
- **[`lab/gvpc/`](./lab/gvpc/)** — Three Kind clusters wired over an SRv6 transit mesh. The full GVPC multi-cluster environment.
- **[`deploy/containerlab/`](./deploy/containerlab/)** — Three Kind clusters wired over an SRv6 transit mesh. The full GVPC multi-cluster environment with FRR underlay and GoBGP L3VPN overlay.

See the [galactic DevContainer](./.devcontainer/galactic/) for development environment setup. On ARM64 / OrbStack, use the [containerlab DevContainer](./.devcontainer/containerlab-dood/) to run ContainerLab via Docker-out-of-Docker.

Expand Down Expand Up @@ -67,7 +66,7 @@ task ci:e2etest # full e2e lifecycle — spins up a Kind cluster, builds an

`task ci:unittest` is the fast path for development; it runs the same command as the CI `test-unit` job. `task ci:e2etest` requires Docker and Kind and mirrors the CI `test-e2e` job exactly, including automatic cluster cleanup via a `trap` on exit.

Lab environments have their own `Taskfile.yaml`; run `task` from the relevant directory (`lab/network/` or `lab/gvpc/`) to see available tasks.
The lab environment has its own `Taskfile.yaml`; run `task` from `deploy/containerlab/` to see available tasks.

## License

Expand Down
86 changes: 48 additions & 38 deletions lab/gvpc/README.md → deploy/containerlab/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# GVPC Lab
# Galactic VPC Lab Deployment

Three Kind clusters connected over an IPv6 SRv6 transit mesh. Each cluster runs FRR
as a node routing daemon (hostNetwork DaemonSet) to peer with the transit layer via
Expand Down Expand Up @@ -111,17 +111,17 @@ Worker SRv6 node SIDs (on `lo-galactic`):
## Lab layout

```
gvpc/
deploy/containerlab/
├── gvpc.clab.yaml
├── Taskfile.yaml
├── containers/
│ └── kindest-node-galactic/ # Custom Kind node image (Cilium, Multus, cert-manager)
│ ├── kindest-node-galactic/ # Custom Kind node image (Cilium, Multus, cert-manager, galactic)
│ ├── gobgp/ # GoBGP container built from upstream release binary
│ └── frr/ # FRR container built from Alpine edge
├── resources/
│ ├── iad-underlay.k8s.yaml # FRR DaemonSet — iad cluster PE
│ ├── sjc-underlay.k8s.yaml # FRR DaemonSet — sjc cluster PE
│ ├── infra-control-plane.k8s.yaml # FRR DaemonSet — infra cluster route reflector
│ ├── iad-overlay.k8s.yaml # GoBGP DaemonSet — iad cluster L3VPN PE
│ └── sjc-overlay.k8s.yaml # GoBGP DaemonSet — sjc cluster L3VPN PE
│ ├── underlay/ # FRR DaemonSet kustomize overlays (iad, sjc, infra)
│ ├── overlay/ # GoBGP DaemonSet kustomize overlays (iad, sjc)
│ └── cosmos/ # Cosmos BGP CRs (BGPInstance, BGPSession, BGPProvider)
├── node_files/
│ ├── iad/ config.yaml
│ ├── sjc/ config.yaml
Expand All @@ -132,8 +132,6 @@ gvpc/
│ └── tr4/ frr.conf startup.sh
├── group_files/
│ ├── common/ hosts vtysh.conf startup-lib.sh
│ ├── control/ daemons
│ ├── pe/ daemons
│ └── transit/ daemons
└── scripts/
├── host-setup.sh
Expand All @@ -151,37 +149,51 @@ gvpc/
## Quick start

```bash
task up # build Kind node image, apply host sysctls, deploy lab
task underlay # apply FRR DaemonSets to all three clusters
task overlay # apply GoBGP DaemonSets to iad and sjc clusters
cd deploy/containerlab
task deploy # build all images, apply host sysctls, deploy lab end-to-end
```

To tear down and start fresh:

```bash
task destroy # remove all lab containers and Kind clusters
task clean # destroy + delete built images and lab artifacts
task deploy
```

## Tasks

| Task | Description |
|------------------|-----------------------------------------------------------|
| `build` | Build the custom `kindest/node:galactic` image |
| `up` | Build, apply host sysctls, and deploy the lab |
| `down` | Destroy the lab and remove state |
| `reload` | Full rebuild — destroy then redeploy |
| `inspect` | Show running nodes and management addresses |
| `graph` | Generate a draw.io diagram for the topology |
| `host-setup` | Apply required host sysctls (IPv6 forwarding etc.) |
| `underlay` | Apply FRR DaemonSets to all three clusters |
| `overlay` | Pull GoBGP image, load into clusters, apply DaemonSets |
| `clean` | Destroy lab, remove state, and delete the Kind node image |
| Task | Description |
|--------------------|----------------------------------------------------------------|
| `build` | Build all container images (node, cosmos, gobgp, frr) |
| `build:node` | Build the custom `kindest/node:galactic` image |
| `build:cosmos` | Build the Cosmos BGP operator image from source |
| `build:gobgp` | Build the GoBGP container from upstream release binary |
| `build:frr` | Build the FRR container from Alpine edge |
| `deploy` | Build images, apply host sysctls, and deploy the lab |
| `deploy:topology` | Deploy the ContainerLab topology (transit routers + clusters) |
| `deploy:images` | Load images into Kind clusters and wait for cosmos rollout |
| `deploy:underlay` | Apply FRR DaemonSets to all three clusters |
| `deploy:overlay` | Apply GoBGP DaemonSets and Cosmos BGP CRs to iad and sjc |
| `destroy` | Destroy the lab and remove all Kind clusters |
| `reload` | Full rebuild — destroy then redeploy |
| `inspect` | Show running nodes and management addresses |
| `graph` | Generate a draw.io diagram for the topology |
| `host-setup` | Apply required host sysctls (IPv6 forwarding, inotify limits) |
| `clean` | Destroy lab, delete built images, and remove lab artifacts |
| `test` | Run all verification checks |

## Verification

### Transit underlay

```bash
# iBGP full mesh — expect all sessions Established
docker exec tr1 vtysh -c "show bgp ipv6 unicast summary"
docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast summary"

# Worker SRv6 prefixes should be present on all TR nodes
docker exec tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff01::/48"
docker exec tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff02::/48"
docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff01::/48"
docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff02::/48"
```

### FRR DaemonSets (eBGP underlay)
Expand All @@ -190,10 +202,10 @@ docker exec tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff02::/48"
# Check pods are running
docker exec iad-control-plane kubectl get pods -n iad-underlay
docker exec sjc-control-plane kubectl get pods -n sjc-underlay
docker exec infra-control-plane kubectl get pods -n infra-control-plane
docker exec infra-control-plane kubectl get pods -n infra-underlay

# Run vtysh inside a pod
docker exec iad-control-plane kubectl exec -n iad-underlay deploy/iad-underlay \
docker exec iad-control-plane kubectl exec -n iad-underlay ds/iad-underlay \
-- vtysh -c "show bgp ipv6 unicast summary"
```

Expand All @@ -205,14 +217,12 @@ docker exec iad-control-plane kubectl get pods -n iad-overlay
docker exec sjc-control-plane kubectl get pods -n sjc-overlay

# Check iBGP session to infra-control-plane
docker exec iad-control-plane kubectl exec -n iad-overlay deploy/iad-overlay \
-- gobgp neighbor
docker exec sjc-control-plane kubectl exec -n sjc-overlay deploy/sjc-overlay \
-- gobgp neighbor

# Inspect VRF RIB
docker exec iad-control-plane kubectl exec -n iad-overlay deploy/iad-overlay \
-- gobgp vrf
docker exec iad-control-plane kubectl exec -n iad-overlay ds/iad-overlay -- gobgp neighbor
docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobgp neighbor

# Inspect VPN RIB
docker exec iad-control-plane kubectl exec -n iad-overlay ds/iad-overlay -- gobgp global rib -a vpnv6
docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobgp global rib -a vpnv6
```

## Notes
Expand Down
173 changes: 173 additions & 0 deletions deploy/containerlab/Taskfile.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
version: '3'

vars:
LAB:
sh: awk '/^name:/ {print $2; exit}' *.clab.yaml
TOPO:
sh: echo *.clab.yaml
COSMOS_IMAGE: ghcr.io/milo-os/cosmos:latest
GOBGP_VERSION: "3.37.0"
GOBGP_IMAGE: gobgp:{{.GOBGP_VERSION}}
FRR_VERSION: "10.6.1"
FRR_IMAGE: frr:{{.FRR_VERSION}}

tasks:
default:
silent: true
cmds:
- task --list

build:
desc: Build all container images
cmds:
- task: "build:node"
- task: "build:cosmos"
- task: "build:gobgp"
- task: "build:frr"

"build:node":
desc: Build the Kind node image with the galactic CNI plugin
cmds:
- docker build --network=host -t kindest/node:galactic -f containers/kindest-node-galactic/Dockerfile ../..

"build:cosmos":
desc: Build the cosmos BGP operator image from source
status:
- docker image inspect {{.COSMOS_IMAGE}} > /dev/null 2>&1
cmds:
- |
if [ ! -d "build/cosmos" ]; then
git clone --depth=1 https://github.com/milo-os/cosmos build/cosmos
fi
docker build -t {{.COSMOS_IMAGE}} -f build/cosmos/build/Dockerfile build/cosmos

"build:gobgp":
desc: Build the GoBGP container using the pre-built v{{.GOBGP_VERSION}} release binary
status:
- docker image inspect {{.GOBGP_IMAGE}} > /dev/null 2>&1
cmds:
- docker build --build-arg GOBGP_VERSION={{.GOBGP_VERSION}} -t {{.GOBGP_IMAGE}} containers/gobgp/

"build:frr":
desc: Build the FRR container from Alpine edge (v{{.FRR_VERSION}})
status:
- docker image inspect {{.FRR_IMAGE}} > /dev/null 2>&1
cmds:
- docker build --build-arg FRR_VERSION={{.FRR_VERSION}} -t {{.FRR_IMAGE}} containers/frr/

deploy:
desc: Build images and deploy the full lab end-to-end
deps: [build, host-setup]
cmds:
- task: "deploy:topology"
- task: "deploy:images"
- task: "deploy:underlay"
- task: "deploy:overlay"

"deploy:topology":
desc: Deploy the ContainerLab topology (transit routers + Kind clusters)
cmds:
- sudo containerlab deploy -t {{.TOPO}}

"deploy:images":
desc: Load all container images into the Kind clusters and wait for cosmos
cmds:
- kind load docker-image {{.COSMOS_IMAGE}} --name iad
- kind load docker-image {{.COSMOS_IMAGE}} --name sjc
- kind load docker-image {{.GOBGP_IMAGE}} --name iad
- kind load docker-image {{.GOBGP_IMAGE}} --name sjc
- kind load docker-image {{.FRR_IMAGE}} --name iad
- kind load docker-image {{.FRR_IMAGE}} --name sjc
- kind load docker-image {{.FRR_IMAGE}} --name infra
- docker exec iad-control-plane kubectl -n bgp-system rollout status daemonset bgp --timeout=120s
- docker exec sjc-control-plane kubectl -n bgp-system rollout status daemonset bgp --timeout=120s

destroy:
desc: Destroy the lab
cmds:
- sudo containerlab destroy -t {{.TOPO}} --cleanup

reload:
desc: Full rebuild and redeploy
cmds:
- task: destroy
- task: deploy

inspect:
desc: Inspect deployed nodes and management addresses
cmds:
- sudo containerlab inspect -t {{.TOPO}}

graph:
desc: Generate a draw.io diagram for the current topology
cmds:
- sudo containerlab graph -t {{.TOPO}} --drawio --drawio-dir .

host-setup:
desc: Apply host sysctls for this dual-stack lab
cmds:
- sudo ./scripts/host-setup.sh --no-persist

"deploy:underlay":
desc: Install the FRR underlay DaemonSets
cmds:
- ./scripts/install-underlay.sh

"deploy:overlay":
desc: Install the GoBGP overlay DaemonSets and cosmos BGP resources
cmds:
- ./scripts/install-overlay.sh

test:
desc: Run all verification checks
cmds:
- task: "test:bgp-transit"
- task: "test:bgp-underlay"
- task: "test:srv6"
- task: "test:l3vpn"

"test:bgp-transit":
desc: Verify transit router BGP sessions (iBGP full mesh)
cmds:
- |
for r in tr1 tr2 tr3 tr4; do
echo "--- $r ---"
docker exec clab-gvpc-$r vtysh -c "show bgp ipv6 unicast summary"
done

"test:bgp-underlay":
desc: Verify underlay BGP sessions on iad and sjc workers
cmds:
- |
docker exec iad-control-plane \
kubectl exec -n iad-underlay ds/iad-underlay \
-- vtysh -c "show bgp ipv6 unicast summary"
- |
docker exec sjc-control-plane \
kubectl exec -n sjc-underlay ds/sjc-underlay \
-- vtysh -c "show bgp ipv6 unicast summary"

"test:srv6":
desc: Verify SRv6 forwarding prefixes on tr1
cmds:
- docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff01::/48"
- docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff02::/48"
- docker exec clab-gvpc-tr1 vtysh -c "show bgp ipv6 unicast 2001:db8:ff03::/48"

"test:l3vpn":
desc: Verify GoBGP L3VPN neighbors and RIB on iad and sjc
cmds:
- docker exec iad-control-plane kubectl exec -n iad-overlay ds/iad-overlay -- gobgp neighbor
- docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobgp neighbor
- docker exec iad-control-plane kubectl exec -n iad-overlay ds/iad-overlay -- gobgp global rib -a vpnv6
- docker exec sjc-control-plane kubectl exec -n sjc-overlay ds/sjc-overlay -- gobgp global rib -a vpnv6

clean:
desc: Destroy the lab and delete artifacts
cmds:
- task: destroy
- docker rmi kindest/node:galactic || true
- docker rmi {{.GOBGP_IMAGE}} || true
- docker rmi {{.FRR_IMAGE}} || true
- docker rmi {{.COSMOS_IMAGE}} || true
- rm -rf clab-{{.LAB}} build/
12 changes: 12 additions & 0 deletions deploy/containerlab/containers/frr/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
ARG FRR_VERSION=10.6.1
FROM alpine:edge
ARG FRR_VERSION

# alpine:edge is a rolling release — base OS packages update on every rebuild.
# FRR is pinned via "frr=${FRR_VERSION}-r0". If Alpine bumps the package to -r1
# for the same FRR version, the build will fail; drop or adjust the -r0 suffix.
RUN apk add --no-cache \
"frr=${FRR_VERSION}-r0" \
frr-pythontools \
iproute2 \
tcpdump
13 changes: 13 additions & 0 deletions deploy/containerlab/containers/gobgp/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
ARG GOBGP_VERSION=4.5.0
FROM alpine:3.20 AS fetch
ARG GOBGP_VERSION
RUN apk add --no-cache wget ca-certificates && \
wget -O /tmp/gobgp.tar.gz \
"https://github.com/osrg/gobgp/releases/download/v${GOBGP_VERSION}/gobgp_${GOBGP_VERSION}_linux_amd64.tar.gz" && \
tar -xzf /tmp/gobgp.tar.gz -C /usr/local/bin gobgpd gobgp

FROM alpine:3.20
COPY --from=fetch /usr/local/bin/gobgpd /usr/local/bin/gobgpd
COPY --from=fetch /usr/local/bin/gobgp /usr/local/bin/gobgp
EXPOSE 50051 179
ENTRYPOINT ["gobgpd"]
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,18 @@ SHELL ["/bin/bash", "-eo", "pipefail", "-c"]

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git \
tcpdump \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /galactic

COPY --from=builder /src/bin/galactic ./bin/galactic
COPY --chmod=0755 lab/gvpc/containers/kindest-node-galactic/scripts/ ./scripts/
COPY deploy/containerlab/resources/ ./resources/
COPY --chmod=0755 deploy/containerlab/containers/kindest-node-galactic/scripts/ ./scripts/

# Wrap kubectl to work around OrbStack DooD: kind runs `kubectl apply` for the
# StorageClass step immediately after kubeadm init, but the apiserver briefly
# goes offline while OrbStack's bridge finishes coming up, causing worker nodes
# to never reach kubeadm join. The wrapper polls /healthz before passing through.
COPY --chmod=0755 lab/gvpc/containers/kindest-node-galactic/kubectl-wrapper /usr/local/bin/kubectl
COPY --chmod=0755 deploy/containerlab/containers/kindest-node-galactic/kubectl-wrapper /usr/local/bin/kubectl
Loading
Loading