diff --git a/sdx/README.md b/sdx/README.md index 46c2f2f..b9ae7f1 100644 --- a/sdx/README.md +++ b/sdx/README.md @@ -6,7 +6,7 @@ A Kubernetes deployment for running SDX (Secure Data Exchange) Edge Servers as h ### Helm Chart (`chart/sdx-edge`) -**Chart Version:** 0.3.4 +**Chart Version:** 0.3.6 **App Version:** 3.9.1 Deploys a Kong Gateway data plane node configured for secure data exchange operations. The chart includes: @@ -54,7 +54,7 @@ helm upgrade --install ${EDGE_ID} \ ```sh helm package sdx-edge -helm push sdx-edge-0.3.5.tgz oci://ghcr.io/bcgov/aps-devops +helm push sdx-edge-0.3.6.tgz oci://ghcr.io/bcgov/aps-devops ``` #### Configuration @@ -73,7 +73,12 @@ The following table lists the configurable parameters in `values.yaml`: | `sdx_aggregator_url` | SDX aggregator service endpoint | `gwaggregator-api-gov-bc-ca-lab.dev.api.gov.bc.ca` | | `mtls_required` | Enable/disable mutual TLS requirement | `true` | | `https_proxy` | HTTP proxy URL for restricted network environments | `""` (empty, disabled) | -| `bootstrap.tls.token` | Bootstrap token for initial certificate request | `""` (must be provided) | +| `bootstrap.tls.token` | One-time CA token for certificate bootstrap. Clear on promote with `--set-string bootstrap.tls.token=""` so the bootstrap Job is dropped, not mutated. | `""` (must be provided) | +| `bootstrap.stageSecret` | Write `{release}-client-next` and skip Kong restart | `false` | +| `bootstrap.deferRestart` | Skip Kong restart after writing live bootstrap secrets | `false` | +| `renewal.deferRestart` | Skip Kong restart after certificate renewal | `false` | +| `rotation.promote` | One-shot: copy `{release}-client-next` to live secrets and restart Kong. Reset to `false` after the Job succeeds. | `false` | +| `rotation.nonce` | Suffix for the promote Job name (bump on each promote) | `"1"` | | `tls.client.cn` | Common Name for client certificate | `example.com` | | `tls.server.ip` | IP address to add as SAN to edge server certificate | `""` (optional) | | `tls.public_ca` | PEM-encoded public CA certificates for trust chain | (includes Sectigo, USERTrust, SDX, APS, Amazon, Let's Encrypt root CAs) | @@ -97,6 +102,84 @@ The following values must be set during installation: - `bootstrap.tls.token` - Required for certificate bootstrapping - `route.host` - Required for proper external routing +#### Runtime-group key rotation + +Private-key generation and Kong restart are separate Helm steps so the new +public key can be published before the data plane starts signing with it. + +`--reuse-values` persists flags in the release, so stage and promote must +clear one-shot values instead of only toggling `stageSecret`. + +1. Create a one-time CA token, then bootstrap with staging. Use + `--wait --wait-for-jobs` so the bootstrap Job finishes before the next + step (it is not a Helm hook): + + ```sh + helm upgrade ${EDGE_ID} oci://ghcr.io/bcgov/aps-devops/sdx-edge \ + --reuse-values \ + --wait --wait-for-jobs \ + --set bootstrap.tls.token=${TOKEN} \ + --set bootstrap.stageSecret=true + ``` + + This writes `sdx-edge-${EDGE_ID}-client-next` and does **not** restart Kong. + The Job signs the CSR itself (`step ca sign`); only `tls.crt` and `tls.key` + are stored. + +2. Extract the staged certificate and publish it with `sdx-keys.r1` + `operation=rotate` (keeps the old key for overlap). Confirm JWKS lists both + kids. + + ```sh + kubectl get secret sdx-edge-${EDGE_ID}-client-next \ + -o jsonpath='{.data.tls\.crt}' | base64 -d + ``` + + Pass that PEM as `certificatePem` on `sdx-keys.r1` (see + `docs/sdx-keys-rotation.md` in api-services-portal). + +3. Promote the staged secret and rolling-restart Kong. Clear the consumed + bootstrap token with an **empty string** so Helm **drops** the + `*-boot-` Job and its Secret. Do not use `--set …=null`: with + `--reuse-values`, Helm 3 coalescing strips the null and the previous token + is rendered again. Leaving the token set keeps that Job in the release + while `stageSecret` flips the pod template from staging writes to live + writes. Kubernetes Job specs are immutable, so the upgrade is rejected + while the completed Job still exists; if TTL already removed it, Helm + recreates the Job with the spent one-time token. + + ```sh + helm upgrade ${EDGE_ID} oci://ghcr.io/bcgov/aps-devops/sdx-edge \ + --reuse-values \ + --wait \ + --set-string bootstrap.tls.token="" \ + --set bootstrap.stageSecret=false \ + --set rotation.promote=true \ + --set rotation.nonce=$(date +%s) + ``` + + Helm waits for the post-upgrade promote Job before this command returns. + +4. Reset `rotation.promote` so later `--reuse-values` upgrades do not render + the hook again (that would copy `{release}-client-next` over the live + secrets and restart Kong): + + ```sh + helm upgrade ${EDGE_ID} oci://ghcr.io/bcgov/aps-devops/sdx-edge \ + --reuse-values \ + --set rotation.promote=false + ``` + +5. Confirm `trust-sign` emits the kid that matches the mounted private key, + wait through the verifier grace period, then delete the old kid with + `operation=delete`. + +Certificate **renewal** (`step ca renew`) keeps the same private key, so the +cron job still restarts Kong by default. Set `renewal.deferRestart=true` only +when restart is handled separately. + +See `docs/sdx-keys-rotation.md` in api-services-portal for the full contract. + **Example Override:** ```yaml diff --git a/sdx/chart/sdx-edge/Chart.yaml b/sdx/chart/sdx-edge/Chart.yaml index 93f82b9..843fad1 100644 --- a/sdx/chart/sdx-edge/Chart.yaml +++ b/sdx/chart/sdx-edge/Chart.yaml @@ -2,5 +2,5 @@ apiVersion: v2 name: sdx-edge description: A Helm chart for deploying an SDX Edge Server to a Kubernetes cluster type: application -version: 0.3.5 +version: 0.3.6 appVersion: "3.9-21984e88" diff --git a/sdx/chart/sdx-edge/templates/NOTES.txt b/sdx/chart/sdx-edge/templates/NOTES.txt new file mode 100644 index 0000000..3e827c4 --- /dev/null +++ b/sdx/chart/sdx-edge/templates/NOTES.txt @@ -0,0 +1,9 @@ +{{- if and .Values.rotation .Values.rotation.promote }} +rotation.promote is true. After the promote Job succeeds, reset it so later +--reuse-values upgrades do not copy {{ include "sdx-edge.fullname" . }}-client-next +over the live TLS secrets and restart Kong: + + helm upgrade {{ .Release.Name }} oci://ghcr.io/bcgov/aps-devops/sdx-edge \ + --reuse-values \ + --set rotation.promote=false +{{- end }} diff --git a/sdx/chart/sdx-edge/templates/job-cert-bootstrap.yaml b/sdx/chart/sdx-edge/templates/job-cert-bootstrap.yaml index e7d9101..6694638 100644 --- a/sdx/chart/sdx-edge/templates/job-cert-bootstrap.yaml +++ b/sdx/chart/sdx-edge/templates/job-cert-bootstrap.yaml @@ -1,6 +1,11 @@ # Cert Bootstrap Job # - This job is used to register a new TLS certificate for Kong # - It runs a container that executes the certificate sign script +# - Rendered only while bootstrap.tls.token is set. Clear the token on promote +# with --set-string bootstrap.tls.token="" (not null; see README) so Helm +# deletes this Job instead of changing its pod template (stageSecret). Job +# specs are immutable; a consumed one-time token must not recreate the Job +# after TTL either. {{ if .Values.bootstrap.tls.token }} apiVersion: batch/v1 kind: Job @@ -63,6 +68,13 @@ spec: curl -sLO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl +{{- if .Values.bootstrap.stageSecret | default false }} + kubectl create secret \ + --save-config --dry-run=client -o yaml \ + tls ${EDGE_NAME}-client-next \ + --cert=./tls.crt \ + --key=./tls.key | kubectl apply -f - +{{- else }} kubectl delete secret ${EDGE_NAME}-client || true kubectl delete secret ${EDGE_NAME}-server || true @@ -77,6 +89,7 @@ spec: tls ${EDGE_NAME}-server \ --cert=./tls.crt \ --key=./tls.key | kubectl apply -f - +{{- end }} {{- if .Values.shared.ca_secret }} @@ -85,7 +98,7 @@ spec: generic sdx-edge-ca \ --from-file=ca.crt=./roots.pem | kubectl apply -f - {{- end }} -{{- if .Values.kong.enabled }} +{{- if and .Values.kong.enabled (not (.Values.bootstrap.stageSecret | default false)) (not (.Values.bootstrap.deferRestart | default false)) }} kubectl rollout restart deployment $EDGE_NAME {{- end }} diff --git a/sdx/chart/sdx-edge/templates/job-cert-promote.yaml b/sdx/chart/sdx-edge/templates/job-cert-promote.yaml new file mode 100644 index 0000000..82074a3 --- /dev/null +++ b/sdx/chart/sdx-edge/templates/job-cert-promote.yaml @@ -0,0 +1,96 @@ +{{ if and .Values.rotation .Values.rotation.promote }} +# Promote a staged runtime-group TLS secret ({release}-client-next) to the live +# client and server secrets, then rolling-restart Kong so trust-sign mounts the +# new private key. Run only after sdx-keys.r1 operation=rotate has published +# the matching public key. +# +# One-shot: Helm stores rotation.promote in release values. Reset it to false +# after this hook succeeds; otherwise the next --reuse-values upgrade renders +# the hook again, overwrites live secrets from client-next, and restarts Kong. +apiVersion: batch/v1 +kind: Job +metadata: + name: "{{ include "sdx-edge.fullname" . }}-promote-{{ .Values.rotation.nonce | default "1" }}" + labels: + app.kubernetes.io/name: sdx-edge + app.kubernetes.io/component: job-cert-promote + app.kubernetes.io/instance: "{{ include "sdx-edge.fullname" . }}" + data-plane: "{{ include "sdx-edge.fullname" . }}" + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + parallelism: 1 + completions: 1 + backoffLimit: 0 + ttlSecondsAfterFinished: 604800 + template: + metadata: + labels: + DataClass: Low + app.kubernetes.io/name: sdx-edge + app.kubernetes.io/component: job-cert-promote + app.kubernetes.io/instance: "{{ include "sdx-edge.fullname" . }}" + data-plane: "{{ include "sdx-edge.fullname" . }}" + spec: + restartPolicy: Never + serviceAccountName: sdx-job-cert-bootstrap + schedulerName: default-scheduler + enableServiceLinks: true + terminationGracePeriodSeconds: 30 + containers: + - name: cert-promote + image: 'docker.io/smallstep/step-cli:0.23.0' + imagePullPolicy: IfNotPresent + command: + - /bin/bash + - +x + - -e + - -c + - |- + cd /tmp + export PATH=$PATH:/tmp + + curl -sLO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" + chmod +x kubectl + + kubectl get secret ${EDGE_NAME}-client-next -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt + kubectl get secret ${EDGE_NAME}-client-next -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key + + kubectl create secret \ + --save-config --dry-run=client -o yaml \ + tls ${EDGE_NAME}-client \ + --cert=./tls.crt \ + --key=./tls.key | kubectl apply -f - + + kubectl create secret \ + --save-config --dry-run=client -o yaml \ + tls ${EDGE_NAME}-server \ + --cert=./tls.crt \ + --key=./tls.key | kubectl apply -f - + +{{- if .Values.kong.enabled }} + kubectl rollout restart deployment $EDGE_NAME +{{- end }} + env: + - name: EDGE_NAME + value: "{{ include "sdx-edge.fullname" . }}" + {{- if .Values.https_proxy }} + - name: HTTPS_PROXY + value: "{{ .Values.https_proxy }}" + - name: NO_PROXY + value: ".cluster.local,.svc,10.91.0.0/16,10.93.0.0/16,172.30.0.0/16,127.0.0.1,localhost,.gov.bc.ca" + {{- end }} + volumeMounts: + - name: working-dir + mountPath: /tmp + - name: kube-cache + mountPath: .kube + automountServiceAccountToken: true + volumes: + - name: working-dir + emptyDir: {} + - name: kube-cache + emptyDir: {} + dnsPolicy: ClusterFirst +{{ end }} diff --git a/sdx/chart/sdx-edge/templates/job-cert-renewal.yaml b/sdx/chart/sdx-edge/templates/job-cert-renewal.yaml index fdc3e46..1b68ef5 100644 --- a/sdx/chart/sdx-edge/templates/job-cert-renewal.yaml +++ b/sdx/chart/sdx-edge/templates/job-cert-renewal.yaml @@ -77,7 +77,7 @@ spec: --cert=./tls.crt \ --key=./tls.key | kubectl apply -f - -{{- if .Values.kong.enabled }} +{{- if and .Values.kong.enabled (not (.Values.renewal.deferRestart | default false)) }} kubectl rollout restart deployment $EDGE_NAME {{- end }} {{- if .Values.shared.fluentbit.enabled }} diff --git a/sdx/chart/sdx-edge/values.yaml b/sdx/chart/sdx-edge/values.yaml index 37ee8ad..9e210ff 100644 --- a/sdx/chart/sdx-edge/values.yaml +++ b/sdx/chart/sdx-edge/values.yaml @@ -20,12 +20,35 @@ kong: bootstrap: tls: + # One-time CA token. Clear on promote with an empty string + # (`--set-string bootstrap.tls.token=""`), not null: Helm 3 --reuse-values + # coalescing drops null and restores the previous token (see README). token: "" cn: "example.com" # Added as a SAN to the edge certificate ip: "" + # Write {release}-client-next instead of live client/server secrets and skip + # Kong restart. Use after generating a new key/CSR so the public key can be + # published (operation=rotate) before the private key is mounted. + stageSecret: false + # Skip Kong restart after writing live bootstrap secrets. + deferRestart: false + +renewal: + # Certificate renewal keeps the same private key. Restart afterwards so Kong + # picks up the new certificate; set true only when restart is handled separately. + deferRestart: false + +rotation: + # One-shot: copy {release}-client-next onto live client and server TLS + # secrets, then rolling-restart Kong. Helm persists this across + # --reuse-values, so set false in a follow-up upgrade after the Job succeeds + # (see README). Bump nonce so each intentional promote gets a new Job name. + promote: false + nonce: "1" + environment: prod sdx_control_url_by_env: