feat(eventing): Phase 2 — make event signing deployable on a cluster

eventing/agentdocs/DESIGN_PHASE1.md

@@ -691,9 +691,15 @@ checks 8–9 exist before anything is deployed.
 | 10 | **Topic name collisions**: no pre-existing `KafkaTopic` or broker topic named `<ns>-requests` / `<ns>-responses` that this deploy does not own | another user took the name | resolves RQ-5; prevents consuming someone else's data |
 | 11 | Namespace exists or is creatable; quota admits the pods | quota exhausted | otherwise pods sit `Pending` with an unhelpful message |
 | 12 | Default StorageClass exists, if the overlay requests a PVC | no provisioner | PVC stays `Pending` forever |
+| 13 | **The claude CLI image is anonymously pullable and carries the CLI** (demo overlay only; skipped otherwise) | image missing, or built where the bundled `bun` cannot execute | T2.4's gate. Without it every real run fails with `claude binary not found` — and mock mode is exactly what stops the e2e from catching that. Runs on a cluster node rather than at build time because the CLI's `bun` aborts under QEMU user-mode emulation, so a cross-built image cannot execute it on an arm64 builder |
+
+Check 13 was added with the demo overlay and this table documented 12 for a while, which
+`IMPLEMENTATION_REPORT1.md` and `README_PHASE1.md` both reference by number — so the
+count is worth keeping right rather than discovering later that the design documents 12
+of 13.
 
 Check 9 necessarily runs *after* EventBridge is deployed, so preflight has two
-phases: checks 1–8 and 10–12 run before anything is applied, and check 9 runs
+phases: checks 1–8 and 10–13 run before anything is applied, and check 9 runs
 as the gate between deployment step 4 and step 5 (§13).
 
 **On a Kind cluster (§3.2)** the same checks apply, with three differences that are
@@ -714,7 +720,7 @@ implementation.
 these in order. The manual equivalents are shown because they are what you
 will type when debugging.
 
-**0. Preflight** — §12 checks 1–8, 10–12. Abort on any failure.
+**0. Preflight** — §12 checks 1–8, 10–13. Abort on any failure.
 
 **1. Namespace.**
 

eventing/k8s/base/eventbridge-deployment.yaml

@@ -67,6 +67,15 @@ spec:
           volumeMounts:
             - name: data
               mountPath: /data
+            # §11. Appended, not inserted: the demo overlay JSON-patches
+            # `/spec/template/spec/volumes/0` to swap `data` for a PVC, so anything
+            # placed before `data` would be replaced instead of it.
+            - name: signing-key
+              mountPath: /etc/ce-signing
+              readOnly: true
+            - name: keyset
+              mountPath: /etc/ce-keyset
+              readOnly: true
           readinessProbe:
             httpGet:
               path: /healthz
@@ -91,3 +100,29 @@ spec:
       volumes:
         - name: data
           emptyDir: {}
+        # §11 — the seed EventBridge signs requests and group lifecycle events with.
+        # NOT declared in any manifest: `k8s_deploy.py --signing-keys` applies the
+        # Secret via stdin, because a kustomize secretGenerator would inline the seed
+        # into `kubectl kustomize` output and `--from-literal` would leak it to argv.
+        #
+        # `optional: true` keeps signing-less overlays working — without it the pod
+        # blocks in ContainerCreating waiting for a Secret that will never exist. No
+        # `defaultMode`: 0400 would be root-only, and with no `runAsUser`/`fsGroup`
+        # (OpenShift assigns the UID) the default 0644 is what makes it readable.
+        # A Secret PER SERVICE, not one shared between them. A single seed would mean
+        # EventBridge holds the key it verifies runner responses with, so it could
+        # forge any agent's answer and an agent could forge another's — which is the
+        # exact property that made HMAC unacceptable (DESIGN_PHASE2.md §4.3). Two
+        # Secrets is what keeps "only this agent could have produced this" true.
+        - name: signing-key
+          secret:
+            secretName: eventbridge-signing-key
+            optional: true
+        # The approved-key set — `kid` -> public key. A ConfigMap, never a Secret: a
+        # 32-byte seed is indistinguishable from a public key by length, so keeping
+        # the set somewhere a Secret is not is what makes a pasted private key
+        # obvious in review (shared/keyset.py).
+        - name: keyset
+          configMap:
+            name: eventing-keyset
+            optional: true

eventing/k8s/base/eventrunner-deployment.yaml

@@ -48,6 +48,15 @@ spec:
           volumeMounts:
             - name: data
               mountPath: /data
+            # §11. Appended, not inserted: the demo overlay JSON-patches
+            # `/spec/template/spec/volumes/0` to swap `data` for a PVC, so anything
+            # placed before `data` would be replaced instead of it.
+            - name: signing-key
+              mountPath: /etc/ce-signing
+              readOnly: true
+            - name: keyset
+              mountPath: /etc/ce-keyset
+              readOnly: true
           # §8.4 — the failure this catches is a LIVE process with a DEAD thread.
           # If the Kafka consumer thread raises, the main thread keeps running, so
           # the pod stays Running and reports healthy while consuming nothing. A
@@ -73,3 +82,28 @@ spec:
       volumes:
         - name: data
           emptyDir: {}
+        # §11 — the Ed25519 seed this runner signs terminal responses with. The
+        # Secret is NOT declared in any manifest: `k8s_deploy.py --signing-keys`
+        # applies it via stdin, because a kustomize secretGenerator would inline the
+        # seed into `kubectl kustomize` output (which the §14.1 digest hashes and
+        # people paste into terminals) and `--from-literal` would put it in argv.
+        #
+        # `optional: true` is what keeps every signing-less overlay working: without
+        # it the pod blocks in ContainerCreating until a Secret that will never exist
+        # appears. No `defaultMode` on purpose — 0400 would be root-only, and there is
+        # no `runAsUser`/`fsGroup` here (OpenShift assigns the UID), so the default
+        # 0644 is what makes the file readable by whatever UID the SCC picks.
+        # A Secret PER SERVICE: see the matching note in eventbridge-deployment.yaml.
+        # If both mounted one seed, EventBridge would hold the key it verifies this
+        # runner's responses with and could forge them — the HMAC property §4.3 rejected.
+        - name: signing-key
+          secret:
+            secretName: eventrunner-signing-key
+            optional: true
+        # The approved-key set this runner verifies incoming requests against. A
+        # ConfigMap, never a Secret — see shared/keyset.py on why the location is
+        # what keeps a pasted private key visible in review.
+        - name: keyset
+          configMap:
+            name: eventing-keyset
+            optional: true

eventing/k8s/base/keyset-configmap.yaml

@@ -0,0 +1,28 @@
+# §11 — the approved-agent key set. THIS FILE IS THE AUTHORIZATION LIST: an event
+# whose `kid` is not in it is refused, and so is one carrying no signature at all.
+# Approving an agent means adding its public key; revoking it means removing the entry.
+#
+# Committed EMPTY, and a test asserts it stays that way. Two reasons:
+#
+#   * A non-empty committed keyset would mean someone pasted a key into the repository.
+#     Public keys are not secrets, but a 32-byte seed is indistinguishable from a public
+#     key by length (shared/keyset.py), so "there is never a key in this file" is a rule
+#     that can be checked, where "only public keys in this file" is not.
+#   * `{}` is a valid keyset that approves nobody, and `keyset.load_if_set` treats an
+#     empty PATH as "not enabled" — so an operator who mounts this without populating it
+#     gets a clean "no approved kids" at startup rather than a confusing partial state.
+#
+# Populated at deploy time by `k8s_deploy.py --signing-keys DIR`, which replaces the
+# whole ConfigMap from the `agents.json` that `gen_signing_keys.py` derived from the
+# seeds. Unlike the seeds this may safely appear in `kubectl kustomize` output.
+#
+# Rotation is a restart, not a reload: `shared/keyset.py` deliberately has no live
+# reload, because a verifier that silently widens the set it trusts is harder to reason
+# about than one that restarts.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: eventing-keyset
+data:
+  agents.json: |
+    {}

eventing/k8s/base/kustomization.yaml

@@ -5,6 +5,7 @@ kind: Kustomization
 # it targets ns kafka and cannot participate in an overlay's namespace rewrite.
 resources:
   - configmap.yaml
+  - keyset-configmap.yaml
   - eventbridge-deployment.yaml
   - eventbridge-service.yaml
   - eventbridge-route.yaml

eventing/k8s/overlays/kind-signed/kustomization.yaml

@@ -0,0 +1,90 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: kev1
+
+# §11 — the `kind` overlay with signing ENFORCED on both directions. Everything that
+# makes it work (the volumes, the keyset ConfigMap, the config keys) is in the base and
+# defaults to off; this overlay only flips the switches.
+#
+# A separate overlay rather than a patch on `kind`, for two reasons:
+#
+#   * S3 and S5 (#2593/#2595) are explicitly **unsigned** milestones, so the plain
+#     `kind` path has to keep demonstrating the system without signatures — the
+#     before/after contrast is the demo.
+#   * `tests/test_manifests.py` asserts the `test` overlay mounts no credential at all
+#     so mock mode is auto-detected. Keeping enforcement out of the shared overlays
+#     means that assertion stays true and keeps meaning something.
+#
+# The Secret holding the seeds is NOT here and never will be:
+#   python3 scripts/gen_signing_keys.py --out ~/.eventing-keys
+#   python3 scripts/k8s_deploy.py --yes --overlay kind-signed --signing-keys ~/.eventing-keys
+resources:
+  - ../kind
+
+patches:
+  # Enforcement is a ConfigMap merge, not a `kubectl set env`: the values belong in the
+  # manifest so the digest in §14.1 covers them, and so `kubectl kustomize` shows an
+  # auditor exactly what is enforced. Only paths and kids live here — never a seed.
+  - target:
+      kind: ConfigMap
+      name: eventing-config
+    patch: |-
+      - op: replace
+        path: /data/ER_REQUIRE_SIGNATURE
+        value: "true"
+      - op: replace
+        path: /data/EB_REQUIRE_RESPONSE_SIGNATURE
+        value: "true"
+      # Both services verify against the SAME approved set, mounted from the
+      # `eventing-keyset` ConfigMap. One list, so approving an agent is one edit.
+      - op: replace
+        path: /data/ER_VERIFY_KEYSET_PATH
+        value: /etc/ce-keyset/agents.json
+      - op: replace
+        path: /data/EB_VERIFY_KEYSET_PATH
+        value: /etc/ce-keyset/agents.json
+      # The kid each service names in what it signs. EB_SIGNING_KID is also the only
+      # kid accepted on group lifecycle events, so an approved runner cannot forge a
+      # `group.completed` and end a batch early.
+      - op: replace
+        path: /data/ER_SIGNING_KID
+        value: runner-01
+      - op: replace
+        path: /data/EB_SIGNING_KID
+        value: eb-01
+      # Seed paths are NOT here. `configmap.yaml` says they "name Secret mounts and
+      # belong in the Deployment beside the volume", so they are set as container `env`
+      # entries on each Deployment below. An earlier revision of this overlay put them
+      # here and quoted that rule immediately before breaking it — no leak, since the
+      # value is a path, but a rule the repo states and a patch cites should not be a
+      # rule the patch breaks. The §14.1 digest covers the Deployments too, so nothing
+      # is lost by keeping them there.
+
+  # The seed paths, beside the volume that supplies the seed — which is the whole point
+  # of the location rule. `env` rather than the ConfigMap also keeps
+  # `test_manifests.py`'s Secret-vs-ConfigMap assertions able to mean what they say,
+  # including the Phase 3 tuple in #883 that is scoped to seed-path variables.
+  - target:
+      kind: Deployment
+      name: eventbridge
+    patch: |-
+      # `env` does not exist on the base container (only `envFrom`), so this adds the
+      # whole list rather than appending to it. A trailing-slash `env/-` op would fail
+      # with "path does not exist" at render time.
+      - op: add
+        path: /spec/template/spec/containers/0/env
+        value:
+          - name: EB_SIGNING_KEY_PATH
+            value: /etc/ce-signing/seed.hex
+  - target:
+      kind: Deployment
+      name: eventrunner
+    patch: |-
+      # `env` does not exist on the base container (only `envFrom`), so this adds the
+      # whole list rather than appending to it. A trailing-slash `env/-` op would fail
+      # with "path does not exist" at render time.
+      - op: add
+        path: /spec/template/spec/containers/0/env
+        value:
+          - name: ER_SIGNING_KEY_PATH
+            value: /etc/ce-signing/seed.hex