From d34bc81c4ca4944a228caf747a64551b5913e92e Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Wed, 21 Oct 2026 23:27:29 +0200 Subject: [PATCH 1/3] feat(charts,ci): every chart passes Pod Security restricted by default, and CI checks it Phase 3 (chart side) of the "Kates on any cluster" plan, decision D1: chart defaults pass PSS restricted, a CPU and memory requests plus memory limit policy, and an image tag policy, with no separate profile file. Every chart's pods, Helm test pods, hooks and jobs included, get a RuntimeDefault seccomp profile, a non-root user their image runs as, no privilege escalation, every capability dropped, and requests with a memory limit. The Strimzi CR charts write the same settings into the pod templates the operator fills, so the pods pass on a shared operator; strimzi-operator also runs Strimzi's restricted pod security provider. A podLabels value reaches every pod each chart renders (the chart's labels win a clash), and global.imageRegistry and the pull-secret values reach every image and pod, pull secrets as plain names or {name: ...}. The kates and kates-chaos charts take kyvernoPolicy.exceptions, rendered as a PolicyException in the release namespace; their ClusterPolicies render nothing under rbac.scope=namespace. CPU_STRESS and IO_STRESS run a pinned, configurable image (kates.chaos.stress-image, chaos.stressImage) as the target's own user, else 65534, under a restricted security context. scripts/check-chart-policies.sh renders every chart with defaults and with optional workloads on, and runs four sample policies from .github/kyverno/ (restricted, requests and limits, image tag, a team label on every pod) with named exceptions for the host-level node-exporter and Velero node agent and for the Litmus operator, whose subchart takes no security context. It replaces the Kyverno CI job, which matched nothing. Workloads that cannot pass are documented with their switches: node-exporter, Velero's node agent and label-namespace Job, the Litmus operator and exporter, SeaweedFS with the values the chart ships, and its bucket hook. --- .../exceptions/kates-chaos-litmus.yaml | 41 ++++ .../exceptions/monitoring-node-exporter.yaml | 36 +++ .../kyverno/exceptions/velero-node-agent.yaml | 44 ++++ .../kyverno/policies/disallow-latest-tag.yaml | 35 +++ .github/kyverno/policies/pss-restricted.yaml | 20 ++ .../kyverno/policies/require-pod-labels.yaml | 25 +++ .../policies/require-requests-limits.yaml | 34 +++ .github/workflows/ci.yml | 91 ++++---- README.md | 28 +-- charts/apicurio-registry/Chart.yaml | 10 +- charts/apicurio-registry/README.md | 12 + .../apicurio-registry/templates/_helpers.tpl | 9 + .../templates/deployment.yaml | 2 +- .../templates/tests/test-connection.yaml | 33 +-- .../templates/ui-deployment.yaml | 3 +- charts/apicurio-registry/values.schema.json | 18 ++ charts/apicurio-registry/values.yaml | 30 +++ charts/connect-cluster/Chart.yaml | 19 +- charts/connect-cluster/README.md | 5 +- charts/connect-cluster/templates/_helpers.tpl | 8 +- .../templates/preflight-job.yaml | 4 +- .../templates/secret-sync.yaml | 4 +- .../templates/tests/test-01-connect.yaml | 7 +- .../tests/test-03-test-connectors.yaml | 11 +- charts/connect-cluster/values.schema.json | 19 +- charts/connect-cluster/values.yaml | 25 ++- charts/headlamp/Chart.yaml | 9 +- charts/headlamp/README.md | 8 + charts/headlamp/templates/_helpers.tpl | 17 ++ charts/headlamp/templates/deployment.yaml | 7 +- .../templates/tests/test-connection.yaml | 31 ++- charts/headlamp/values.schema.json | 13 +- charts/headlamp/values.yaml | 31 +++ charts/kafka-cluster/Chart.yaml | 22 +- charts/kafka-cluster/README.md | 24 +- charts/kafka-cluster/templates/_helpers.tpl | 6 +- charts/kafka-cluster/templates/kafka.yaml | 28 ++- charts/kafka-cluster/templates/nodepools.yaml | 5 +- charts/kafka-cluster/templates/rbac.yaml | 7 +- .../templates/tests/test-00-profiler.yaml | 5 +- .../templates/tests/test-01-connectivity.yaml | 5 +- .../tests/test-02-produce-consume.yaml | 7 +- .../tests/test-03-authorization.yaml | 5 +- .../templates/tests/test-04-kraft-quorum.yaml | 5 +- .../templates/tests/test-05-topics.yaml | 5 +- .../templates/tests/test-06-listeners.yaml | 5 +- .../templates/tests/test-07-nodepools.yaml | 5 +- .../tests/test-08-cruise-control.yaml | 5 +- .../templates/tests/test-09-metrics.yaml | 5 +- .../templates/tests/test-10-performance.yaml | 7 +- .../tests/test-11-tiered-storage.yaml | 5 +- charts/kafka-cluster/tests/kafka_test.yaml | 8 + charts/kafka-cluster/values.schema.json | 19 ++ charts/kafka-cluster/values.yaml | 47 +++- charts/kafka-common/Chart.yaml | 8 +- charts/kafka-common/README.md | 7 +- charts/kafka-common/templates/_labels.tpl | 15 ++ charts/kafka-common/templates/_secretsync.tpl | 12 +- charts/kafka-common/templates/_values.tpl | 33 +++ charts/kafka-common/templates/_worker.tpl | 16 +- charts/kafka-common/tests/harness/Chart.yaml | 2 +- .../tests/harness/templates/labels.yaml | 3 + .../tests/harness/templates/secretsync.yaml | 2 +- .../tests/harness/tests/labels_test.yaml | 16 ++ .../tests/harness/tests/resources_test.yaml | 15 ++ .../tests/harness/tests/worker_test.yaml | 18 ++ charts/kafka-common/tests/harness/values.yaml | 3 + charts/kafka-ui/Chart.yaml | 8 +- charts/kafka-ui/README.md | 10 +- charts/kafka-ui/templates/_helpers.tpl | 18 +- charts/kafka-ui/templates/deployment.yaml | 2 +- .../templates/tests/test-connection.yaml | 20 +- charts/kafka-ui/values.schema.json | 22 ++ charts/kafka-ui/values.yaml | 40 +++- charts/kates-chaos/Chart.yaml | 24 +- charts/kates-chaos/README.md | 26 ++- charts/kates-chaos/templates/_helpers.tpl | 9 + charts/kates-chaos/templates/experiments.yaml | 11 +- charts/kates-chaos/templates/gameday.yaml | 15 +- .../templates/kyverno-policy-exceptions.yaml | 47 ++++ .../templates/tests/test-connectivity.yaml | 36 ++- charts/kates-chaos/values.schema.json | 25 ++- charts/kates-chaos/values.yaml | 97 +++++++- charts/kates-platform/Chart.yaml | 26 ++- charts/kates-platform/README.md | 6 +- charts/kates-platform/templates/_helpers.tpl | 18 ++ .../templates/tests/test-connection.yaml | 25 +-- charts/kates-platform/values.schema.json | 20 ++ charts/kates-platform/values.yaml | 21 +- charts/kates/Chart.yaml | 18 +- charts/kates/README.md | 19 +- charts/kates/templates/_helpers.tpl | 39 +++- charts/kates/templates/backup-cronjob.yaml | 15 +- charts/kates/templates/cleanup-cronjob.yaml | 20 +- charts/kates/templates/configmap.yaml | 3 + charts/kates/templates/deployment.yaml | 26 +-- .../templates/kyverno-image-verification.yaml | 2 +- .../templates/kyverno-network-policies.yaml | 2 +- charts/kates/templates/kyverno-policies.yaml | 4 +- .../templates/kyverno-policy-exceptions.yaml | 42 +++- charts/kates/templates/migration-job.yaml | 20 +- charts/kates/templates/postgresql.yaml | 8 +- charts/kates/templates/tests/test-api.yaml | 29 ++- charts/kates/templates/tests/test-health.yaml | 29 ++- charts/kates/templates/tests/test-kafka.yaml | 29 ++- charts/kates/values-corporate.yaml | 5 +- charts/kates/values.schema.json | 52 ++++- charts/kates/values.yaml | 95 +++++++- charts/legacy-kafka/Chart.yaml | 8 +- charts/legacy-kafka/README.md | 12 +- charts/legacy-kafka/templates/_helpers.tpl | 12 + charts/legacy-kafka/templates/job-topics.yaml | 3 +- .../templates/statefulset-kafka.yaml | 3 +- .../templates/statefulset-zookeeper.yaml | 3 +- .../templates/tests/test-broker.yaml | 9 +- charts/legacy-kafka/values.schema.json | 13 +- charts/legacy-kafka/values.yaml | 18 ++ charts/mirror-maker2/Chart.yaml | 15 +- charts/mirror-maker2/README.md | 3 +- charts/mirror-maker2/templates/_helpers.tpl | 5 + .../templates/kafka-mirror-maker2.yaml | 2 + .../templates/preflight-job.yaml | 4 +- .../mirror-maker2/templates/secret-sync.yaml | 4 +- .../templates/tests/test-connectors.yaml | 10 +- .../templates/tests/test-failover.yaml | 11 +- .../templates/tests/test-mm2.yaml | 10 +- .../tests/test-offset-translation.yaml | 11 +- .../templates/tests/test-replication.yaml | 11 +- charts/mirror-maker2/values.schema.json | 20 ++ charts/mirror-maker2/values.yaml | 22 ++ charts/monitoring/Chart.yaml | 19 +- charts/monitoring/README.md | 35 +++ .../templates/tests/test-connection.yaml | 26 +-- charts/monitoring/values.schema.json | 21 +- charts/monitoring/values.yaml | 130 +++++++++++ charts/strimzi-operator/Chart.yaml | 14 +- charts/strimzi-operator/README.md | 5 +- .../strimzi-operator/templates/_helpers.tpl | 24 ++ .../templates/crd-upgrade.yaml | 4 +- .../templates/drain-cleaner.yaml | 3 +- .../templates/tests/test-operator.yaml | 16 +- charts/strimzi-operator/values.schema.json | 19 +- charts/strimzi-operator/values.yaml | 28 ++- charts/velero/Chart.yaml | 12 +- charts/velero/README.md | 39 ++++ .../templates/tests/test-connection.yaml | 30 +-- charts/velero/values.schema.json | 16 ++ charts/velero/values.yaml | 122 +++++++--- cli/cmd/mcp_caveats.go | 4 +- cli/cmd/mcp_tools_scenario.go | 2 +- config/velero/velero-values-offline.yaml | 12 + config/velero/velero-values.yaml | 12 + docs/book/07-chaos-practice.md | 4 +- docs/book/17-security.md | 211 ++++++++++++++++-- docs/book/20-installation-guide.md | 32 +-- docs/book/21-kafka-connect.md | 2 +- docs/book/appendix-a-glossary.md | 2 +- docs/book/appendix-b-troubleshooting.md | 2 +- docs/book/appendix-c-cicd.md | 18 +- docs/book/appendix-d-versions.md | 28 +-- docs/book/deploying-strimzi-operator.md | 4 +- .../kates/chaos/KubernetesChaosProvider.java | 50 ++++- .../src/main/resources/application.properties | 5 + .../chaos/KubernetesChaosProviderTest.java | 119 +++++++++- scripts/check-chart-policies.sh | 187 ++++++++++++++++ 165 files changed, 2971 insertions(+), 547 deletions(-) create mode 100644 .github/kyverno/exceptions/kates-chaos-litmus.yaml create mode 100644 .github/kyverno/exceptions/monitoring-node-exporter.yaml create mode 100644 .github/kyverno/exceptions/velero-node-agent.yaml create mode 100644 .github/kyverno/policies/disallow-latest-tag.yaml create mode 100644 .github/kyverno/policies/pss-restricted.yaml create mode 100644 .github/kyverno/policies/require-pod-labels.yaml create mode 100644 .github/kyverno/policies/require-requests-limits.yaml create mode 100644 charts/kates-chaos/templates/kyverno-policy-exceptions.yaml create mode 100755 scripts/check-chart-policies.sh diff --git a/.github/kyverno/exceptions/kates-chaos-litmus.yaml b/.github/kyverno/exceptions/kates-chaos-litmus.yaml new file mode 100644 index 00000000..4ea0686e --- /dev/null +++ b/.github/kyverno/exceptions/kates-chaos-litmus.yaml @@ -0,0 +1,41 @@ +# Why: litmus-core, the upstream LitmusChaos chart kates-chaos wraps, renders +# its chaos operator (and, with litmus-core.exporter.enabled, its exporter) +# with no securityContext and takes none from values, so neither pod can set +# seccomp, runAsNonRoot, allowPrivilegeEscalation or drop capabilities. Exactly +# those four controls are excepted, for those two images only; every other +# control of pss-restricted, and the other policies, still apply to them. +# Removed by: litmus-core.enabled=false on charts/kates-chaos (no chaos +# operator: Kates then cannot run Litmus faults), or by a litmus-core release +# that takes a securityContext. A cluster that enforces the profile exempts +# them with the chart's own kyvernoPolicy.exceptions instead. +# Names the Deployments and their pods as scripts/check-chart-policies.sh +# renders them: release kates-chaos in namespace kates-chaos (the operator's +# name is litmus-core's fullname, litmus, whatever the release is called). +apiVersion: kyverno.io/v2 +kind: PolicyException +metadata: + name: kates-chaos-litmus + namespace: kates-chaos +spec: + exceptions: + - policyName: pss-restricted + ruleNames: [restricted, autogen-restricted] + match: + any: + - resources: + kinds: [Deployment] + namespaces: [kates-chaos] + names: [litmus, litmus-monitor] + - resources: + kinds: [Pod] + namespaces: [kates-chaos] + names: ["litmus-*-?????", "litmus-monitor-*-?????"] + podSecurity: + - controlName: Privilege Escalation + images: ["*litmuschaos/chaos-operator:*", "*litmuschaos/chaos-exporter:*"] + - controlName: Capabilities + images: ["*litmuschaos/chaos-operator:*", "*litmuschaos/chaos-exporter:*"] + - controlName: Running as Non-root + images: ["*litmuschaos/chaos-operator:*", "*litmuschaos/chaos-exporter:*"] + - controlName: Seccomp + images: ["*litmuschaos/chaos-operator:*", "*litmuschaos/chaos-exporter:*"] diff --git a/.github/kyverno/exceptions/monitoring-node-exporter.yaml b/.github/kyverno/exceptions/monitoring-node-exporter.yaml new file mode 100644 index 00000000..76513519 --- /dev/null +++ b/.github/kyverno/exceptions/monitoring-node-exporter.yaml @@ -0,0 +1,36 @@ +# Why: the node exporter reports the node, not itself. It reads the host's +# /proc, /sys and root filesystem through hostPath volumes and joins the host's +# network and PID namespaces, and a pod on the host network gets its container +# port as a host port. No value of the chart makes that pass the restricted +# profile, so exactly those four controls are excepted; every other control of +# pss-restricted, and the other policies, still apply to it. +# Removed by: kube-prometheus-stack.nodeExporter.enabled=false on charts/monitoring +# (the cluster then has no node metrics). +# Names the DaemonSet and its pods as scripts/check-chart-policies.sh renders +# them: release monitoring in namespace monitoring. On a cluster, install it +# where the release runs and edit the names if the release is called otherwise. +apiVersion: kyverno.io/v2 +kind: PolicyException +metadata: + name: monitoring-node-exporter + namespace: monitoring +spec: + exceptions: + - policyName: pss-restricted + ruleNames: [restricted, autogen-restricted] + match: + any: + - resources: + kinds: [DaemonSet] + namespaces: [monitoring] + names: [monitoring-prometheus-node-exporter] + - resources: + kinds: [Pod] + namespaces: [monitoring] + names: ["monitoring-prometheus-node-exporter-?????"] + podSecurity: + - controlName: Host Namespaces + - controlName: HostPath Volumes + - controlName: Volume Types + - controlName: Host Ports + images: ["*prometheus/node-exporter:*"] diff --git a/.github/kyverno/exceptions/velero-node-agent.yaml b/.github/kyverno/exceptions/velero-node-agent.yaml new file mode 100644 index 00000000..c9c7712b --- /dev/null +++ b/.github/kyverno/exceptions/velero-node-agent.yaml @@ -0,0 +1,44 @@ +# Why: the node agent backs pod volumes up from the node itself. It reads them +# through hostPath mounts of /var/lib/kubelet/pods and /var/lib/kubelet/plugins, +# and runs as root (uid 0) with the runtime's default capabilities, which it +# needs to read, and on restore write and chown, files whatever user owns +# them: dropping ALL would leave it a root that cannot. Its container security +# context is empty, as upstream ships it, so it does not forbid privilege +# escalation either (a default of false would clash with privileged: true). +# No value of the chart makes that pass the restricted profile, so exactly +# those controls are excepted; every other control of pss-restricted +# (seccomp, ...), and the other policies, still apply to it. +# Removed by: deployNodeAgent=false on charts/velero, the default (file-system +# backups and the CSI data mover then do not run). +# Names the DaemonSet and its pods as scripts/check-chart-policies.sh renders +# them: release velero in namespace velero. On a cluster, install it where the +# release runs. +apiVersion: kyverno.io/v2 +kind: PolicyException +metadata: + name: velero-node-agent + namespace: velero +spec: + exceptions: + - policyName: pss-restricted + ruleNames: [restricted, autogen-restricted] + match: + any: + - resources: + kinds: [DaemonSet] + namespaces: [velero] + names: [node-agent] + - resources: + kinds: [Pod] + namespaces: [velero] + names: ["node-agent-?????"] + podSecurity: + - controlName: HostPath Volumes + - controlName: Volume Types + - controlName: Running as Non-root + images: ["*velero/velero:*"] + - controlName: Running as Non-root user + - controlName: Capabilities + images: ["*velero/velero:*"] + - controlName: Privilege Escalation + images: ["*velero/velero:*"] diff --git a/.github/kyverno/policies/disallow-latest-tag.yaml b/.github/kyverno/policies/disallow-latest-tag.yaml new file mode 100644 index 00000000..66f51bfd --- /dev/null +++ b/.github/kyverno/policies/disallow-latest-tag.yaml @@ -0,0 +1,35 @@ +# Every image names a tag or a digest, and the tag is not latest. +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: disallow-latest-tag +spec: + validationFailureAction: Enforce + background: true + rules: + - name: require-tag + match: + any: + - resources: + kinds: [Pod] + validate: + message: "Every image needs an explicit tag or digest." + pattern: + spec: + =(initContainers): + - image: "*:* | *@*" + containers: + - image: "*:* | *@*" + - name: no-latest + match: + any: + - resources: + kinds: [Pod] + validate: + message: "The latest tag is not allowed." + pattern: + spec: + =(initContainers): + - image: "!*:latest" + containers: + - image: "!*:latest" diff --git a/.github/kyverno/policies/pss-restricted.yaml b/.github/kyverno/policies/pss-restricted.yaml new file mode 100644 index 00000000..44344c4f --- /dev/null +++ b/.github/kyverno/policies/pss-restricted.yaml @@ -0,0 +1,20 @@ +# The Pod Security Standards "restricted" profile, as Kyverno evaluates it. +# scripts/check-chart-policies.sh holds every chart under charts/ to it; the +# workloads that cannot pass are named one by one in ../exceptions/. +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: pss-restricted +spec: + validationFailureAction: Enforce + background: true + rules: + - name: restricted + match: + any: + - resources: + kinds: [Pod] + validate: + podSecurity: + level: restricted + version: latest diff --git a/.github/kyverno/policies/require-pod-labels.yaml b/.github/kyverno/policies/require-pod-labels.yaml new file mode 100644 index 00000000..57d7f038 --- /dev/null +++ b/.github/kyverno/policies/require-pod-labels.yaml @@ -0,0 +1,25 @@ +# A sample of a cluster's own labelling rule: every Pod carries a team label. +# The charts set no such label; a cluster that requires one sets it through +# each chart's podLabels value (and a wrapped subchart's own label keys), and +# scripts/check-chart-policies.sh sets team=ci through them in every render it +# makes to prove they reach every pod the chart renders, helm tests and jobs +# included. +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: require-pod-labels +spec: + validationFailureAction: Enforce + background: true + rules: + - name: require-team-label + match: + any: + - resources: + kinds: [Pod] + validate: + message: "Every Pod needs the label team (set it through the chart's podLabels value)." + pattern: + metadata: + labels: + team: "?*" diff --git a/.github/kyverno/policies/require-requests-limits.yaml b/.github/kyverno/policies/require-requests-limits.yaml new file mode 100644 index 00000000..984a464d --- /dev/null +++ b/.github/kyverno/policies/require-requests-limits.yaml @@ -0,0 +1,34 @@ +# CPU and memory requests and a memory limit on every container and init +# container. No CPU limit is required: one throttles a pod even when the node +# has CPU to spare. +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: require-requests-limits +spec: + validationFailureAction: Enforce + background: true + rules: + - name: requests-and-memory-limit + match: + any: + - resources: + kinds: [Pod] + validate: + message: "Every container and init container needs CPU and memory requests and a memory limit." + pattern: + spec: + =(initContainers): + - resources: + requests: + memory: "?*" + cpu: "?*" + limits: + memory: "?*" + containers: + - resources: + requests: + memory: "?*" + cpu: "?*" + limits: + memory: "?*" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6db95114..e3aa29f2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,6 +44,8 @@ jobs: helm: - 'charts/**' - 'versions.env' + - '.github/kyverno/**' + - 'scripts/check-chart-policies.sh' - 'scripts/check-versions.sh' # check-versions.sh now also asserts the kind node image tags in # this file against versions.env, so editing it has to run it. @@ -500,57 +502,52 @@ jobs: with: kyverno: "true" - # kates-chaos depends on litmus-core from the LitmusChaos repo. Only the - # Helm Lint job added the repo, so `helm dependency build - # charts/kates-chaos/` in the last step of this job died with "no - # repository definition for https://litmuschaos.github.io/litmus-helm/" - # — every run of this job that got that far was red. + # Every chart with dependencies from a repository. The Helm repositories + # (litmus-core, kube-prometheus-stack, seaweedfs) are added here; + # Strimzi's operator chart is an OCI chart and needs none. Not + # kates-platform: its Chart.lock pins its file:// subcharts' versions and + # is stale after any bump of one, so `helm dependency build` refuses it. + # The script updates it in a copy, after the subcharts, and builds no + # chart this step left current, so nothing is downloaded twice. - name: Fetch chart dependencies uses: ./.github/actions/helm-deps with: - charts: kates kafka-cluster kates-chaos - - - name: Validate kates policies - run: | - helm template kates charts/kates/ \ - --set kyvernoPolicy.enabled=true \ - --api-versions kyverno.io/v1 \ - --show-only templates/kyverno-policies.yaml \ - > /tmp/kates-policies.yaml - echo "--- Rendered kates policies ---" - head -5 /tmp/kates-policies.yaml - # `|| true` used to follow this: the step could not fail, so the - # policies were rendered and never actually applied to anything as - # far as CI could tell. The deployment manifest is meant to pass. - kyverno apply /tmp/kates-policies.yaml \ - --resource kates/k8s/deployment.yaml - - - name: Validate kates workload policies - run: | - helm template kates charts/kates/ \ - --set kyvernoPolicy.enabled=true \ - --api-versions kyverno.io/v1 \ - --show-only templates/kyverno-policies.yaml \ - > /tmp/kates-workload-policies.yaml - echo "✅ Kates workload policies render successfully" - - - name: Validate kafka policies + charts: connect-cluster kafka-cluster kates-chaos mirror-maker2 monitoring strimzi-operator + + # Every chart, with its defaults and with its optional workloads on, held + # to PSS restricted, requests and a memory limit, and pinned images, with + # only the host-level workloads in .github/kyverno/exceptions/ excepted. + - name: Check charts against the cluster policies + run: scripts/check-chart-policies.sh + + # The policies kates, kafka-cluster and kates-chaos ship (kyvernoPolicy), + # applied to the workloads of the same render. They match the release + # namespace only, and Helm installs a template that names no namespace + # into it, so the workloads get it here: before, the kates step applied + # its policies to kates/k8s/deployment.yaml and matched nothing. A run + # where no rule matched anything fails too. yq ships on GitHub's Ubuntu + # runners. + - name: Apply each chart's own policies to its workloads run: | - helm template kafka charts/kafka-cluster/ \ - --set kyvernoPolicy.enabled=true \ - --api-versions kyverno.io/v1 \ - --show-only templates/kyverno-policy.yaml \ - > /tmp/kafka-policies.yaml - echo "✅ Kafka policies render successfully" - - - name: Validate chaos policies - run: | - helm template chaos charts/kates-chaos/ \ - --set kyvernoPolicy.enabled=true \ - --api-versions kyverno.io/v1 \ - --show-only templates/kyverno-policies.yaml \ - > /tmp/chaos-policies.yaml - echo "✅ Chaos policies render successfully" + rc=0 + for chart in kates kafka-cluster kates-chaos; do + echo "==> ${chart}" + out="${RUNNER_TEMP}/${chart}" + if ! helm template "${chart}" "charts/${chart}" -n "${chart}" \ + --set kyvernoPolicy.enabled=true --api-versions kyverno.io/v1 > "${out}.yaml"; then + rc=1; continue + fi + yq 'select(.kind == "ClusterPolicy" or .kind == "Policy")' "${out}.yaml" > "${out}-policies.yaml" + NS="${chart}" yq 'select(.kind == "Pod" or .kind == "Deployment" or .kind == "StatefulSet" or .kind == "DaemonSet" or .kind == "Job" or .kind == "CronJob") + | .metadata.namespace |= (. // strenv(NS))' "${out}.yaml" > "${out}-workloads.yaml" + kyverno apply "${out}-policies.yaml" --resource "${out}-workloads.yaml" \ + -o "${out}-mutated.yaml" > "${out}.log" 2>&1 || rc=1 + cat "${out}.log" + if grep -q '^pass: 0, fail: 0,' "${out}.log"; then + echo "::error::${chart}: its own policies matched none of its workloads"; rc=1 + fi + done + exit ${rc} yaml-validation: name: YAML Validation diff --git a/README.md b/README.md index b03e2640..0e54ea99 100644 --- a/README.md +++ b/README.md @@ -205,20 +205,20 @@ When you install charts with Helm directly: | Chart | Version | App Version | Description | |:------|:--------|:------------|:------------| -| [`apicurio-registry`](charts/apicurio-registry/) | 0.4.0 | 3.3.0 | Apicurio Registry, backed by the kafka-cluster chart (Strimzi) in this repository | -| [`connect-cluster`](charts/connect-cluster/) | 2.1.9 | 4.3.1 | A Helm chart for deploying Strimzi Kafka Connect clusters | -| [`headlamp`](charts/headlamp/) | 0.2.0 | 0.40.1 | Headlamp — Kubernetes Dashboard for cluster visualization and management | -| [`kafka-cluster`](charts/kafka-cluster/) | 1.0.11 | 4.3.1 | A Strimzi Kafka cluster in KRaft mode — node pools, topics, users, rebalancing, tiered storage, network policy and observability | -| [`kafka-common`](charts/kafka-common/) | 0.1.0 | | Library chart — the templates the Strimzi charts share (names and labels, rails, Kafka client authentication, the Connect worker spec, KafkaUser, secret sync, NetworkPolicy and monitoring fragments, the Grafana grid, the preflight probe) | -| [`kafka-ui`](charts/kafka-ui/) | 0.3.0 | v1.5.0 | A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication | -| [`kates-chaos`](charts/kates-chaos/) | 2.2.1 | 3.28.0 | Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring | -| [`kates-platform`](charts/kates-platform/) | 0.8.2 | 1.0.0 | Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure | -| [`kates`](charts/kates/) | 0.12.2 | 1.28.0 | Kates — Kafka Advanced Testing & Engineering Suite | -| [`legacy-kafka`](charts/legacy-kafka/) | 0.2.0 | 3.9.1 | A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to migrate FROM — the replication source that the pinned Strimzi release cannot deploy | -| [`mirror-maker2`](charts/mirror-maker2/) | 0.11.7 | 4.3.1 | Strimzi KafkaMirrorMaker2 — cross-cluster replication, DR, and cross-version migration (2.x/3.x → 4.x) | -| [`monitoring`](charts/monitoring/) | 1.6.0 | 82.4.3 | Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration | -| [`strimzi-operator`](charts/strimzi-operator/) | 0.3.7 | 1.2.0 | The Strimzi Kafka Operator — wraps the upstream chart with pinned kates defaults, an owned CRD-upgrade hook, and a strict values schema | -| [`velero`](charts/velero/) | 11.3.3 | 1.17.1 | A Helm chart for velero | +| [`apicurio-registry`](charts/apicurio-registry/) | 0.4.1 | 3.3.0 | Apicurio Registry, backed by the kafka-cluster chart (Strimzi) in this repository | +| [`connect-cluster`](charts/connect-cluster/) | 2.1.10 | 4.3.1 | A Helm chart for deploying Strimzi Kafka Connect clusters | +| [`headlamp`](charts/headlamp/) | 0.2.1 | 0.40.1 | Headlamp — Kubernetes Dashboard for cluster visualization and management | +| [`kafka-cluster`](charts/kafka-cluster/) | 1.0.12 | 4.3.1 | A Strimzi Kafka cluster in KRaft mode — node pools, topics, users, rebalancing, tiered storage, network policy and observability | +| [`kafka-common`](charts/kafka-common/) | 0.1.1 | | Library chart — the templates the Strimzi charts share (names and labels, rails, Kafka client authentication, the Connect worker spec, KafkaUser, secret sync, NetworkPolicy and monitoring fragments, the Grafana grid, the preflight probe) | +| [`kafka-ui`](charts/kafka-ui/) | 0.3.1 | v1.5.0 | A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication | +| [`kates-chaos`](charts/kates-chaos/) | 2.3.0 | 3.28.0 | Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring | +| [`kates-platform`](charts/kates-platform/) | 0.8.3 | 1.0.0 | Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure | +| [`kates`](charts/kates/) | 0.13.0 | 1.28.0 | Kates — Kafka Advanced Testing & Engineering Suite | +| [`legacy-kafka`](charts/legacy-kafka/) | 0.2.1 | 3.9.1 | A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to migrate FROM — the replication source that the pinned Strimzi release cannot deploy | +| [`mirror-maker2`](charts/mirror-maker2/) | 0.11.8 | 4.3.1 | Strimzi KafkaMirrorMaker2 — cross-cluster replication, DR, and cross-version migration (2.x/3.x → 4.x) | +| [`monitoring`](charts/monitoring/) | 1.7.0 | 82.4.3 | Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration | +| [`strimzi-operator`](charts/strimzi-operator/) | 0.4.0 | 1.2.0 | The Strimzi Kafka Operator — wraps the upstream chart with pinned kates defaults, an owned CRD-upgrade hook, and a strict values schema | +| [`velero`](charts/velero/) | 11.3.4 | 1.17.1 | A Helm chart for velero | ## Building from source diff --git a/charts/apicurio-registry/Chart.yaml b/charts/apicurio-registry/Chart.yaml index 178359d7..fb546c0d 100644 --- a/charts/apicurio-registry/Chart.yaml +++ b/charts/apicurio-registry/Chart.yaml @@ -5,4 +5,12 @@ description: A Helm chart for Kubernetes of Apicurio Registry, backed by the icon: https://design.jboss.org/apicurio/logo/logofiles/apicurio_icon_lightbkg_color.svg name: apicurio-registry type: application -version: 0.4.0 +version: 0.4.1 +annotations: + artifacthub.io/changes: | + - kind: fixed + description: "The Helm test pod passes Pod Security Standards restricted with requests and a limit: 10m/32Mi requests, 64Mi limit, a pod security context with seccompProfile RuntimeDefault, all as tests.resources, tests.podSecurityContext and tests.securityContext" + - kind: changed + description: "The Helm test pod pulls its image from image.registry, as the registry and the UI do, so one mirror serves every pod: it is quay.io/curl/curl:8.7.1 (tests.image), running as the image's own uid 100, instead of busybox:1.36 from Docker Hub. It uses imagePullSecrets too. A mirror set through image.registry needs curl/curl:8.7.1 for helm test" + - kind: added + description: "podLabels (default empty) adds labels to the registry, UI and Helm test pods; the chart's own labels win on a clash, so selectors never change" diff --git a/charts/apicurio-registry/README.md b/charts/apicurio-registry/README.md index 3a7eaabd..17a3d2fc 100644 --- a/charts/apicurio-registry/README.md +++ b/charts/apicurio-registry/README.md @@ -54,5 +54,17 @@ global: Production journal topics need `kafkaTopics.replicas: 3` and `minInsyncReplicas: 2` (defaults are sized for single-broker kind clusters). +## Pod security, images and labels + +Every pod the chart renders (registry, UI, Helm test) passes Pod Security +Standards `restricted` with the defaults, with CPU and memory requests and a +memory limit. `image.registry` is the registry of every image, the Helm test's +`tests.image` (`curl/curl:8.7.1`) included, so a mirror needs +`apicurio/apicurio-registry`, `apicurio/apicurio-registry-ui` (with +`ui.enabled`) and `curl/curl`. `imagePullSecrets` reach every pod. +`podLabels` adds labels to every pod; the chart's own labels win on a clash. +The Helm test pod is tuned with `tests.podSecurityContext`, +`tests.securityContext` and `tests.resources`. + Values are validated by `values.schema.json` — unknown keys fail loudly at install time rather than being silently ignored. diff --git a/charts/apicurio-registry/templates/_helpers.tpl b/charts/apicurio-registry/templates/_helpers.tpl index c3630dd8..d9618a06 100644 --- a/charts/apicurio-registry/templates/_helpers.tpl +++ b/charts/apicurio-registry/templates/_helpers.tpl @@ -86,6 +86,15 @@ a separate image in Registry 3.x; it tracks the same appVersion by default. {{- end }} {{- end }} +{{/* +Labels for a pod template: the chart's own labels (pass them as .labels) +plus podLabels. The chart's own win on a clash, so a selector never changes. +Usage: include "apicurio-registry.podLabels" (dict "ctx" . "labels" (include "apicurio-registry.selectorLabels" .)) +*/}} +{{- define "apicurio-registry.podLabels" -}} +{{- toYaml (merge (fromYaml .labels) .ctx.Values.podLabels) }} +{{- end }} + {{/* Browser-facing REST API URL injected into the UI SPA. Falls back to the backend Service (works under port-forward); override ui.registryApiUrl when diff --git a/charts/apicurio-registry/templates/deployment.yaml b/charts/apicurio-registry/templates/deployment.yaml index 49e7b92a..ef658709 100644 --- a/charts/apicurio-registry/templates/deployment.yaml +++ b/charts/apicurio-registry/templates/deployment.yaml @@ -18,7 +18,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} labels: - {{- include "apicurio-registry.selectorLabels" . | nindent 8 }} + {{- include "apicurio-registry.podLabels" (dict "ctx" . "labels" (include "apicurio-registry.selectorLabels" .)) | nindent 8 }} spec: {{- with .Values.imagePullSecrets }} imagePullSecrets: diff --git a/charts/apicurio-registry/templates/tests/test-connection.yaml b/charts/apicurio-registry/templates/tests/test-connection.yaml index 5f986fcc..f0641520 100644 --- a/charts/apicurio-registry/templates/tests/test-connection.yaml +++ b/charts/apicurio-registry/templates/tests/test-connection.yaml @@ -3,26 +3,33 @@ kind: Pod metadata: name: "{{ include "apicurio-registry.fullname" . }}-test-connection" labels: - {{- include "apicurio-registry.labels" . | nindent 4 }} + {{- include "apicurio-registry.podLabels" (dict "ctx" . "labels" (include "apicurio-registry.labels" .)) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.tests.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: registry-api - image: busybox:1.36 - command: ['wget'] + # From image.registry like the registry itself, so one mirror serves both. + image: "{{ .Values.image.registry }}/{{ .Values.tests.image.repository }}:{{ .Values.tests.image.tag }}" + command: ['curl'] # Hits the v3 REST API, not just the HTTP listener: proves the app is # up, storage initialised, and the API surface actually served. - args: ['-q', '-O-', 'http://{{ include "apicurio-registry.fullname" . }}:{{ .Values.service.port }}/apis/registry/v3/system/info'] + args: ['-fsS', '--max-time', '30', 'http://{{ include "apicurio-registry.fullname" . }}:{{ .Values.service.port }}/apis/registry/v3/system/info'] + {{- with .Values.tests.securityContext }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - runAsNonRoot: true - runAsUser: 65534 - capabilities: - drop: - - ALL - seccompProfile: - type: RuntimeDefault + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tests.resources }} + resources: + {{- toYaml . | nindent 8 }} + {{- end }} restartPolicy: Never diff --git a/charts/apicurio-registry/templates/ui-deployment.yaml b/charts/apicurio-registry/templates/ui-deployment.yaml index 8f88f88f..5be79b04 100644 --- a/charts/apicurio-registry/templates/ui-deployment.yaml +++ b/charts/apicurio-registry/templates/ui-deployment.yaml @@ -15,8 +15,7 @@ spec: template: metadata: labels: - {{- include "apicurio-registry.selectorLabels" . | nindent 8 }} - app.kubernetes.io/component: ui + {{- include "apicurio-registry.podLabels" (dict "ctx" . "labels" (printf "%s\napp.kubernetes.io/component: ui" (include "apicurio-registry.selectorLabels" .))) | nindent 8 }} spec: {{- with .Values.imagePullSecrets }} imagePullSecrets: diff --git a/charts/apicurio-registry/values.schema.json b/charts/apicurio-registry/values.schema.json index 3dc82860..c21f3c42 100644 --- a/charts/apicurio-registry/values.schema.json +++ b/charts/apicurio-registry/values.schema.json @@ -30,6 +30,24 @@ } }, "podAnnotations": { "type": "object" }, + "podLabels": { "type": "object", "additionalProperties": { "type": "string" } }, + "tests": { + "type": "object", + "additionalProperties": false, + "properties": { + "image": { + "type": "object", + "additionalProperties": false, + "properties": { + "repository": { "type": "string", "minLength": 1 }, + "tag": { "type": "string", "minLength": 1 } + } + }, + "podSecurityContext": { "type": "object" }, + "securityContext": { "type": "object" }, + "resources": { "type": "object" } + } + }, "podSecurityContext": { "type": "object" }, "securityContext": { "type": "object" }, "service": { diff --git a/charts/apicurio-registry/values.yaml b/charts/apicurio-registry/values.yaml index c9f648c5..6d0a7bf0 100644 --- a/charts/apicurio-registry/values.yaml +++ b/charts/apicurio-registry/values.yaml @@ -36,6 +36,10 @@ serviceAccount: # -- Pod annotations podAnnotations: {} +# -- Extra labels for every pod the chart renders (registry, UI and the Helm +# test). The chart's own labels win on a clash, so a selector never changes. +podLabels: {} + # -- Pod-level security context. The quay.io image runs as UID 185 (Quarkus). podSecurityContext: runAsNonRoot: true @@ -163,6 +167,32 @@ networkPolicy: # -- Extra environment variables appended to the registry container env: [] +# -- Helm test pod (`helm test`): curl against the v3 REST API. Its image comes +# from image.registry like the registry's (quay.io/curl/curl by default), and +# it uses imagePullSecrets. +tests: + image: + repository: curl/curl + tag: "8.7.1" + # -- The curl image's own user (curl_user, uid 100) + podSecurityContext: + runAsNonRoot: true + runAsUser: 100 + seccompProfile: + type: RuntimeDefault + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 64Mi + # -- Web console (separate image in Registry 3.x). Off by default; the chart # deploys the backend only unless ui.enabled is set. ui: diff --git a/charts/connect-cluster/Chart.yaml b/charts/connect-cluster/Chart.yaml index 432437aa..cf365b16 100644 --- a/charts/connect-cluster/Chart.yaml +++ b/charts/connect-cluster/Chart.yaml @@ -2,10 +2,11 @@ apiVersion: v2 name: connect-cluster description: A Helm chart for deploying Strimzi Kafka Connect clusters type: application -# Patch: the Helm-test image moves to kates-tester 1.28.0, the tag the -# v1.28.0 release publishes; every Strimzi chart pins the same one. -# Templates and defaults are otherwise unchanged. -version: 2.1.9 +# Patch: with default values every pod passes Pod Security "restricted" +# (the Helm test pods get requests and limits, the rack init container the +# restricted security context); podLabels reach every pod and +# global.imageRegistry every image. +version: 2.1.10 # Tracks the Kafka line the workers run, as mirror-maker2's does. The plugin # set is in the image; its pin is the kates.io/connect-image annotation. appVersion: "4.3.1" @@ -25,13 +26,21 @@ keywords: icon: https://raw.githubusercontent.com/bmscomp/kates/main/docs/icon.png dependencies: - name: kafka-common - version: 0.1.0 + version: 0.1.1 repository: file://../kafka-common annotations: kates.io/connect-image: "ghcr.io/bmscomp/connect:3.6.2-kafka-4.3.1" artifacthub.io/category: streaming-messaging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: fixed + description: "The Helm test pods had no requests or limits, so a namespace that enforces a requests-and-limits policy refused them. They get tests.resources (10m and 64Mi, a 128Mi limit); the test-connectors pod also gets imagePullSecrets, which only the test-connect pod had" + - kind: changed + description: "With rack awareness (on by default) the operator adds an init container to every worker, and an operator without Strimzi's restricted security provider leaves its security context empty, which Pod Security \"restricted\" refuses. The KafkaConnect's spec.template.initContainer now gets no privilege escalation and no capabilities; templateExtra.initContainer still merges over it. The workers roll once on upgrade" + - kind: added + description: "podLabels: labels added to the workers' pod template, the Helm test pods and the secret-sync and preflight Jobs. The chart's own labels win on a clash, so no selector changes" + - kind: added + description: "global.imageRegistry: a mirror registry put in front of every image the chart names (image, rack.clientRackInitImage, testImages.kubectl, secretSync.image, the preflight image), as charts/kates does. The secret-sync Job also takes imagePullSecrets now" - kind: changed description: "testImages.kubectl (and testImages.kafka where present) move to kates-tester 1.28.0, the tag the v1.28.0 release publishes; every Strimzi chart in the repository pins the same one (scripts/check-versions.sh holds them to the kates chart's appVersion). The image is unchanged since 1.27.0 — nothing under tester/ moved — so this is the pin, not the tooling." - kind: changed diff --git a/charts/connect-cluster/README.md b/charts/connect-cluster/README.md index bac46950..c9636ccc 100644 --- a/charts/connect-cluster/README.md +++ b/charts/connect-cluster/README.md @@ -117,7 +117,7 @@ The workers resolve `${secrets:/:}` with Strimzi's `Kubern |:---|:---|:---| | `image` | a Connect image with the plugins inside (default: `ghcr.io/bmscomp/connect`, Debezium and Aiven connectors) | recommended | | `plugins` | OCI images mounted as volumes at start-up (`spec.plugins`) | no custom image, and every node has the ImageVolume feature (alpha in Kubernetes 1.31, beta from 1.33) | -| `build` | the operator builds and pushes an image (`spec.build`) | a registry the operator can push to | +| `build` | the operator builds and pushes an image (`spec.build`) | a registry the operator can push to, and a namespace that admits the operator's build pod: it runs as root, so Pod Security "restricted" refuses it | `plugins` renders only with `imageVolumes.acknowledged: true` — the chart cannot see a feature gate — and on Kubernetes 1.31 or later. `plugins[].expect` adds connector classes to the Helm test's list. @@ -252,6 +252,9 @@ The test connectors expect the platform's demo PostgreSQL; `values-prod.yaml` tu | `dashboards.*` | on, folder `Kafka` | | | `preflight.*` | off | | | `tests.expectedPlugins` | the image's connectors | | +| `tests.resources` | 10m and 64Mi, a 128Mi limit | the test pods | +| `podLabels` | `{}` | labels on every pod: the workers, the test pods and the Jobs (the chart's own labels win) | +| `global.imageRegistry` | `""` | a mirror registry put in front of every image the chart names | | `imagePullSecrets`, `imagePullPolicy`, `testImages.kubectl`, `clusterDomain`, `extraLabels` | | | 1.x keys still read in 2.x, listed as `DEPRECATED` in NOTES: `podMonitors`, `monitoring.{enabled,interval,…}`, `metricsConfig`, `autoRestart`, `template`, `databaseEgress`, `kafkaUser.secretSync`, `networkPolicy.{kafkaPorts,monitoringNamespace,restApiClients}`, `alerts.thresholds.sourceLagMinutes`, a `connectors` list. diff --git a/charts/connect-cluster/templates/_helpers.tpl b/charts/connect-cluster/templates/_helpers.tpl index 5f2506b1..25a5aeb5 100644 --- a/charts/connect-cluster/templates/_helpers.tpl +++ b/charts/connect-cluster/templates/_helpers.tpl @@ -71,6 +71,11 @@ capabilities: {{- include "kafka-common.imagePullPolicy" .Values.imagePullPolicy }} {{- end }} +{{/* An image this chart names, behind global.imageRegistry. Call with (list $ ). */}} +{{- define "connect-cluster.image" -}} +{{- include "kafka-common.image" (dict "image" (index . 1) "registry" ((index . 0).Values.global).imageRegistry) }} +{{- end }} + {{/* Namespace of the Kafka cluster (default: the release namespace). */}} {{- define "connect-cluster.kafkaNamespace" -}} {{- .Values.kafka.namespace | default (include "connect-cluster.namespace" .) -}} @@ -166,6 +171,7 @@ connectWorker.spec reads (shared with mirror-maker2). "namespace" (include "connect-cluster.namespace" .) "version" .Values.version "image" .Values.image + "imageRegistry" (.Values.global).imageRegistry "replicas" .Values.replicas "minReplicasDefault" 3 "autoscaling" .Values.autoscaling @@ -187,7 +193,7 @@ connectWorker.spec reads (shared with mirror-maker2). "podSelector" (include "connect-cluster.podSelectorLabels" . | fromYaml) "template" (dict "pdb" .Values.podDisruptionBudget - "podMetadata" (dict "annotations" $annotations) + "podMetadata" (dict "labels" .Values.podLabels "annotations" $annotations) "podSecurityContext" $r.podSecurityContext "imagePullSecrets" .Values.imagePullSecrets "priorityClassName" .Values.priorityClassName diff --git a/charts/connect-cluster/templates/preflight-job.yaml b/charts/connect-cluster/templates/preflight-job.yaml index f72c875e..831965c9 100644 --- a/charts/connect-cluster/templates/preflight-job.yaml +++ b/charts/connect-cluster/templates/preflight-job.yaml @@ -1,7 +1,7 @@ {{- if (.Values.preflight).enabled }} {{- $fullname := include "connect-cluster.fullname" . }} {{- $pf := .Values.preflight }} -{{- $image := $pf.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" .Values.strimziVersion .Values.version) }} +{{- $image := include "connect-cluster.image" (list . ($pf.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" .Values.strimziVersion .Values.version))) }} {{- $k := include "connect-cluster.kafka" . | fromYaml }} {{- $auth := $k.authentication | default dict }} {{- $t := $auth.type | default "" }} @@ -38,7 +38,7 @@ spec: tests' egress policy, whose rules are DNS, the API server and the workers' REST port — none of them Kafka, which is the one thing this Job dials. */}} - {{- include "connect-cluster.componentLabels" (dict "ctx" . "component" "preflight") | nindent 8 }} + {{- include "kafka-common.podLabels" (dict "labels" (include "connect-cluster.componentLabels" (dict "ctx" . "component" "preflight")) "podLabels" .Values.podLabels) | nindent 8 }} spec: restartPolicy: Never # Not the chart's ServiceAccount: it does not exist yet when a diff --git a/charts/connect-cluster/templates/secret-sync.yaml b/charts/connect-cluster/templates/secret-sync.yaml index 22c6a337..1ee2b19a 100644 --- a/charts/connect-cluster/templates/secret-sync.yaml +++ b/charts/connect-cluster/templates/secret-sync.yaml @@ -10,8 +10,10 @@ "namespace" (include "connect-cluster.namespace" .) "labels" (include "connect-cluster.componentLabels" (dict "ctx" . "component" "secret-sync")) "secrets" $sync.resolvedSecrets - "image" ($sync.image | default .Values.testImages.kubectl) + "image" (include "connect-cluster.image" (list . ($sync.image | default .Values.testImages.kubectl))) "pullPolicy" .Values.imagePullPolicy + "imagePullSecrets" .Values.imagePullSecrets + "podLabels" .Values.podLabels "timeoutSeconds" $sync.timeoutSeconds "watch" $sync.watch "schedule" $sync.schedule) }} diff --git a/charts/connect-cluster/templates/tests/test-01-connect.yaml b/charts/connect-cluster/templates/tests/test-01-connect.yaml index 907c2809..792366a0 100644 --- a/charts/connect-cluster/templates/tests/test-01-connect.yaml +++ b/charts/connect-cluster/templates/tests/test-01-connect.yaml @@ -7,8 +7,7 @@ metadata: name: {{ $name }}-test-connect namespace: {{ include "connect-cluster.namespace" . }} labels: - {{- include "connect-cluster.componentLabels" (dict "ctx" . "component" "test") | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "connect-cluster.componentLabels" (dict "ctx" . "component" "test")) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "1" @@ -28,7 +27,7 @@ spec: type: RuntimeDefault containers: - name: connect - image: {{ .Values.testImages.kubectl }} + image: {{ include "connect-cluster.image" (list . .Values.testImages.kubectl) }} {{- include "connect-cluster.imagePullPolicy" . | nindent 6 }} command: ["sh", "-c"] args: @@ -124,6 +123,8 @@ spec: echo "" echo "Connect: $PASS passed, $FAIL failed" [ "$FAIL" -eq 0 ] + resources: + {{- toYaml .Values.tests.resources | nindent 8 }} securityContext: {{- include "connect-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/connect-cluster/templates/tests/test-03-test-connectors.yaml b/charts/connect-cluster/templates/tests/test-03-test-connectors.yaml index 45d9fb8c..ccc986c7 100644 --- a/charts/connect-cluster/templates/tests/test-03-test-connectors.yaml +++ b/charts/connect-cluster/templates/tests/test-03-test-connectors.yaml @@ -8,8 +8,7 @@ metadata: name: {{ $name }}-test-connectors namespace: {{ include "connect-cluster.namespace" $ }} labels: - {{- include "connect-cluster.componentLabels" (dict "ctx" $ "component" "test") | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "connect-cluster.componentLabels" (dict "ctx" $ "component" "test")) "podLabels" $.Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "6" @@ -17,6 +16,10 @@ metadata: spec: serviceAccountName: {{ include "connect-cluster.serviceAccountName" $ }} restartPolicy: Never + {{- with $.Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} securityContext: runAsNonRoot: true runAsUser: 65534 @@ -24,7 +27,7 @@ spec: type: RuntimeDefault containers: - name: connectors - image: {{ $.Values.testImages.kubectl }} + image: {{ include "connect-cluster.image" (list $ $.Values.testImages.kubectl) }} {{- include "connect-cluster.imagePullPolicy" $ | nindent 6 }} command: ["sh", "-c"] args: @@ -50,6 +53,8 @@ spec: fi done [ "$FAIL" -eq 0 ] + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "connect-cluster.containerSecurityContext" $ | nindent 8 }} {{- end }} diff --git a/charts/connect-cluster/values.schema.json b/charts/connect-cluster/values.schema.json index accf968d..2cc06368 100644 --- a/charts/connect-cluster/values.schema.json +++ b/charts/connect-cluster/values.schema.json @@ -13,7 +13,13 @@ "description": "The library dependency (no values)" }, "global": { - "type": "object" + "type": "object", + "properties": { + "imageRegistry": { + "type": "string", + "description": "Registry put in front of every image this chart names (empty: as written)" + } + } }, "nameOverride": { "type": "string" @@ -39,6 +45,13 @@ "type": "string" } }, + "podLabels": { + "type": "object", + "description": "Labels added to every pod: the workers, the test pods and the Jobs; the chart labels win on a clash", + "additionalProperties": { + "type": "string" + } + }, "clusterDomain": { "type": "string" }, @@ -1726,6 +1739,10 @@ "items": { "type": "string" } + }, + "resources": { + "type": "object", + "description": "Kubernetes resource requirements of the test pods" } }, "additionalProperties": false diff --git a/charts/connect-cluster/values.yaml b/charts/connect-cluster/values.yaml index 36ac18a0..aad863cb 100644 --- a/charts/connect-cluster/values.yaml +++ b/charts/connect-cluster/values.yaml @@ -28,6 +28,20 @@ productionMode: false # -- Extra labels added to every resource extraLabels: {} +# -- Labels added to every pod: the workers (through the KafkaConnect's pod +# template), the Helm test pods and the secret-sync and preflight Jobs. The +# chart's own labels win on a clash, so no selector changes. +podLabels: {} + +# -- Settings shared with a parent chart +global: + # -- Registry put in front of every image this chart names, for a mirror: + # `image`, rack.clientRackInitImage, testImages.kubectl, secretSync.image and + # the preflight image (`mirror.example.com` gives + # mirror.example.com/ghcr.io/bmscomp/connect:…). Empty: as written. An empty + # `image` or clientRackInitImage is the operator's own, which the operator's + # registry setting redirects. + imageRegistry: "" # -- Kubernetes cluster domain (bootstrap and Schema Registry FQDNs) clusterDomain: cluster.local @@ -182,7 +196,9 @@ plugins: [] imageVolumes: acknowledged: false -# -- spec.build passthrough (output + plugins); rendered when output is set +# -- spec.build passthrough (output + plugins); rendered when output is set. +# The operator's build pod runs as root, so Pod Security "restricted" refuses +# it: prefer `image` in a restricted namespace. build: {} # output: # type: docker @@ -698,3 +714,10 @@ tests: - io.aiven.connect.jdbc.JdbcSourceConnector - io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector - io.aiven.kafka.connect.s3.source.S3SourceConnector + # -- Requests and limits of the test pods (kubectl and a shell) + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi diff --git a/charts/headlamp/Chart.yaml b/charts/headlamp/Chart.yaml index 09e09847..007e02e3 100644 --- a/charts/headlamp/Chart.yaml +++ b/charts/headlamp/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: headlamp description: Headlamp — Kubernetes Dashboard for cluster visualization and management type: application -version: 0.2.0 +version: 0.2.1 appVersion: "0.40.1" keywords: - kubernetes @@ -18,3 +18,10 @@ maintainers: annotations: artifacthub.io/category: monitoring-logging artifacthub.io/license: Apache-2.0 + artifacthub.io/changes: | + - kind: fixed + description: "The Helm test pod passes Pod Security Standards restricted: it sets seccompProfile RuntimeDefault. Its security context and resources, hardcoded before with the same settings, are values now: test.podSecurityContext, test.securityContext and test.resources" + - kind: added + description: "global.imageRegistry (default empty) is prepended to the Headlamp and Helm test images, for clusters that pull from a mirror; the test pod gets imagePullSecrets too" + - kind: changed + description: "podLabels also reach the Helm test pod, and the chart's own labels win on a clash: a podLabels key the chart sets itself, such as app.kubernetes.io/name, is dropped instead of rendered a second time next to the selector label" diff --git a/charts/headlamp/README.md b/charts/headlamp/README.md index 0d3fef9f..9cb63424 100644 --- a/charts/headlamp/README.md +++ b/charts/headlamp/README.md @@ -27,6 +27,14 @@ helm test headlamp -n headlamp | `rbac.rules` | see values | Full read-only rule set (core, apps, batch, networking, RBAC, Strimzi, Litmus, monitoring) | | `service.type` / `service.port` | `ClusterIP` / `80` | Service exposure | | `securityContext` / `containerSecurityContext` | hardened | Non-root, read-only rootfs, all capabilities dropped | +| `global.imageRegistry` | `""` | Registry prepended to every image (Headlamp and the Helm test) | +| `imagePullSecrets` | `[]` | Pull secrets for every pod, the Helm test included | +| `podLabels` | `{}` | Extra labels on every pod (the chart's own labels win on a clash) | +| `test.image` | `curlimages/curl:8.7.1` | Helm test image | +| `test.podSecurityContext` / `test.securityContext` | uid 100, seccomp `RuntimeDefault`, all capabilities dropped | Helm test pod security | +| `test.resources` | `25m`/`32Mi` requests, `100m`/`64Mi` limits | Helm test pod resources | + +Every pod, the Helm test included, passes Pod Security Standards `restricted` with the defaults. The full value reference is `values.yaml`; every key is validated by `values.schema.json`. diff --git a/charts/headlamp/templates/_helpers.tpl b/charts/headlamp/templates/_helpers.tpl index fec860d1..c68467c6 100644 --- a/charts/headlamp/templates/_helpers.tpl +++ b/charts/headlamp/templates/_helpers.tpl @@ -58,3 +58,20 @@ Create the name of the service account to use {{- default "default" .Values.serviceAccount.name }} {{- end }} {{- end }} + +{{/* +Labels for a pod template: the chart's own labels (pass them as .labels) +plus podLabels. The chart's own win on a clash, so a selector never changes. +Usage: include "headlamp.podLabels" (dict "ctx" . "labels" (include "headlamp.selectorLabels" .)) +*/}} +{{- define "headlamp.podLabels" -}} +{{- toYaml (merge (fromYaml .labels) .ctx.Values.podLabels) }} +{{- end }} + +{{/* +An image reference behind global.imageRegistry when that is set. +Usage: include "headlamp.image" (dict "ctx" . "image" "repository:tag") +*/}} +{{- define "headlamp.image" -}} +{{- with .ctx.Values.global.imageRegistry }}{{ . }}/{{ end }}{{ .image }} +{{- end }} diff --git a/charts/headlamp/templates/deployment.yaml b/charts/headlamp/templates/deployment.yaml index 72cf3d52..644570a1 100644 --- a/charts/headlamp/templates/deployment.yaml +++ b/charts/headlamp/templates/deployment.yaml @@ -12,10 +12,7 @@ spec: template: metadata: labels: - {{- include "headlamp.selectorLabels" . | nindent 8 }} - {{- with .Values.podLabels }} - {{- toYaml . | nindent 8 }} - {{- end }} + {{- include "headlamp.podLabels" (dict "ctx" . "labels" (include "headlamp.selectorLabels" .)) | nindent 8 }} {{- with .Values.podAnnotations }} annotations: {{- toYaml . | nindent 8 }} @@ -33,7 +30,7 @@ spec: {{- end }} containers: - name: headlamp - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + image: {{ include "headlamp.image" (dict "ctx" . "image" (printf "%s:%s" .Values.image.repository .Values.image.tag)) | quote }} imagePullPolicy: {{ .Values.image.pullPolicy }} {{- with .Values.containerSecurityContext }} securityContext: diff --git a/charts/headlamp/templates/tests/test-connection.yaml b/charts/headlamp/templates/tests/test-connection.yaml index af11757d..5939e728 100644 --- a/charts/headlamp/templates/tests/test-connection.yaml +++ b/charts/headlamp/templates/tests/test-connection.yaml @@ -3,32 +3,31 @@ kind: Pod metadata: name: {{ include "headlamp.fullname" . }}-test labels: - {{- include "headlamp.labels" . | nindent 4 }} + {{- include "headlamp.podLabels" (dict "ctx" . "labels" (include "headlamp.labels" .)) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.test.podSecurityContext }} securityContext: - runAsNonRoot: true - runAsUser: 100 - fsGroup: 100 + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: connection-test - image: {{ .Values.test.image }} + image: {{ include "headlamp.image" (dict "ctx" . "image" .Values.test.image) | quote }} + {{- with .Values.test.securityContext }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: - - ALL + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with .Values.test.resources }} resources: - requests: - memory: "32Mi" - cpu: "25m" - limits: - memory: "64Mi" - cpu: "100m" + {{- toYaml . | nindent 6 }} + {{- end }} command: - sh - -c diff --git a/charts/headlamp/values.schema.json b/charts/headlamp/values.schema.json index 8c8aa932..c31b9408 100644 --- a/charts/headlamp/values.schema.json +++ b/charts/headlamp/values.schema.json @@ -2,6 +2,12 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { + "global": { + "type": "object", + "properties": { + "imageRegistry": { "type": "string" } + } + }, "replicaCount": { "type": "integer", "minimum": 1 }, "image": { "type": "object", @@ -51,14 +57,17 @@ "containerSecurityContext": { "type": "object" }, "resources": { "type": "object" }, "podAnnotations": { "type": "object" }, - "podLabels": { "type": "object" }, + "podLabels": { "type": "object", "additionalProperties": { "type": "string" } }, "nodeSelector": { "type": "object" }, "tolerations": { "type": "array" }, "affinity": { "type": "object" }, "test": { "type": "object", "properties": { - "image": { "type": "string" } + "image": { "type": "string", "minLength": 1 }, + "podSecurityContext": { "type": "object" }, + "securityContext": { "type": "object" }, + "resources": { "type": "object" } } } } diff --git a/charts/headlamp/values.yaml b/charts/headlamp/values.yaml index 3f8a1f73..2c0ced60 100644 --- a/charts/headlamp/values.yaml +++ b/charts/headlamp/values.yaml @@ -1,3 +1,10 @@ +global: + # -- Registry prepended to every image the chart renders (Headlamp and the + # Helm test), for clusters that pull from a mirror: + # ghcr.io/headlamp-k8s/headlamp becomes /ghcr.io/headlamp-k8s/headlamp. + # Empty pulls as written. + imageRegistry: "" + replicaCount: 1 image: @@ -149,10 +156,34 @@ networkPolicy: port: 6443 podAnnotations: {} +# -- Extra labels for every pod the chart renders (Headlamp and the Helm test). +# The chart's own labels win on a clash, so a selector never changes. podLabels: {} nodeSelector: {} tolerations: [] affinity: {} +# -- Helm test pod (`helm test`). It uses imagePullSecrets, and +# global.imageRegistry is prepended to its image. test: image: curlimages/curl:8.7.1 + # -- The curl image's own user (curl_user, uid 100) + podSecurityContext: + runAsNonRoot: true + runAsUser: 100 + fsGroup: 100 + seccompProfile: + type: RuntimeDefault + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + memory: "32Mi" + cpu: "25m" + limits: + memory: "64Mi" + cpu: "100m" diff --git a/charts/kafka-cluster/Chart.yaml b/charts/kafka-cluster/Chart.yaml index 73b7ba93..12fd8f7b 100644 --- a/charts/kafka-cluster/Chart.yaml +++ b/charts/kafka-cluster/Chart.yaml @@ -2,10 +2,12 @@ apiVersion: v2 name: kafka-cluster description: A Strimzi Kafka cluster in KRaft mode — node pools, topics, users, rebalancing, tiered storage, network policy and observability type: application -# Patch: the Helm-test images move to kates-tester 1.28.0, the tag the -# v1.28.0 release publishes; every Strimzi chart pins the same one. -# Templates and defaults are otherwise unchanged. -version: 1.0.11 +# Patch: with default values every pod passes Pod Security "restricted" — +# the Helm test pods get requests and limits, the operator's pods get the +# restricted security contexts through their CR templates — podLabels reach +# every pod, and global.imagePullSecrets also takes plain names. SeaweedFS +# (off by default) is unchanged and cannot pass restricted. +version: 1.0.12 appVersion: "4.3.1" icon: https://strimzi.io/assets/images/strimzi_logo.png home: https://strimzi.io/ @@ -21,7 +23,7 @@ sources: - https://github.com/strimzi/strimzi-kafka-operator dependencies: - name: kafka-common - version: 0.1.0 + version: 0.1.1 repository: file://../kafka-common - name: seaweedfs version: "3.68.0" @@ -39,6 +41,16 @@ annotations: # newer; never lower it without removing the feature that set it. kates.io/kafka-floor: "4.2.0" artifacthub.io/changes: | + - kind: fixed + description: "global.imagePullSecrets takes plain names ([regcred]) as well as references ([{name: regcred}]). A plain name rendered an invalid ServiceAccount, so no single list served both this chart and a subchart that reads the same global as names only, as SeaweedFS does (it rendered [{name: regcred}] as a secret named map[name:regcred])" + - kind: changed + description: "With default values the operator's pods pass Pod Security \"restricted\" on an operator that is not configured with Strimzi's restricted security provider: the entity operator, Cruise Control and the Kafka Exporter get runAsNonRoot and a RuntimeDefault seccomp profile on the pod and no privilege escalation and no capabilities on every container, and the init container that rack awareness adds (on by default) gets the same container context. entityOperator.template, cruiseControl.template and kafkaExporter.template are deep-merged over these, as kafka.template already was. The pod templates change, so on upgrade the operator rolls these pods once, brokers and controllers included (one at a time, as for any template change)" + - kind: fixed + description: "The Helm test pods had no requests or limits, and test-produce-consume and test-performance kept every capability, so a namespace that enforces Pod Security \"restricted\" or a requests-and-limits policy refused them. They get tests.resources (kubectl pods: 10m and 64Mi, a 128Mi limit) or tests.jvmResources (Kafka CLI pods: 100m and 384Mi, a 1Gi limit), and drop every capability" + - kind: added + description: "podLabels: labels added to every Helm test pod and to the pod template of every Strimzi component (node pools, entity operator, Cruise Control, Kafka Exporter). The chart's own labels win on a clash, so no selector changes" + - kind: added + description: "tests.imageRegistry: a mirror registry put in front of testImages.kafka and testImages.kubectl. global.imageRegistry keeps its meaning: the registry of the Strimzi client image used when testImages.kafka is empty" - kind: changed description: "testImages.kubectl (and testImages.kafka where present) move to kates-tester 1.28.0, the tag the v1.28.0 release publishes; every Strimzi chart in the repository pins the same one (scripts/check-versions.sh holds them to the kates chart's appVersion). The image is unchanged since 1.27.0 — nothing under tester/ moved — so this is the pin, not the tooling." - kind: changed diff --git a/charts/kafka-cluster/README.md b/charts/kafka-cluster/README.md index 9cca236a..059a944d 100644 --- a/charts/kafka-cluster/README.md +++ b/charts/kafka-cluster/README.md @@ -114,12 +114,12 @@ Each pool is `defaults`, then `roleDefaults` for each of its roles, then the poo | `kafka.quotas` | `{}` | `spec.kafka.quotas`; prod sets `type: strimzi` with `minAvailableRatioPerVolume: 0.1` | | `kafka.rack.topologyKey` | `topology.kubernetes.io/zone` | Rack awareness | | `kafka.logging`, `jmxOptions`, `livenessProbe`, `readinessProbe` | `{}` | Passed through | -| `kafka.template` | PDB `maxUnavailable: 1` | Merged over a hardened `kafkaContainer` security context | +| `kafka.template` | PDB `maxUnavailable: 1` | Merged over a hardened `kafkaContainer` security context and the restricted context on the init container | | `kafka.clusterCa`, `kafka.clientsCa` | generated, 5 years, renew 180 days before | | | `maintenanceTimeWindows` | `[]` | Cron windows for certificate renewal | -| `entityOperator.{topicOperator,userOperator,template}` | sized for production | | -| `kafkaExporter.*` | enabled, all topics and groups | | -| `cruiseControl.*` | enabled, capacity, `autoRebalance` add/remove | `apiUsers`, `template`, `config`, `resources` pass through; Cruise Control keeps the JMX exporter and its own rule set | +| `entityOperator.{topicOperator,userOperator,template}` | sized for production | `template` is merged over the restricted pod and container security contexts | +| `kafkaExporter.*` | enabled, all topics and groups | `template` as for the entity operator | +| `cruiseControl.*` | enabled, capacity, `autoRebalance` add/remove | `apiUsers`, `config`, `resources` pass through, `template` as for the entity operator; Cruise Control keeps the JMX exporter and its own rule set | ## Topics and users @@ -351,7 +351,7 @@ The chart refuses, with the reason and the fix: ## Helm tests -`helm test ` runs, in order: a profiler, connectivity (CR readiness, pods, DNS, listeners), produce/consume, authorization, the KRaft quorum, topics, listeners and TLS, node pools, Cruise Control, metrics, a producer benchmark, and — with tiered storage — offload and remote reads. They authenticate as `-helm-test` (SCRAM, `helm-test-*` topics and groups only); `tests.user.create: false` uses the first managed user instead. The test pods use the chart's `-admin` ServiceAccount and the `-test-egress` NetworkPolicy. +`helm test ` runs, in order: a profiler, connectivity (CR readiness, pods, DNS, listeners), produce/consume, authorization, the KRaft quorum, topics, listeners and TLS, node pools, Cruise Control, metrics, a producer benchmark, and — with tiered storage — offload and remote reads. They authenticate as `-helm-test` (SCRAM, `helm-test-*` topics and groups only); `tests.user.create: false` uses the first managed user instead. The test pods use the chart's `-admin` ServiceAccount (and its `global.imagePullSecrets`) and the `-test-egress` NetworkPolicy; their requests, limits and mirror registry are in [Pod Security and pod labels](#pod-security-and-pod-labels). ## Policy and access @@ -387,3 +387,17 @@ mTLS on 9093 uses the user's certificate Secret and the cluster CA (`krafter-clu - **`helm dependency build` fails.** A `Chart.lock` from 0.4 lists only seaweedfs; run `helm dependency update charts/kafka-cluster`. - **Brokers rolled on upgrade to 1.0.** Expected unless `compatibility.legacyResourceNames` was set: the metrics ConfigMap's name is part of the Kafka CR. Strimzi rolls one broker at a time. The new exporter rules take effect as each broker restarts. +## Pod Security and pod labels + +With default values every pod passes Pod Security Standards "restricted" and carries CPU and memory requests and a memory limit, the operator's pods included. Strimzi sets the restricted security contexts only when its operator runs the restricted security provider, so the chart puts them in each component's CR template: `runAsNonRoot` and a `RuntimeDefault` seccomp profile on the pod, no privilege escalation and no capabilities on every container, the rack-awareness init container included. `kafka.template`, `entityOperator.template`, `cruiseControl.template`, `kafkaExporter.template` and each pool's `template` are deep-merged over them. + +| Key | Default | | +|:---|:---|:---| +| `podLabels` | `{}` | Labels on every Helm test pod and on every component's pod template (pools, entity operator, Cruise Control, Kafka Exporter). The chart's own labels win on a clash, so no selector changes | +| `tests.resources`, `tests.jvmResources` | 10m and 64Mi with a 128Mi limit; 100m and 384Mi with a 1Gi limit | The kubectl test pods, and the Kafka CLI ones | +| `tests.imageRegistry` | `""` | A mirror registry put in front of `testImages.kafka` and `testImages.kubectl`. `global.imageRegistry` stays the registry of the Strimzi client image used when `testImages.kafka` is empty | + +The SeaweedFS subchart (`seaweedfs.enabled`, off by default) cannot pass restricted: its master, volume and filer StatefulSets mount the nodes' hostPath directories for their data (and the master's and filer's logs) and run as root with no requests or limits, as upstream ships them. In a namespace that enforces restricted, keep `seaweedfs.enabled: false` and point [tiered storage](#tiered-storage) at an object store outside the namespace: its endpoint, bucket and region in `tieredStorage.remoteStorageManager.config` (`storage.s3.endpoint.url`, `storage.s3.bucket.name`, `storage.s3.region`), and its credentials in `tieredStorage.credentials.existingSecret`. The post-install bucket Job that `seaweedfs.filer.s3.createBuckets` adds takes neither resources nor pod labels from any value. + +SeaweedFS reads `global.imagePullSecrets` too, but only plain names; this chart takes both `[regcred]` and `[{name: regcred}]`, so give the list as `[regcred]` when SeaweedFS is on. Its pods take labels from `seaweedfs.podLabels` and their image registry from `seaweedfs.image.registry`, not from `podLabels` or `global.imageRegistry`. + diff --git a/charts/kafka-cluster/templates/_helpers.tpl b/charts/kafka-cluster/templates/_helpers.tpl index fbbcabe1..9ec89190 100644 --- a/charts/kafka-cluster/templates/_helpers.tpl +++ b/charts/kafka-cluster/templates/_helpers.tpl @@ -154,17 +154,17 @@ capabilities: drop: ["ALL"] {{- end }} -{{/* Test images. */}} +{{/* Test images; tests.imageRegistry goes in front of the two testImages. */}} {{- define "kafka-cluster.kafkaImage" -}} {{- if .Values.testImages.kafka -}} -{{- .Values.testImages.kafka -}} +{{- include "kafka-common.image" (dict "image" .Values.testImages.kafka "registry" .Values.tests.imageRegistry) -}} {{- else -}} {{- printf "%s/%s/kafka:%s-kafka-%s" .Values.global.imageRegistry .Values.global.imageRepository .Values.strimziVersion .Values.kafkaVersion -}} {{- end -}} {{- end }} {{- define "kafka-cluster.kubectlImage" -}} -{{- .Values.testImages.kubectl -}} +{{- include "kafka-common.image" (dict "image" .Values.testImages.kubectl "registry" .Values.tests.imageRegistry) -}} {{- end }} {{/* ..svc.; call with (dict "service" "namespace" "root"). */}} diff --git a/charts/kafka-cluster/templates/kafka.yaml b/charts/kafka-cluster/templates/kafka.yaml index 3551c7d3..72785fda 100644 --- a/charts/kafka-cluster/templates/kafka.yaml +++ b/charts/kafka-cluster/templates/kafka.yaml @@ -5,10 +5,25 @@ {{- $cc := include "kafka-common.enabled" (list $v.cruiseControl.enabled true) -}} {{- $rebalance := and $cc (include "kafka-common.enabled" (list $v.rebalance.enabled true)) -}} -{{- /* spec.kafka.template: the chart's defaults, then kafka.template. */ -}} -{{- $ktpl := dict "kafkaContainer" (dict "securityContext" (include "kafka-cluster.containerSecurityContext" . | fromYaml)) -}} +{{- /* spec.kafka.template: the chart's defaults, then kafka.template. The + init container (rack awareness, NodePort listeners) is set here rather + than in the node pools, so kafka.template.initContainer still reaches it. */ -}} +{{- $rc := include "kafka-common.restricted.container" . | fromYaml -}} +{{- $ktpl := dict "kafkaContainer" (dict "securityContext" (include "kafka-cluster.containerSecurityContext" . | fromYaml)) "initContainer" (dict "securityContext" (deepCopy $rc)) -}} {{- include "kafka-cluster.merge" (dict "dst" $ktpl "src" ($v.kafka.template | default dict)) -}} +{{- /* The other operands' templates: podLabels and the restricted pod and + container security contexts — which an operator without the restricted + provider leaves empty — then each component's own `template`. */ -}} +{{- $pod := dict "securityContext" (include "kafka-common.restricted.pod" . | fromYaml) -}} +{{- with $v.podLabels }}{{- $_ := set $pod "metadata" (dict "labels" .) }}{{- end -}} +{{- $eoTpl := dict "pod" (deepCopy $pod) "topicOperatorContainer" (dict "securityContext" (deepCopy $rc)) "userOperatorContainer" (dict "securityContext" (deepCopy $rc)) -}} +{{- include "kafka-cluster.merge" (dict "dst" $eoTpl "src" ($v.entityOperator.template | default dict)) -}} +{{- $keTpl := dict "pod" (deepCopy $pod) "container" (dict "securityContext" (deepCopy $rc)) -}} +{{- include "kafka-cluster.merge" (dict "dst" $keTpl "src" ($v.kafkaExporter.template | default dict)) -}} +{{- $ccTpl := dict "pod" (deepCopy $pod) "cruiseControlContainer" (dict "securityContext" (deepCopy $rc)) -}} +{{- include "kafka-cluster.merge" (dict "dst" $ccTpl "src" ($v.cruiseControl.template | default dict)) -}} + {{- /* Tiered storage: the RSM config, with the SeaweedFS endpoint filled in. */ -}} {{- $rsmConfig := dict -}} {{- if $r.tiered -}} @@ -116,14 +131,13 @@ spec: {{- toYaml ($v.entityOperator.topicOperator | default dict) | nindent 6 }} userOperator: {{- toYaml ($v.entityOperator.userOperator | default dict) | nindent 6 }} - {{- with $v.entityOperator.template }} template: - {{- toYaml . | nindent 6 }} - {{- end }} + {{- toYaml $eoTpl | nindent 6 }} {{- if include "kafka-common.enabled" (list $v.kafkaExporter.enabled true) }} kafkaExporter: {{- $ke := omit $v.kafkaExporter "enabled" }} {{- range $k, $val := $ke }}{{- if not $val }}{{- $_ := unset $ke $k }}{{- end }}{{- end }} + {{- $_ := set $ke "template" $keTpl }} {{- toYaml $ke | nindent 4 }} {{- end }} {{- if $cc }} @@ -149,12 +163,14 @@ spec: config: {{- toYaml . | nindent 6 }} {{- end }} - {{- range $key := (list "resources" "apiUsers" "template" "image" "jvmOptions" "logging" "livenessProbe" "readinessProbe") }} + {{- range $key := (list "resources" "apiUsers" "image" "jvmOptions" "logging" "livenessProbe" "readinessProbe") }} {{- with (index $v.cruiseControl $key) }} {{ $key }}: {{- toYaml . | nindent 6 }} {{- end }} {{- end }} + template: + {{- toYaml $ccTpl | nindent 6 }} {{- if $r.metrics.enabled }} {{- /* Cruise Control keeps the JMX exporter and its own rule set. */}} metricsConfig: diff --git a/charts/kafka-cluster/templates/nodepools.yaml b/charts/kafka-cluster/templates/nodepools.yaml index 71c486e8..4b2478fb 100644 --- a/charts/kafka-cluster/templates/nodepools.yaml +++ b/charts/kafka-cluster/templates/nodepools.yaml @@ -20,7 +20,10 @@ rendered here and not on the Kafka CR. {{- $pod := dict -}} {{- $podMeta := dict -}} -{{- with $p.zone }}{{- $_ := set $podMeta "labels" (dict "zone" .) }}{{- end -}} +{{- /* podLabels, under the chart's own zone label. */ -}} +{{- $podLabels := deepCopy ($v.podLabels | default dict) -}} +{{- with $p.zone }}{{- $_ := set $podLabels "zone" . }}{{- end -}} +{{- with $podLabels }}{{- $_ := set $podMeta "labels" . }}{{- end -}} {{- /* With tiered storage active, a broker-only pool's log segments live in object storage, so a file-level backup of its volumes is redundant. diff --git a/charts/kafka-cluster/templates/rbac.yaml b/charts/kafka-cluster/templates/rbac.yaml index e54141d9..53bb06e0 100644 --- a/charts/kafka-cluster/templates/rbac.yaml +++ b/charts/kafka-cluster/templates/rbac.yaml @@ -13,10 +13,13 @@ metadata: {{- end }} {{- /* The chart's own pods (the Helm tests, the backup hooks) run as this account, so its pull secrets are theirs. The operator's pods take - theirs from the Kafka CR's template instead. */}} + theirs from the Kafka CR's template instead. Plain names are taken + too: a subchart reading the same global may take only those. */}} {{- with (.Values.global).imagePullSecrets }} imagePullSecrets: - {{- toYaml . | nindent 2 }} + {{- range . }} + - name: {{ if kindIs "map" . }}{{ .name }}{{ else }}{{ . }}{{ end }} + {{- end }} {{- end }} --- apiVersion: rbac.authorization.k8s.io/v1 diff --git a/charts/kafka-cluster/templates/tests/test-00-profiler.yaml b/charts/kafka-cluster/templates/tests/test-00-profiler.yaml index 9e1b4ba4..800ae103 100644 --- a/charts/kafka-cluster/templates/tests/test-00-profiler.yaml +++ b/charts/kafka-cluster/templates/tests/test-00-profiler.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "0" @@ -55,6 +54,8 @@ spec: echo "========================================" echo "✓ Profiler complete." + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-01-connectivity.yaml b/charts/kafka-cluster/templates/tests/test-01-connectivity.yaml index 9a06964d..cf4bd2f1 100644 --- a/charts/kafka-cluster/templates/tests/test-01-connectivity.yaml +++ b/charts/kafka-cluster/templates/tests/test-01-connectivity.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "1" @@ -165,6 +164,8 @@ spec: echo " Tier 1 Results: $PASS passed, $FAIL failed" echo "───────────────────────────────────" [ "$FAIL" -eq 0 ] + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-02-produce-consume.yaml b/charts/kafka-cluster/templates/tests/test-02-produce-consume.yaml index a47c3972..2758a270 100644 --- a/charts/kafka-cluster/templates/tests/test-02-produce-consume.yaml +++ b/charts/kafka-cluster/templates/tests/test-02-produce-consume.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "2" @@ -153,9 +152,13 @@ spec: echo "Cleaning up test topic..." kubectl delete kafkatopic ${TOPIC} -n $NAMESPACE --ignore-not-found echo "✓ Tier 2 complete" + resources: + {{- toYaml $.Values.tests.jvmResources | nindent 8 }} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: false + capabilities: + drop: ["ALL"] volumeMounts: - name: tmp mountPath: /tmp diff --git a/charts/kafka-cluster/templates/tests/test-03-authorization.yaml b/charts/kafka-cluster/templates/tests/test-03-authorization.yaml index 04aab08d..6bd4b05e 100644 --- a/charts/kafka-cluster/templates/tests/test-03-authorization.yaml +++ b/charts/kafka-cluster/templates/tests/test-03-authorization.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "3" @@ -69,6 +68,8 @@ spec: echo "✗ Some users not ready" && exit 1 fi echo "✓ Tier 3 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-04-kraft-quorum.yaml b/charts/kafka-cluster/templates/tests/test-04-kraft-quorum.yaml index eb5e42ad..9c4928fc 100644 --- a/charts/kafka-cluster/templates/tests/test-04-kraft-quorum.yaml +++ b/charts/kafka-cluster/templates/tests/test-04-kraft-quorum.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "4" @@ -67,6 +66,8 @@ spec: fi echo "✓ Tier 4 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-05-topics.yaml b/charts/kafka-cluster/templates/tests/test-05-topics.yaml index fa57b71a..97e7efec 100644 --- a/charts/kafka-cluster/templates/tests/test-05-topics.yaml +++ b/charts/kafka-cluster/templates/tests/test-05-topics.yaml @@ -9,8 +9,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "5" @@ -75,6 +74,8 @@ spec: echo "✗ Topic validation failed" && exit 1 fi echo "✓ Tier 5 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-06-listeners.yaml b/charts/kafka-cluster/templates/tests/test-06-listeners.yaml index ea7636c9..f4fc4621 100644 --- a/charts/kafka-cluster/templates/tests/test-06-listeners.yaml +++ b/charts/kafka-cluster/templates/tests/test-06-listeners.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "6" @@ -77,6 +76,8 @@ spec: echo "✗ Listener validation failed" && exit 1 fi echo "✓ Tier 6 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-07-nodepools.yaml b/charts/kafka-cluster/templates/tests/test-07-nodepools.yaml index 3e4a76fa..cd3f7ac9 100644 --- a/charts/kafka-cluster/templates/tests/test-07-nodepools.yaml +++ b/charts/kafka-cluster/templates/tests/test-07-nodepools.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "7" @@ -75,6 +74,8 @@ spec: echo "✗ Node pool validation failed" && exit 1 fi echo "✓ Tier 7 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-08-cruise-control.yaml b/charts/kafka-cluster/templates/tests/test-08-cruise-control.yaml index b2dc5796..f2475ba2 100644 --- a/charts/kafka-cluster/templates/tests/test-08-cruise-control.yaml +++ b/charts/kafka-cluster/templates/tests/test-08-cruise-control.yaml @@ -9,8 +9,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "8" @@ -66,6 +65,8 @@ spec: echo " ✗ CruiseControl not configured in Kafka CR" && exit 1 fi echo "✓ Tier 8 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-09-metrics.yaml b/charts/kafka-cluster/templates/tests/test-09-metrics.yaml index 22ef27e5..1a27eaad 100644 --- a/charts/kafka-cluster/templates/tests/test-09-metrics.yaml +++ b/charts/kafka-cluster/templates/tests/test-09-metrics.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "9" @@ -72,6 +71,8 @@ spec: {{- end }} echo "✓ Tier 9 complete" + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} securityContext: {{- include "kafka-cluster.containerSecurityContext" . | nindent 8 }} volumeMounts: diff --git a/charts/kafka-cluster/templates/tests/test-10-performance.yaml b/charts/kafka-cluster/templates/tests/test-10-performance.yaml index 221d8bb7..ff0e2cc6 100644 --- a/charts/kafka-cluster/templates/tests/test-10-performance.yaml +++ b/charts/kafka-cluster/templates/tests/test-10-performance.yaml @@ -8,8 +8,7 @@ metadata: namespace: {{ include "kafka-cluster.namespace" . }} labels: {{- /* strimzi.io/cluster comes from kafka-cluster.labels — do not repeat it */}} - {{- include "kafka-cluster.labels" . | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.labels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "10" @@ -112,9 +111,13 @@ spec: echo "Cleaning up..." kubectl delete kafkatopic ${PERF_TOPIC} -n $NAMESPACE --ignore-not-found echo "✓ Tier 10 complete" + resources: + {{- toYaml $.Values.tests.jvmResources | nindent 8 }} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: false + capabilities: + drop: ["ALL"] volumeMounts: - name: tmp mountPath: /tmp diff --git a/charts/kafka-cluster/templates/tests/test-11-tiered-storage.yaml b/charts/kafka-cluster/templates/tests/test-11-tiered-storage.yaml index 30a1cbdd..df30a3dc 100644 --- a/charts/kafka-cluster/templates/tests/test-11-tiered-storage.yaml +++ b/charts/kafka-cluster/templates/tests/test-11-tiered-storage.yaml @@ -19,8 +19,7 @@ metadata: name: "{{ include "kafka-cluster.clusterName" . }}-test-tiered-storage" namespace: {{ include "kafka-cluster.namespace" . }} labels: - {{- include "kafka-cluster.componentLabels" (dict "ctx" . "component" "helm-test") | nindent 4 }} - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-cluster.componentLabels" (dict "ctx" . "component" "helm-test")) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-weight: "11" @@ -102,6 +101,8 @@ spec: echo " ✗ offset 0 could not be read" && exit 1 fi echo "✓ Tier 11 complete" + resources: + {{- toYaml $.Values.tests.jvmResources | nindent 8 }} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: false diff --git a/charts/kafka-cluster/tests/kafka_test.yaml b/charts/kafka-cluster/tests/kafka_test.yaml index 94f21437..aff2f663 100644 --- a/charts/kafka-cluster/tests/kafka_test.yaml +++ b/charts/kafka-cluster/tests/kafka_test.yaml @@ -46,6 +46,14 @@ tests: - equal: {path: 'spec.kafka.tieredStorage.remoteStorageManager.config["storage.s3.endpoint.url"]', value: http://seaweedfs-filer.NAMESPACE.svc.cluster.local:8333} - equal: {path: 'spec.kafka.tieredStorage.remoteStorageManager.config["chunk.size"]', value: "4194304"} + - it: takes pull secrets as plain names or as references + templates: [templates/rbac.yaml] + set: + global: {imagePullSecrets: [regcred, {name: mirror}]} + asserts: + - equal: {path: imagePullSecrets, value: [{name: regcred}, {name: mirror}]} + documentIndex: 0 + - it: derives topic replication from the broker count templates: [templates/topics.yaml] set: diff --git a/charts/kafka-cluster/values.schema.json b/charts/kafka-cluster/values.schema.json index 5bdf8258..9ccf27cb 100644 --- a/charts/kafka-cluster/values.schema.json +++ b/charts/kafka-cluster/values.schema.json @@ -61,6 +61,13 @@ "type": "string" } }, + "podLabels": { + "type": "object", + "description": "Labels added to every pod the chart renders and to every Strimzi component pod template; the chart labels win on a clash", + "additionalProperties": { + "type": "string" + } + }, "profile": { "type": "string", "enum": [ @@ -2134,6 +2141,18 @@ } }, "additionalProperties": false + }, + "imageRegistry": { + "type": "string", + "description": "Registry put in front of testImages.kafka and testImages.kubectl (empty: as written)" + }, + "resources": { + "type": "object", + "description": "Kubernetes resource requirements of the kubectl test pods" + }, + "jvmResources": { + "type": "object", + "description": "Kubernetes resource requirements of the Kafka CLI test pods" } }, "additionalProperties": false diff --git a/charts/kafka-cluster/values.yaml b/charts/kafka-cluster/values.yaml index ede56eb6..74ac98a9 100644 --- a/charts/kafka-cluster/values.yaml +++ b/charts/kafka-cluster/values.yaml @@ -60,7 +60,7 @@ global: imageRegistry: "quay.io" # -- Repository of that image imageRepository: "strimzi" - # -- Pull secrets for the pods this chart owns (tests) + # -- Pull secrets of the pods this chart owns (tests): names or {name: …} imagePullSecrets: [] # -- Kubernetes DNS domain clusterDomain: "cluster.local" @@ -68,8 +68,8 @@ global: # nodePools (a class change on an existing volume is refused by Strimzi). storageClass: "" -# -- Images of the Helm-test pods. Pinned: scripts/check-versions.sh keeps -# them aligned with the other charts. +# -- Images of the Helm-test pods (tests.imageRegistry mirrors them). Pinned: +# scripts/check-versions.sh keeps them aligned with the other charts. testImages: kafka: "ghcr.io/bmscomp/kates-tester:1.28.0" kubectl: "ghcr.io/bmscomp/kates-tester:1.28.0" @@ -170,8 +170,8 @@ kafka: livenessProbe: {} readinessProbe: {} - # -- spec.kafka.template, deep-merged over the chart's (PDB and a hardened - # kafkaContainer security context). Node pools render their own pod, + # -- spec.kafka.template, deep-merged over the chart's (PDB, hardened kafkaContainer + # and initContainer security contexts). Node pools render their own pod, # kafkaContainer and persistentVolumeClaim templates, which Strimzi uses # instead of these: set per-pool pod settings under nodePools. template: @@ -290,6 +290,12 @@ nodePools: # volumes: # - { id: 0, size: 200Gi, class: gp3-az1 } +# -- Labels added to every pod: the Helm test pods, and the pod template of +# every Strimzi component (brokers and controllers, entity operator, Cruise +# Control, Kafka Exporter), so the operator's pods carry them too. The chart's +# own labels win on a clash, so no selector changes. +podLabels: {} + # ── Operands ───────────────────────────────────────────────────────────────── entityOperator: topicOperator: @@ -315,7 +321,11 @@ entityOperator: jvmOptions: -Xms: 128m -Xmx: 256m - # -- spec.entityOperator.template + # -- spec.entityOperator.template, deep-merged over the chart's: podLabels, + # and the Pod Security "restricted" pod and container security contexts + # (runAsNonRoot, RuntimeDefault seccomp, no privilege escalation, every + # capability dropped), which an operator without the restricted security + # provider does not set template: {} # -- Kafka Exporter (consumer lag and topic/partition metrics) @@ -332,6 +342,8 @@ kafkaExporter: limits: memory: 256Mi cpu: 200m + # -- spec.kafkaExporter.template, deep-merged over the chart's (podLabels and + # the restricted security contexts, as for entityOperator.template) template: {} # -- Cruise Control @@ -356,7 +368,8 @@ cruiseControl: cpu: 500m # -- spec.cruiseControl.apiUsers (hashLoginService Secret) apiUsers: {} - # -- spec.cruiseControl.template + # -- spec.cruiseControl.template, deep-merged over the chart's (podLabels and + # the restricted security contexts, as for entityOperator.template) template: {} # -- KafkaRebalance templates used by cruiseControl.autoRebalance, and an @@ -661,6 +674,26 @@ tests: # helm-test-* topics and groups. Off: the tests use the first managed user. user: create: true + # -- Registry put in front of testImages.kafka and testImages.kubectl, for a + # mirror (`mirror.example.com` gives mirror.example.com/ghcr.io/bmscomp/…). + # Empty: the references as written. The Strimzi client image used when + # testImages.kafka is empty takes global.imageRegistry instead. + imageRegistry: "" + # -- Requests and limits of the test pods that run kubectl and a shell + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi + # -- Requests and limits of the test pods that run the Kafka CLI, a JVM + # (produce-consume, performance, tiered storage) + jvmResources: + requests: + cpu: 100m + memory: 384Mi + limits: + memory: 1Gi # -- external-secrets SecretStore/PushSecret/ExternalSecret wiring externalSecrets: diff --git a/charts/kafka-common/Chart.yaml b/charts/kafka-common/Chart.yaml index 5a61ef37..ef71de0b 100644 --- a/charts/kafka-common/Chart.yaml +++ b/charts/kafka-common/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: kafka-common description: Library chart — the templates the Strimzi charts share (names and labels, rails, Kafka client authentication, the Connect worker spec, KafkaUser, secret sync, NetworkPolicy and monitoring fragments, the Grafana grid, the preflight probe) type: library -version: 0.1.0 +version: 0.1.1 kubeVersion: ">=1.27.0-0" home: https://github.com/bmscomp/kates sources: @@ -19,5 +19,11 @@ annotations: artifacthub.io/category: streaming-messaging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: added + description: "kafka-common.podLabels (the chart's labels over a user's podLabels map, so a selector never changes), kafka-common.image (a mirror registry in front of an image reference, as charts/kates does with global.imageRegistry) and kafka-common.restricted.pod / .restricted.container (the Pod Security Standards restricted contexts Strimzi's restricted provider sets)" + - kind: added + description: "kafka-common.secretSync takes imagePullSecrets and podLabels for the sync Job's pod" + - kind: changed + description: "connectWorker.spec: with rack awareness and a hardened container, the operator's rack init container gets the restricted security context (allowPrivilegeEscalation false, every capability dropped), so the workers pass Pod Security restricted on an operator without the restricted provider. A worker `imageRegistry` goes in front of image and clientRackInitImage" - kind: added description: First release — the templates mirror-maker2 0.8.0, kafka-cluster 1.0.0 and connect-cluster 2.0.0 are built on, lifted from mirror-maker2 0.7 (docs/kafka-charts-refactor-plan.md §3.3), plus the Grafana grid layout and the preflight probe diff --git a/charts/kafka-common/README.md b/charts/kafka-common/README.md index 98616e98..8f93bd4c 100644 --- a/charts/kafka-common/README.md +++ b/charts/kafka-common/README.md @@ -10,7 +10,7 @@ A Helm **library** chart: the templates the Strimzi charts in this repository sh # Chart.yaml dependencies: - name: kafka-common - version: 0.1.0 + version: 0.1.1 repository: file://../kafka-common ``` @@ -24,6 +24,9 @@ Every template takes the calling chart's context or explicit arguments, so names |:---|:---|:---| | `kafka-common.name` · `.fullname` · `.namespace` · `.chart` · `.clusterDomain` | Names. `namespaceOverride` is honoured, and the domain is `global.clusterDomain`, then `clusterDomain`, then `cluster.local`. | These used to be three copies with different defaults. | | `kafka-common.labels` · `.selectorLabels` | Standard labels, built as a map. | A map cannot hold a duplicate key; YAML silently keeps the last one. | +| `kafka-common.podLabels` | The labels of a pod the chart renders: the chart's own over the user's `podLabels`, plus `kates.io/test-pod` for a test pod. | A `podLabels` key can add a label but never move a selector. | +| `kafka-common.image` | `/` when a mirror registry is set, the image unchanged otherwise. | The `global.imageRegistry` convention of `charts/kates`, in one place. | +| `kafka-common.restricted.pod` · `.restricted.container` | The pod and container security contexts Pod Security Standards "restricted" asks for. | Strimzi sets them only when its operator runs the restricted provider; the charts set them in the CR templates so the pods pass on any operator. | | `kafka-common.enabled` | `"true"` or nothing, for `(list )`. | `false \| default true` is `true` in Helm, which is how a switch documented as "set false to disable" stays on. | | `kafka-common.semver` | `4.2-IV1` → `4.2.0`, `2.8` → `2.8.0`. | `semverCompare` rejects the short forms. | | `kafka-common.rails.noFloatingTag` | Refuses `:latest` and untagged images. | What runs should not depend on when a pod last restarted. | @@ -53,7 +56,7 @@ This template covers the whole worker surface: - a PDB only when more than one worker will run; - `nodeSelector` rendered as required node affinity, because Strimzi's pod template has no `nodeSelector` field; - pod anti-affinity and topology spread on the worker selector; - - a hardened container security context that the chart's pass-through can override one field at a time. + - a hardened container security context that the chart's pass-through can override one field at a time, and, with rack awareness, the restricted security context on the operator's init container. The caller's `extra` is **deep-merged** into that map last. mirror-maker2 0.7 wrote its pass-through next to its own keys instead, and lost its entire pod template to a duplicate `pod:` key. diff --git a/charts/kafka-common/templates/_labels.tpl b/charts/kafka-common/templates/_labels.tpl index b74d6c78..3d968d7c 100644 --- a/charts/kafka-common/templates/_labels.tpl +++ b/charts/kafka-common/templates/_labels.tpl @@ -29,3 +29,18 @@ Call with (dict "ctx" $ "component" "connect" "partOf" "kates" "extra" (dict … {{- $l = mustMergeOverwrite $l (deepCopy (.extra | default dict)) (deepCopy ($ctx.Values.extraLabels | default dict)) -}} {{- toYaml $l }} {{- end }} + +{{/* +The labels of a pod the chart renders (a test pod, a Job's pod template): the +chart's own `labels` (YAML, as the helpers above emit it) over the user's +`podLabels` map. A podLabels key adds a label and never replaces one of the +chart's, so no selector changes. `testPod` adds kates.io/test-pod: "true", +which the Helm tests' NetworkPolicies select. + +Call with (dict "labels" "podLabels" "testPod" ). +*/}} +{{- define "kafka-common.podLabels" -}} +{{- $own := .labels | fromYaml -}} +{{- if .testPod }}{{- $_ := set $own "kates.io/test-pod" "true" }}{{- end -}} +{{- toYaml (merge $own (.podLabels | default dict)) }} +{{- end }} diff --git a/charts/kafka-common/templates/_secretsync.tpl b/charts/kafka-common/templates/_secretsync.tpl index 6802f17d..c96dae0f 100644 --- a/charts/kafka-common/templates/_secretsync.tpl +++ b/charts/kafka-common/templates/_secretsync.tpl @@ -12,7 +12,9 @@ read grant, so it is opt-in. Call with (dict "ctx" $ "fullname" … "namespace" "labels" "secrets" (list (dict "name" "fromNamespace" "as")) "image" … -"pullPolicy" … "timeoutSeconds" 300 "watch" false "schedule" ). +"pullPolicy" … "timeoutSeconds" 300 "watch" false "schedule" +"imagePullSecrets" "podLabels" ). podLabels go on the Job's pod +under the chart's labels (kafka-common.podLabels). Entries whose source namespace and name equal the destination are skipped. Renders nothing when no entry is left. */}} @@ -31,7 +33,7 @@ Renders nothing when no entry is left. {{- $names := list }}{{- range $secrets }}{{- $names = append $names .as }}{{- end }} {{- $srcNamespaces := list }}{{- range $secrets }}{{- $srcNamespaces = append $srcNamespaces .fromNamespace }}{{- end }} {{- $srcNamespaces = $srcNamespaces | uniq }} -{{- $args := dict "dstNs" $dstNs "secrets" $secrets "image" .image "pullPolicy" .pullPolicy "timeoutSeconds" (.timeoutSeconds | default 300) "fullname" $fullname "labels" $labels }} +{{- $args := dict "dstNs" $dstNs "secrets" $secrets "image" .image "pullPolicy" .pullPolicy "timeoutSeconds" (.timeoutSeconds | default 300) "fullname" $fullname "labels" $labels "imagePullSecrets" .imagePullSecrets "podLabels" .podLabels }} apiVersion: v1 kind: ServiceAccount metadata: @@ -147,10 +149,14 @@ activeDeadlineSeconds: {{ .timeoutSeconds }} template: metadata: labels: - {{- .labels | nindent 6 }} + {{- include "kafka-common.podLabels" (dict "labels" .labels "podLabels" .podLabels) | nindent 6 }} spec: serviceAccountName: {{ .fullname }}-secret-sync restartPolicy: Never + {{- with .imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 6 }} + {{- end }} securityContext: runAsNonRoot: true runAsUser: 65534 diff --git a/charts/kafka-common/templates/_values.tpl b/charts/kafka-common/templates/_values.tpl index c1d41dec..b4607ddc 100644 --- a/charts/kafka-common/templates/_values.tpl +++ b/charts/kafka-common/templates/_values.tpl @@ -42,3 +42,36 @@ nothing and leaves Kubernetes' own rule. Call with the policy string. imagePullPolicy: {{ . }} {{- end }} {{- end }} + +{{/* +An image reference behind a mirror registry: / when registry +is set, as charts/kates does with global.imageRegistry; the image unchanged +otherwise, and an empty image stays empty. Call with (dict "image" +"registry" ). +*/}} +{{- define "kafka-common.image" -}} +{{- if and .registry .image -}} +{{- printf "%s/%s" (trimSuffix "/" .registry) .image -}} +{{- else -}} +{{- .image -}} +{{- end -}} +{{- end }} + +{{/* +The security contexts Pod Security Standards "restricted" asks for, and no +more — what Strimzi's restricted security provider sets. Strimzi operands get +them through their CR templates, so the pods pass on a shared operator that +is not configured with that provider. readOnlyRootFilesystem is not part of +the standard and is left to the caller. YAML. +*/}} +{{- define "kafka-common.restricted.pod" -}} +runAsNonRoot: true +seccompProfile: + type: RuntimeDefault +{{- end }} + +{{- define "kafka-common.restricted.container" -}} +allowPrivilegeEscalation: false +capabilities: + drop: ["ALL"] +{{- end }} diff --git a/charts/kafka-common/templates/_worker.tpl b/charts/kafka-common/templates/_worker.tpl index ad8dfb89..94347f1a 100644 --- a/charts/kafka-common/templates/_worker.tpl +++ b/charts/kafka-common/templates/_worker.tpl @@ -9,6 +9,8 @@ from its own values: kind, name, namespace the CR this spec belongs to version, image + imageRegistry prefixed to image and rack.clientRackInitImage + when both are set (kafka-common.image) replicas, minReplicasDefault autoscaling {enabled, minReplicas} resources, jvmOptions, livenessProbe, readinessProbe, jmxOptions @@ -27,7 +29,9 @@ from its own values: nodeSelector, nodeAffinity podAntiAffinity {enabled, topologyKey} topologySpread {enabled, maxSkew, topologyKey, whenUnsatisfiable, additional} - container {env, securityContext, volumeMounts, harden, extra} + container {env, securityContext, volumeMounts, harden, extra}; + harden also gives the rack init container the + restricted security context serviceAccount ServiceAccount template (metadata) extra raw spec.template, deep-merged last @@ -55,7 +59,7 @@ templateExtra.pod. Everything below is merged, then emitted once. {{- end -}} version: {{ $w.version | quote }} {{- with $w.image }} -image: {{ . | quote }} +image: {{ include "kafka-common.image" (dict "image" . "registry" $w.imageRegistry) | quote }} {{- end }} replicas: {{ $replicas }} {{- with $w.resources }} @@ -111,7 +115,7 @@ rack: {{- /* The operator reads the node's topology label with an init container; on a mirrored or air-gapped registry its image needs naming. */}} {{- with $rack.clientRackInitImage }} -clientRackInitImage: {{ . | quote }} +clientRackInitImage: {{ include "kafka-common.image" (dict "image" . "registry" $w.imageRegistry) | quote }} {{- end }} {{- end }} {{- with $w.jmxOptions }} @@ -222,6 +226,12 @@ spec.template for a Connect worker group, as YAML — see connectWorker.spec. {{- with $sc }}{{- $_ := set $cont "securityContext" . }}{{- end -}} {{- $cont = mustMergeOverwrite $cont (deepCopy ($c.extra | default dict)) -}} {{- with $cont }}{{- $_ := set $tpl "connectContainer" . }}{{- end -}} +{{- /* Rack awareness adds the operator's init container, whose security + context an operator without the restricted provider leaves empty — + which Pod Security "restricted" refuses. */ -}} +{{- if and $c.harden ($w.rack | default dict).enabled -}} +{{- $_ := set $tpl "initContainer" (dict "securityContext" (include "kafka-common.restricted.container" . | fromYaml)) -}} +{{- end -}} {{- with $t.serviceAccount }}{{- $_ := set $tpl "serviceAccount" . }}{{- end -}} {{- $tpl = mustMergeOverwrite $tpl (deepCopy ($t.extra | default dict)) -}} {{- if $tpl }}{{ toYaml $tpl }}{{ end -}} diff --git a/charts/kafka-common/tests/harness/Chart.yaml b/charts/kafka-common/tests/harness/Chart.yaml index 88aa61c7..7e6262a0 100644 --- a/charts/kafka-common/tests/harness/Chart.yaml +++ b/charts/kafka-common/tests/harness/Chart.yaml @@ -6,5 +6,5 @@ version: 0.0.0 appVersion: "0.0.0" dependencies: - name: kafka-common - version: 0.1.0 + version: 0.1.1 repository: file://../.. diff --git a/charts/kafka-common/tests/harness/templates/labels.yaml b/charts/kafka-common/tests/harness/templates/labels.yaml index e7392020..9db56d8b 100644 --- a/charts/kafka-common/tests/harness/templates/labels.yaml +++ b/charts/kafka-common/tests/harness/templates/labels.yaml @@ -11,6 +11,9 @@ data: semver: {{ include "kafka-common.semver" "4.2-IV1" | quote }} semverShort: {{ include "kafka-common.semver" "2.8" | quote }} domain: {{ include "kafka-common.clusterDomain" . | quote }} + podLabels: | + {{- include "kafka-common.podLabels" (dict "labels" (include "kafka-common.selectorLabels" .) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} + image: {{ include "kafka-common.image" (dict "image" "ghcr.io/x/y:1" "registry" .Values.registry) | quote }} {{- range $k, $v := .Values.toggles }} toggle-{{ $k }}: {{ include "kafka-common.enabled" (list $v.value $v.default) | quote }} {{- end }} diff --git a/charts/kafka-common/tests/harness/templates/secretsync.yaml b/charts/kafka-common/tests/harness/templates/secretsync.yaml index c2ba49fd..486a9f20 100644 --- a/charts/kafka-common/tests/harness/templates/secretsync.yaml +++ b/charts/kafka-common/tests/harness/templates/secretsync.yaml @@ -1 +1 @@ -{{- include "kafka-common.secretSync" (dict "fullname" "harness" "namespace" "dst" "labels" (include "kafka-common.selectorLabels" .) "secrets" .Values.secretSync.secrets "image" "ghcr.io/bmscomp/kates-tester:1.22.0" "watch" .Values.secretSync.watch) }} +{{- include "kafka-common.secretSync" (dict "fullname" "harness" "namespace" "dst" "labels" (include "kafka-common.selectorLabels" .) "secrets" .Values.secretSync.secrets "image" "ghcr.io/bmscomp/kates-tester:1.22.0" "watch" .Values.secretSync.watch "imagePullSecrets" .Values.secretSync.imagePullSecrets "podLabels" .Values.podLabels) }} diff --git a/charts/kafka-common/tests/harness/tests/labels_test.yaml b/charts/kafka-common/tests/harness/tests/labels_test.yaml index 33d1fc87..9714f1fd 100644 --- a/charts/kafka-common/tests/harness/tests/labels_test.yaml +++ b/charts/kafka-common/tests/harness/tests/labels_test.yaml @@ -18,6 +18,22 @@ tests: - equal: path: 'metadata.labels["app.kubernetes.io/component"]' value: overridden + - it: adds podLabels to a pod's labels, never over the chart's own + release: {name: h} + set: + podLabels: {team: payments, app.kubernetes.io/instance: hijack, kates.io/test-pod: "false"} + asserts: + - matchRegex: {path: data.podLabels, pattern: "(?m)^team: payments$"} + - matchRegex: {path: data.podLabels, pattern: "(?m)^app.kubernetes.io/instance: h$"} + - matchRegex: {path: data.podLabels, pattern: "(?m)^kates.io/test-pod: \"true\"$"} + - it: leaves an image alone without a mirror registry + asserts: + - equal: {path: data.image, value: "ghcr.io/x/y:1"} + - it: prefixes the registry once, without a doubled slash + set: + registry: mirror.example.com/ + asserts: + - equal: {path: data.image, value: "mirror.example.com/ghcr.io/x/y:1"} - it: normalises versions asserts: - equal: {path: data.semver, value: "4.2.0"} diff --git a/charts/kafka-common/tests/harness/tests/resources_test.yaml b/charts/kafka-common/tests/harness/tests/resources_test.yaml index 9c3293de..65ebba5f 100644 --- a/charts/kafka-common/tests/harness/tests/resources_test.yaml +++ b/charts/kafka-common/tests/harness/tests/resources_test.yaml @@ -106,6 +106,21 @@ tests: documentIndex: 7 - isKind: {of: CronJob} documentIndex: 8 + - it: gives the sync pods the pull secrets and podLabels, under the chart's labels + template: secretsync.yaml + release: {name: h} + set: + secretSync: + imagePullSecrets: [{name: regcred}] + secrets: [{name: a, fromNamespace: src}] + podLabels: {team: payments, app.kubernetes.io/instance: hijack} + asserts: + - equal: {path: 'spec.template.spec.imagePullSecrets[0].name', value: regcred} + documentIndex: 5 + - equal: {path: 'spec.template.metadata.labels.team', value: payments} + documentIndex: 5 + - equal: {path: 'spec.template.metadata.labels["app.kubernetes.io/instance"]', value: h} + documentIndex: 5 - it: renders the NetworkPolicy fragments as valid rules template: netpol.yaml set: diff --git a/charts/kafka-common/tests/harness/tests/worker_test.yaml b/charts/kafka-common/tests/harness/tests/worker_test.yaml index fb51a165..ff1e2f01 100644 --- a/charts/kafka-common/tests/harness/tests/worker_test.yaml +++ b/charts/kafka-common/tests/harness/tests/worker_test.yaml @@ -23,6 +23,24 @@ tests: - equal: {path: spec.metricsConfig.type, value: jmxPrometheusExporter} - equal: {path: spec.metricsConfig.valueFrom.configMapKeyRef.name, value: harness-metrics} + - it: hardens the rack init container and puts a mirror registry in front of the images + set: + worker.image: quay.io/strimzi/kafka:1.2.0-kafka-4.3.1 + worker.imageRegistry: mirror.example.com + worker.rack: {enabled: true, clientRackInitImage: "r/i:1"} + worker.template.container: {harden: true} + asserts: + - equal: {path: spec.image, value: "mirror.example.com/quay.io/strimzi/kafka:1.2.0-kafka-4.3.1"} + - equal: {path: spec.clientRackInitImage, value: "mirror.example.com/r/i:1"} + - equal: {path: spec.template.initContainer.securityContext.allowPrivilegeEscalation, value: false} + - equal: {path: 'spec.template.initContainer.securityContext.capabilities.drop', value: [ALL]} + + - it: leaves the init container alone without rack awareness + set: + worker.template.container: {harden: true} + asserts: + - notExists: {path: spec.template.initContainer} + - it: seeds replicas from minReplicas under an HPA (v1 requires the field) set: worker.autoscaling: {enabled: true, minReplicas: 4} diff --git a/charts/kafka-common/tests/harness/values.yaml b/charts/kafka-common/tests/harness/values.yaml index 67e80dae..b4630f6f 100644 --- a/charts/kafka-common/tests/harness/values.yaml +++ b/charts/kafka-common/tests/harness/values.yaml @@ -33,6 +33,9 @@ kafkaUser: secretSync: secrets: [] watch: false + imagePullSecrets: [] +podLabels: {} +registry: "" netpol: dns: {} bootstrapPorts: diff --git a/charts/kafka-ui/Chart.yaml b/charts/kafka-ui/Chart.yaml index ca0df3e8..497f1eaa 100644 --- a/charts/kafka-ui/Chart.yaml +++ b/charts/kafka-ui/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: kafka-ui description: A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication type: application -version: 0.3.0 +version: 0.3.1 appVersion: "v1.5.0" kubeVersion: ">=1.27.0-0" home: https://github.com/bmscomp/kates @@ -23,6 +23,12 @@ annotations: artifacthub.io/category: streaming-messaging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: fixed + description: "Every pod the chart renders passes Pod Security Standards restricted: podSecurityContext sets seccompProfile RuntimeDefault, and the Helm test pod, which had no security context or resources, runs as nobody (65534) with no privilege escalation, all capabilities dropped and 10m/32Mi requests with a 64Mi limit (tests.podSecurityContext, tests.securityContext, tests.resources)" + - kind: added + description: "global.imageRegistry (default empty) is prepended to the Kafka UI and Helm test images, for clusters that pull from a mirror; the test image is tests.image (busybox:1.36, as before) and the test pod gets imagePullSecrets too" + - kind: added + description: "podLabels (default empty) adds labels to the Kafka UI pod and the Helm test pod; the chart's own labels win on a clash, so the selector never changes" - kind: fixed description: Wire the Kafka TLS truststore (mount trustedCertificateSecret + set ssl.truststore.location) so SASL_SSL actually works - kind: fixed diff --git a/charts/kafka-ui/README.md b/charts/kafka-ui/README.md index 6483a526..d4ca3ac4 100644 --- a/charts/kafka-ui/README.md +++ b/charts/kafka-ui/README.md @@ -465,7 +465,7 @@ The chart runs hardened by default: - Dedicated `ServiceAccount` with `automountServiceAccountToken: false` — Kafka UI does not call the Kubernetes API. - `readOnlyRootFilesystem: true` with an `emptyDir` mounted at `/tmp`. -- Non-root, all capabilities dropped, no privilege escalation. +- Non-root, seccomp `RuntimeDefault`, all capabilities dropped, no privilege escalation: the Kafka UI pod and the Helm test pod pass Pod Security Standards `restricted`. - A default-scoped `NetworkPolicy` (no Kubernetes API-server egress unless `networkPolicy.apiServerEgress=true`). - The web login password is randomly generated on first install and preserved across upgrades (never a well-known default). @@ -487,7 +487,9 @@ extraVolumeMounts: [] | `image.tag` | `""` (Chart.appVersion) | Image tag | | `image.pullPolicy` | `IfNotPresent` | Image pull policy | | `image.digest` | `""` | Image digest (overrides tag) | -| `imagePullSecrets` | `[]` | Registry pull secrets | +| `global.imageRegistry` | `""` | Registry prepended to every image (Kafka UI and the Helm test) | +| `imagePullSecrets` | `[]` | Registry pull secrets (the Helm test pod too) | +| `tests.image.repository` / `tests.image.tag` | `busybox` / `1.36` | Helm test image | ### Kafka @@ -543,7 +545,11 @@ extraVolumeMounts: [] | `serviceAccount.create` | `true` | Create a dedicated ServiceAccount | | `serviceAccount.name` | `""` (fullname) | ServiceAccount name | | `serviceAccount.automountServiceAccountToken` | `false` | Mount the API token (not needed) | +| `podSecurityContext` | non-root 1001, seccomp `RuntimeDefault` | Pod security context (passes PSS restricted) | | `securityContext.readOnlyRootFilesystem` | `true` | Read-only root FS (chart mounts `/tmp` emptyDir) | +| `podLabels` | `{}` | Extra labels on every pod (the chart's own labels win on a clash) | +| `tests.podSecurityContext` / `tests.securityContext` | nobody (65534), seccomp `RuntimeDefault`, all capabilities dropped | Helm test pod security | +| `tests.resources` | `10m`/`32Mi` requests, `64Mi` limit | Helm test pod resources | | `extraVolumes` / `extraVolumeMounts` | `[]` | Extra volumes/mounts (custom truststores, serdes) | ### NetworkPolicy diff --git a/charts/kafka-ui/templates/_helpers.tpl b/charts/kafka-ui/templates/_helpers.tpl index 64e16630..d91b5af8 100644 --- a/charts/kafka-ui/templates/_helpers.tpl +++ b/charts/kafka-ui/templates/_helpers.tpl @@ -96,16 +96,28 @@ Kafka bootstrap servers FQDN. {{/* Container image reference. -Returns repository@digest when digest is set, repository:tag otherwise. +Returns repository@digest when digest is set, repository:tag otherwise, +behind global.imageRegistry when that is set. */}} {{- define "kafka-ui.image" -}} +{{- $repo := .Values.image.repository -}} +{{- with .Values.global.imageRegistry }}{{ $repo = printf "%s/%s" . $repo }}{{ end -}} {{- if .Values.image.digest -}} -{{- printf "%s@%s" .Values.image.repository .Values.image.digest -}} +{{- printf "%s@%s" $repo .Values.image.digest -}} {{- else -}} -{{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) -}} +{{- printf "%s:%s" $repo (.Values.image.tag | default .Chart.AppVersion) -}} {{- end -}} {{- end -}} +{{/* +Labels for a pod template: the chart's own labels (pass them as .labels) +plus podLabels. The chart's own win on a clash, so a selector never changes. +Usage: include "kafka-ui.podLabels" (dict "ctx" . "labels" (include "kafka-ui.selectorLabels" .)) +*/}} +{{- define "kafka-ui.podLabels" -}} +{{- toYaml (merge (fromYaml .labels) .ctx.Values.podLabels) }} +{{- end -}} + {{/* Schema Registry URL — auto-computed from namespace if not explicitly set. */}} diff --git a/charts/kafka-ui/templates/deployment.yaml b/charts/kafka-ui/templates/deployment.yaml index 07a730b6..90e373b7 100644 --- a/charts/kafka-ui/templates/deployment.yaml +++ b/charts/kafka-ui/templates/deployment.yaml @@ -15,7 +15,7 @@ spec: template: metadata: labels: - {{- include "kafka-ui.selectorLabels" . | nindent 8 }} + {{- include "kafka-ui.podLabels" (dict "ctx" . "labels" (include "kafka-ui.selectorLabels" .)) | nindent 8 }} annotations: checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} {{- with .Values.podAnnotations }} diff --git a/charts/kafka-ui/templates/tests/test-connection.yaml b/charts/kafka-ui/templates/tests/test-connection.yaml index 9afa5b5e..ef00b0e1 100644 --- a/charts/kafka-ui/templates/tests/test-connection.yaml +++ b/charts/kafka-ui/templates/tests/test-connection.yaml @@ -4,15 +4,31 @@ metadata: name: {{ include "kafka-ui.fullname" . }}-test-connection namespace: {{ include "kafka-ui.namespace" . }} labels: - {{- include "kafka-ui.labels" . | nindent 4 }} + {{- include "kafka-ui.podLabels" (dict "ctx" . "labels" (include "kafka-ui.labels" .)) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.tests.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: test - image: busybox:1.36 + image: "{{ with .Values.global.imageRegistry }}{{ . }}/{{ end }}{{ .Values.tests.image.repository }}:{{ .Values.tests.image.tag }}" + {{- with .Values.tests.securityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tests.resources }} + resources: + {{- toYaml . | nindent 8 }} + {{- end }} command: ['sh', '-c'] args: - | diff --git a/charts/kafka-ui/values.schema.json b/charts/kafka-ui/values.schema.json index 2d8803da..5876e55e 100644 --- a/charts/kafka-ui/values.schema.json +++ b/charts/kafka-ui/values.schema.json @@ -3,6 +3,12 @@ "type": "object", "additionalProperties": true, "properties": { + "global": { + "type": "object", + "properties": { + "imageRegistry": { "type": "string" } + } + }, "image": { "type": "object", "properties": { @@ -178,6 +184,22 @@ "podSecurityContext": { "type": "object" }, "securityContext": { "type": "object" }, "podAnnotations": { "type": "object" }, + "podLabels": { "type": "object", "additionalProperties": { "type": "string" } }, + "tests": { + "type": "object", + "properties": { + "image": { + "type": "object", + "properties": { + "repository": { "type": "string", "minLength": 1 }, + "tag": { "type": "string", "minLength": 1 } + } + }, + "podSecurityContext": { "type": "object" }, + "securityContext": { "type": "object" }, + "resources": { "type": "object" } + } + }, "nodeSelector": { "type": "object" }, "tolerations": { "type": "array" }, "affinity": { "type": "object" }, diff --git a/charts/kafka-ui/values.yaml b/charts/kafka-ui/values.yaml index 23925ef9..45e9bae5 100644 --- a/charts/kafka-ui/values.yaml +++ b/charts/kafka-ui/values.yaml @@ -1,3 +1,10 @@ +# -- Global configuration +global: + # -- Registry prepended to every image the chart renders (Kafka UI and the + # Helm test), for clusters that pull from a mirror: ghcr.io/kafbat/kafka-ui + # becomes /ghcr.io/kafbat/kafka-ui. Empty pulls as written. + imageRegistry: "" + # -- Image configuration image: # -- Container image repository (migrated from deprecated provectuslabs/kafka-ui) @@ -10,7 +17,7 @@ image: # digest: "sha256:abc123..." digest: "" -# -- Image pull secrets for private registries +# -- Image pull secrets for private registries (the Helm test pod uses them too) imagePullSecrets: [] # - name: ghcr-credentials @@ -211,6 +218,10 @@ podAnnotations: {} # prometheus.io/scrape: "true" # prometheus.io/port: "8080" +# -- Extra labels for every pod the chart renders (Kafka UI and the Helm test). +# The chart's own labels win on a clash, so a selector never changes. +podLabels: {} + # -- Pod resource requests and limits resources: requests: @@ -247,6 +258,8 @@ podSecurityContext: runAsNonRoot: true runAsUser: 1001 fsGroup: 1001 + seccompProfile: + type: RuntimeDefault # -- Container-level security context. readOnlyRootFilesystem is on; the chart # mounts an emptyDir at /tmp for the app's scratch space. @@ -303,6 +316,31 @@ metrics: relabelings: [] metricRelabelings: [] +# -- Helm test pod (`helm test`): wget against the Kafka UI health endpoint +tests: + image: + # -- Test image repository (global.imageRegistry is prepended when set) + repository: busybox + tag: "1.36" + # -- busybox runs as root unless told otherwise; the test runs as nobody + podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 64Mi + # -- Extra volumes / mounts (e.g. custom truststores, serde descriptors) extraVolumes: [] extraVolumeMounts: [] diff --git a/charts/kates-chaos/Chart.yaml b/charts/kates-chaos/Chart.yaml index 7c28e548..19d3ba60 100644 --- a/charts/kates-chaos/Chart.yaml +++ b/charts/kates-chaos/Chart.yaml @@ -1,13 +1,9 @@ apiVersion: v2 name: kates-chaos -# Minor bump: the Grafana board moves out of the template into -# dashboards/kates-chaos-infra/ and four of the series it read are corrected, -# so the same board shows different — and for the first time non-empty — data -# after upgrade. docs/grafana-dashboards-refactor-plan.md. -# Patch: the exporter the infrastructure board reads gets a values block of -# its own, and the chart's ServiceMonitor stops scraping the operator pod by -# mistake. Defaults are unchanged. -version: 2.2.1 +# Minor bump: the installer, GameDay and test pods now run as uid 65534 with +# a PSS restricted securityContext and resources by default, so an installer +# or GameDay image that needs root stops starting. +version: 2.3.0 # Tracks the LitmusChaos execution-plane version this chart deploys. appVersion: "3.28.0" description: Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring @@ -31,6 +27,18 @@ annotations: artifacthub.io/category: skip-prediction artifacthub.io/prerelease: "false" artifacthub.io/changes: | + - kind: changed + description: "The experiment installer Job, the GameDay Job and both helm test pods pass Pod Security Standards restricted by default: seccompProfile RuntimeDefault, runAsNonRoot as uid/gid 65534 (registry.k8s.io/kubectl runs as root otherwise), allowPrivilegeEscalation false, every capability dropped, and CPU and memory requests with a memory limit (10m/64Mi, limit 128Mi). Set through experiments.installer.podSecurityContext/securityContext/resources (resources now reach the installer's two init containers too), the new gameday.podSecurityContext/securityContext/resources and the new tests block. An installer or GameDay image that must run as root needs its podSecurityContext set back" + - kind: changed + description: "With values-generic.yaml, whose kyvernoPolicy ClusterPolicy (still on, mutate true) patches a pod with no securityContext to uid 1001 at admission, the installer, GameDay and test pods now run as the uid 65534 they set themselves. litmus-core's operator and exporter take no securityContext from values: they are restricted only through that mutate policy or a kyvernoPolicy.exceptions PolicyException, so a release that turns kyvernoPolicy off on a cluster enforcing restricted needs the exception" + - kind: added + description: "kyvernoPolicy.exceptions, a list of {policyName, ruleNames}: one kyverno.io/v2 PolicyException (-exceptions, rendered where that API exists) in the release namespace exempting litmus-core's operator and exporter (app.kubernetes.io/instance= and app.kubernetes.io/name=litmus-core's nameOverride, litmus; the chart's own pods pass on their own, and chaos runner and experiment pods are not matched) from those rules of any Kyverno policy, the cluster's own included, each rule's autogen- and autogen-cronjob- variants too. litmus-core takes no securityContext from values, so its operator (and exporter) Deployment fails PSS restricted unless kyvernoPolicy's mutate rules patch it; this is how a cluster that enforces restricted without them exempts them" + - kind: added + description: "podLabels, added to the labels of the installer Job, GameDay Job and helm test pods; the chart's own labels win on a clash. litmus-core's pods take labels from litmus-core.customLabels instead" + - kind: added + description: "litmus-core.exporter.resources defaults to requests 50m/64Mi and a 128Mi memory limit; the subchart's own default is none" + - kind: fixed + description: "The helm test pods ignored global.imagePullSecrets" - kind: added description: "litmus-core.exporter — the chaos exporter the Kates — Chaos infrastructure board reads (litmuschaos_* series), off upstream and still off by default here, with its image pinned to docker.io like the operator and runner rather than pulled through the scarf.sh redirector. kates deploy --with-monitoring turns it on together with monitoring.serviceMonitor" - kind: fixed diff --git a/charts/kates-chaos/README.md b/charts/kates-chaos/README.md index 6981b615..823e0fa8 100644 --- a/charts/kates-chaos/README.md +++ b/charts/kates-chaos/README.md @@ -60,6 +60,7 @@ helm upgrade --install chaos charts/kates-chaos \ | `fullnameOverride` | Override full release name | `""` | | `commonLabels` | Labels added to every resource | `{}` | | `commonAnnotations` | Annotations added to every resource | `{}` | +| `podLabels` | Labels added to the installer, GameDay and test pods; the chart's own labels win on a clash. litmus-core's pods use `litmus-core.customLabels` | `{}` | | `serviceAccount.name` | Chaos service account (shared by RBAC + engines) | `litmus-admin` | ### Global & Images @@ -76,7 +77,7 @@ helm upgrade --install chaos charts/kates-chaos \ | `images.goRunner.repository` | Default runtime for experiments | `litmuschaos/go-runner` | | `images.goRunner.tag` | go-runner image tag | `3.28.0` | -The `litmus-core` subchart images (operator/runner) are set under the `litmus-core:` key. +The `litmus-core` subchart images (operator/runner/exporter) are set under the `litmus-core:` key; `global.imageRegistry` and `global.imagePullSecrets` do not reach them (the subchart has no such knobs). ### RBAC @@ -97,7 +98,10 @@ The `litmus-core` subchart images (operator/runner) are set under the `litmus-co | `experiments.targetNamespaces` | Namespaces to install into (empty = every `target` namespace) | `[]` | | `experiments.installer.backoffLimit` | Installer Job retries | `3` | | `experiments.installer.ttlSecondsAfterFinished` | Installer Job TTL | `300` | -| `experiments.installer.resources` / `securityContext` / `nodeSelector` / `tolerations` | Installer pod tuning | `{}` / `[]` | +| `experiments.installer.resources` | Resources of the installer's container and two init containers | requests `10m`/`64Mi`, limit `128Mi` | +| `experiments.installer.podSecurityContext` | Installer pod securityContext | PSS restricted, uid/gid `65534` | +| `experiments.installer.securityContext` | Installer containers' securityContext | no privilege escalation, drop `ALL` | +| `experiments.installer.nodeSelector` / `tolerations` / `affinity` | Installer pod scheduling | `{}` / `[]` / `{}` | | `experiments.definitions[].name` | Experiment name | — | | `experiments.definitions[].scope` | `Namespaced` or `Cluster` | `Namespaced` | | `experiments.definitions[].env` | Default env baked into the ChaosExperiment | — | @@ -166,10 +170,13 @@ Each key under `engines:` renders a `ChaosEngine`. Probes and pod `components` a | `kyvernoPolicy.action` | `Audit` / `Enforce` | `Audit` | | `kyvernoPolicy.namespaces` | Namespaces the policy applies to (empty = release ns) | `[]` | | `kyvernoPolicy.excludeSelector` | Pods skipped (need privileges) | `chaosUID Exists` | +| `kyvernoPolicy.exceptions` | `[{policyName, ruleNames}]` — one `kyverno.io/v2` PolicyException in the release namespace, matching litmus-core's operator and exporter (`app.kubernetes.io/instance` and `app.kubernetes.io/name: litmus`, litmus-core's `nameOverride`), for any policy, the cluster's own included, each rule's `autogen-` and `autogen-cronjob-` variants too. Rendered whenever non-empty and the API exists, regardless of `kyvernoPolicy.enabled`; Kyverno must accept exceptions from the release namespace | `[]` | | `pdb.enabled` | PodDisruptionBudget (requires `selector`) | `false` | | `pdb.minAvailable` / `maxUnavailable` / `selector` | PDB config | `1` / `""` / `{}` | | `gameday.enabled` | Run the validation Job | `false` | | `gameday.checks` | Data-driven `{name, command}` checks (empty = default set) | `[]` | +| `gameday.resources` / `podSecurityContext` / `securityContext` | GameDay pod tuning | as `experiments.installer.*` | +| `tests.resources` / `podSecurityContext` / `securityContext` | Helm test pods tuning | as `experiments.installer.*` | ## Examples @@ -228,7 +235,20 @@ gameday: helm test chaos -n litmus ``` -Runs two hooks: the chaos **operator** is `Available`, and the Litmus **CRDs** are established with experiments installed. Both use a scoped, ephemeral test ServiceAccount. +Runs two hooks: the chaos **operator** is `Available`, and the Litmus **CRDs** are established with experiments installed. Both use a scoped, ephemeral test ServiceAccount, and `tests.*` sets their securityContext and resources. + +## Pod Security + +The installer Job, GameDay Job and test pods pass Pod Security Standards **restricted** with default values. The litmus-core subchart exposes no securityContext, so its operator Deployment (`litmus`) and, when `litmus-core.exporter.enabled`, its exporter (`litmus-monitor`) do not on their own. On a cluster with Kyverno, `kyvernoPolicy` with `mutate: true` (`values-generic.yaml` turns it on) patches them to restricted at admission. On a cluster that enforces restricted through its own Kyverno policy without that, exempt them (the exception matches only those two, by `app.kubernetes.io/name: litmus`): + +```yaml +kyvernoPolicy: + exceptions: + - policyName: + ruleNames: [] +``` + +A namespace labelled `pod-security.kubernetes.io/enforce: restricted` rejects them unless the mutate rules patched them first; they pass `baseline`. ## Upgrading diff --git a/charts/kates-chaos/templates/_helpers.tpl b/charts/kates-chaos/templates/_helpers.tpl index c7525cb2..33448056 100644 --- a/charts/kates-chaos/templates/_helpers.tpl +++ b/charts/kates-chaos/templates/_helpers.tpl @@ -42,6 +42,15 @@ helm.sh/chart: {{ include "kates-chaos.chart" . }} {{- end }} {{- end }} +{{/* +Pod template labels: the chart's own labels (a YAML string) plus podLabels. +The chart's own win on a clash, so a selector never changes. +Usage: include "kates-chaos.podLabels" (list . (include "kates-chaos.selectorLabels" .)) +*/}} +{{- define "kates-chaos.podLabels" -}} +{{- toYaml (merge (index . 1 | fromYaml) ((index . 0).Values.podLabels | default dict)) -}} +{{- end }} + {{/* Common annotations (empty unless commonAnnotations is set). */}} diff --git a/charts/kates-chaos/templates/experiments.yaml b/charts/kates-chaos/templates/experiments.yaml index 9e2acf89..c32d9c3e 100644 --- a/charts/kates-chaos/templates/experiments.yaml +++ b/charts/kates-chaos/templates/experiments.yaml @@ -88,8 +88,7 @@ spec: template: metadata: labels: - app: experiment-installer - {{- include "kates-chaos.selectorLabels" . | nindent 8 }} + {{- include "kates-chaos.podLabels" (list . (printf "app: experiment-installer\n%s" (include "kates-chaos.selectorLabels" .))) | nindent 8 }} spec: restartPolicy: OnFailure serviceAccountName: {{ include "kates-chaos.fullname" . }}-experiment-installer @@ -126,6 +125,10 @@ spec: securityContext: {{- toYaml . | nindent 12 }} {{- end }} + {{- with $installer.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} command: - "kubectl" - "wait" @@ -143,6 +146,10 @@ spec: securityContext: {{- toYaml . | nindent 12 }} {{- end }} + {{- with $installer.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} command: - "kubectl" - "wait" diff --git a/charts/kates-chaos/templates/gameday.yaml b/charts/kates-chaos/templates/gameday.yaml index 8458c2e9..5b2070da 100644 --- a/charts/kates-chaos/templates/gameday.yaml +++ b/charts/kates-chaos/templates/gameday.yaml @@ -83,8 +83,7 @@ spec: template: metadata: labels: - app: gameday-runner - {{- include "kates-chaos.selectorLabels" . | nindent 8 }} + {{- include "kates-chaos.podLabels" (list . (printf "app: gameday-runner\n%s" (include "kates-chaos.selectorLabels" .))) | nindent 8 }} spec: restartPolicy: Never serviceAccountName: {{ include "kates-chaos.fullname" . }}-gameday @@ -92,10 +91,22 @@ spec: imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} + {{- with .Values.gameday.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: gameday image: {{ $image }} imagePullPolicy: {{ $pullPolicy }} + {{- with .Values.gameday.securityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.gameday.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} command: ["/bin/sh"] args: - -c diff --git a/charts/kates-chaos/templates/kyverno-policy-exceptions.yaml b/charts/kates-chaos/templates/kyverno-policy-exceptions.yaml new file mode 100644 index 00000000..e363b9b9 --- /dev/null +++ b/charts/kates-chaos/templates/kyverno-policy-exceptions.yaml @@ -0,0 +1,47 @@ +{{- /* kyvernoPolicy.exceptions: this release's litmus-core operator and + exporter, in its own namespace, exempt from rules of any policy, the + cluster's included (litmus-core takes no securityContext from values). + Not tied to kyvernoPolicy.enabled. */}} +{{- with .Values.kyvernoPolicy.exceptions }} +{{- if $.Capabilities.APIVersions.Has "kyverno.io/v2" }} +apiVersion: kyverno.io/v2 +kind: PolicyException +metadata: + name: {{ printf "%s-exceptions" (include "kates-chaos.fullname" $) | trunc 63 | trimSuffix "-" }} + namespace: {{ $.Release.Namespace }} + labels: + {{- include "kates-chaos.labels" $ | nindent 4 }} + {{- with include "kates-chaos.annotations" $ }} + annotations: + {{- . | nindent 4 }} + {{- end }} +spec: + exceptions: + {{- range . }} + - policyName: {{ .policyName | quote }} + ruleNames: + {{- range .ruleNames }} + {{- /* with the rules Kyverno generates from it for controllers and CronJobs */}} + - {{ . | quote }} + - {{ printf "autogen-%s" . | quote }} + - {{ printf "autogen-cronjob-%s" . | quote }} + {{- end }} + {{- end }} + match: + any: + - resources: + kinds: [Pod, Deployment, ReplicaSet, StatefulSet, Job, CronJob] + namespaces: [{{ $.Release.Namespace | quote }}] + {{- /* litmus-core's operator and exporter only: both carry the + release and litmus-core's name (its nameOverride, litmus). + This chart's own pods pass on their own; chaos runner and + experiment pods carry no instance label */}} + selector: + matchLabels: + app.kubernetes.io/instance: {{ $.Release.Name | quote }} + matchExpressions: + - key: app.kubernetes.io/name + operator: In + values: [{{ (index $.Values "litmus-core").nameOverride | default "litmus" | quote }}] +{{- end }} +{{- end }} diff --git a/charts/kates-chaos/templates/tests/test-connectivity.yaml b/charts/kates-chaos/templates/tests/test-connectivity.yaml index 0d4b389e..c851d5de 100644 --- a/charts/kates-chaos/templates/tests/test-connectivity.yaml +++ b/charts/kates-chaos/templates/tests/test-connectivity.yaml @@ -63,17 +63,33 @@ metadata: name: {{ include "kates-chaos.fullname" . }}-test-operator namespace: {{ .Release.Namespace }} labels: - {{- include "kates-chaos.labels" . | nindent 4 }} + {{- include "kates-chaos.podLabels" (list . (include "kates-chaos.labels" .)) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never serviceAccountName: {{ $testSA }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.tests.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: test image: {{ $kubectl }} imagePullPolicy: {{ $pullPolicy }} + {{- with .Values.tests.securityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tests.resources }} + resources: + {{- toYaml . | nindent 8 }} + {{- end }} command: ['sh', '-c'] args: - | @@ -93,17 +109,33 @@ metadata: name: {{ include "kates-chaos.fullname" . }}-test-crds namespace: {{ .Release.Namespace }} labels: - {{- include "kates-chaos.labels" . | nindent 4 }} + {{- include "kates-chaos.podLabels" (list . (include "kates-chaos.labels" .)) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never serviceAccountName: {{ $testSA }} + {{- with .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.tests.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: test image: {{ $kubectl }} imagePullPolicy: {{ $pullPolicy }} + {{- with .Values.tests.securityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tests.resources }} + resources: + {{- toYaml . | nindent 8 }} + {{- end }} command: ['sh', '-c'] args: - | diff --git a/charts/kates-chaos/values.schema.json b/charts/kates-chaos/values.schema.json index b05d56d0..297ce586 100644 --- a/charts/kates-chaos/values.schema.json +++ b/charts/kates-chaos/values.schema.json @@ -17,6 +17,15 @@ "fullnameOverride": { "type": "string" }, "commonLabels": { "type": "object" }, "commonAnnotations": { "type": "object" }, + "podLabels": { "type": "object", "additionalProperties": { "type": "string" } }, + "tests": { + "type": "object", + "properties": { + "resources": { "type": "object" }, + "podSecurityContext": { "type": "object" }, + "securityContext": { "type": "object" } + } + }, "global": { "type": "object", "properties": { @@ -183,7 +192,18 @@ "action": { "type": "string", "enum": ["Audit", "Enforce"] }, "mutate": { "type": "boolean" }, "namespaces": { "type": "array", "items": { "type": "string" } }, - "excludeSelector": { "type": "object" } + "excludeSelector": { "type": "object" }, + "exceptions": { + "type": "array", + "items": { + "type": "object", + "required": ["policyName", "ruleNames"], + "properties": { + "policyName": { "type": "string", "minLength": 1 }, + "ruleNames": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } } + } + } + } } }, "gameday": { @@ -193,6 +213,9 @@ "image": { "type": "string" }, "backoffLimit": { "type": "integer", "minimum": 0 }, "ttlSecondsAfterFinished": { "type": "integer", "minimum": 0 }, + "resources": { "type": "object" }, + "podSecurityContext": { "type": "object" }, + "securityContext": { "type": "object" }, "checks": { "type": "array", "items": { diff --git a/charts/kates-chaos/values.yaml b/charts/kates-chaos/values.yaml index 9e05279e..e26d8d74 100644 --- a/charts/kates-chaos/values.yaml +++ b/charts/kates-chaos/values.yaml @@ -16,6 +16,11 @@ fullnameOverride: "" commonLabels: {} # -- Extra annotations added to every resource this chart renders commonAnnotations: {} +# -- Extra labels on every pod this chart renders (installer Job, GameDay Job, +# helm test pods). The chart's own labels win on a clash. litmus-core's +# operator and exporter pods take theirs from litmus-core.customLabels, which +# labels every resource that subchart renders. +podLabels: {} # ── Global settings ────────────────────────────────────────────────────────── global: @@ -67,6 +72,13 @@ litmus-core: image: repository: docker.io/litmuschaos/chaos-exporter tag: "3.28.0" + # The subchart sets none; the operator's (litmus-core.resources) it does. + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + memory: 128Mi # ── Chaos service account ──────────────────────────────────────────────────── # The identity ChaosEngines and experiment pods run as. Created by the RBAC @@ -143,9 +155,27 @@ experiments: image: "" backoffLimit: 3 ttlSecondsAfterFinished: 300 - resources: {} - podSecurityContext: {} - securityContext: {} + # -- Resources of the installer Job's containers, its two init containers + # included + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi + # -- Pod securityContext (PSS restricted). registry.k8s.io/kubectl runs as + # root unless told otherwise, so it runs as nobody (65534). + podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + # -- Container securityContext of every installer container (PSS restricted) + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] nodeSelector: {} tolerations: [] affinity: {} @@ -241,6 +271,23 @@ kyvernoPolicy: matchExpressions: - key: chaosUID operator: Exists + # -- Exceptions for this release's litmus-core operator and exporter, in its + # namespace, to rules of any Kyverno policy (the cluster's own included), + # whatever enabled says: a list of {policyName, ruleNames}. Rendered, where + # the kyverno.io/v2 API exists, as one PolicyException matching the + # workloads labelled app.kubernetes.io/instance= and + # app.kubernetes.io/name= (the chart's + # own pods pass on their own; chaos runner and experiment pods carry no + # instance label) as Pod, Deployment, ReplicaSet, StatefulSet, Job and + # CronJob, each rule with the autogen- and autogen-cronjob- + # Kyverno generates from it. A namespaced Policy is named /. + # Kyverno must accept exceptions from this namespace (its exceptionNamespace + # setting). The litmus-core operator takes no securityContext from values, + # so it is restricted only through the mutate rules above (enabled, mutate) + # or this; a cluster enforcing PSS restricted without them needs e.g. + # - policyName: pss-restricted + # ruleNames: [restricted] + exceptions: [] # ── GameDay validation ─────────────────────────────────────────────────────── # Runs a Job of read-only checks against the chaos infra. Checks are @@ -253,3 +300,47 @@ gameday: backoffLimit: 1 ttlSecondsAfterFinished: 600 checks: [] + # -- GameDay container resources + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi + # -- GameDay pod securityContext (PSS restricted; the kubectl image runs as + # root unless told otherwise) + podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + # -- GameDay container securityContext (PSS restricted) + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] + +# ── Helm tests (`helm test`) ───────────────────────────────────────────────── +# The two test pods run images.kubectl, PSS restricted by default. +tests: + # -- Test container resources + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi + # -- Test pod securityContext (the kubectl image runs as root unless told + # otherwise) + podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + # -- Test container securityContext + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] diff --git a/charts/kates-platform/Chart.yaml b/charts/kates-platform/Chart.yaml index 612dd733..44c9556f 100644 --- a/charts/kates-platform/Chart.yaml +++ b/charts/kates-platform/Chart.yaml @@ -2,10 +2,10 @@ apiVersion: v2 name: kates-platform description: Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure type: application -# Patch: the kates floor moves to 0.12.1, whose configmap points the Kates -# API's chaos, topology and conformity at networkPolicy.kafka.namespace, which -# this umbrella sets to the release namespace. -version: 0.8.2 +# Patch: the umbrella's Helm test pod passes Pod Security "restricted" and +# takes podLabels, test.resources, global.imageRegistry and pull secrets; the +# subchart floors move to the versions that pass restricted too. +version: 0.8.3 appVersion: "1.0.0" keywords: - kafka @@ -16,32 +16,40 @@ maintainers: - name: Platform Team dependencies: - name: kafka-cluster - version: ">=1.0.10 <2.0.0" + version: ">=1.0.12 <2.0.0" repository: "file://../kafka-cluster" condition: kafka-cluster.enabled - name: kates - version: ">=0.12.1 <1.0.0" + version: ">=0.13.0 <1.0.0" repository: "file://../kates" condition: kates.enabled - name: apicurio-registry - version: ">=0.4.0 <1.0.0" + version: ">=0.4.1 <1.0.0" repository: "file://../apicurio-registry" condition: apicurio-registry.enabled # Off by default: a mirror needs a second cluster to mean anything, and the # chart's own defaults are a loopback on krafter for smoke-testing. Enable it # with a real mirrors[].source — see charts/mirror-maker2/README.md. - name: mirror-maker2 - version: ">=0.11.6 <1.0.0" + version: ">=0.11.8 <1.0.0" repository: "file://../mirror-maker2" condition: mirror-maker2.enabled # Off by default: `kates deploy` installs Connect as its own release, in its # own namespace, and the platform runs without it. - name: connect-cluster - version: ">=2.1.8 <3.0.0" + version: ">=2.1.10 <3.0.0" repository: "file://../connect-cluster" condition: connect-cluster.enabled annotations: artifacthub.io/changes: | + - kind: changed + description: "ACTION REQUIRED — the subchart floors move to the versions this umbrella is rendered with: kafka-cluster 1.0.12, kates 0.13.0, apicurio-registry 0.4.1, mirror-maker2 0.11.8 and connect-cluster 2.1.10, whose pods pass Pod Security restricted and take podLabels, and of which kafka-cluster and kates read global.imagePullSecrets as plain names or {name: ...}. Rendering fails, naming the subchart, until charts/ holds them: run make platform-chart-deps before upgrading" + - kind: fixed + description: "Under Pod Security Standards restricted, enforced by a namespace label or a Kyverno policy, the umbrella's Helm test pod was refused: it set no seccomp profile. It now runs with seccompProfile RuntimeDefault, still as busybox's nobody user (65534) with no privilege escalation and no capabilities. Its requests and limits, the same as before, are a value now: test.resources" + - kind: added + description: "podLabels (default empty) reach the umbrella's Helm test pod, under its own labels, which win a clash; each subchart's pods take theirs from its own podLabels, such as kates.podLabels and kafka-cluster.podLabels. imagePullSecrets (default empty) are the test pod's pull secrets, after global.imagePullSecrets; an entry of either is a Secret name or {name: ...}" + - kind: changed + description: "A release that sets global.imageRegistry or global.imagePullSecrets for the subcharts now has them on the Helm test pod too: the registry is prepended to test.image (busybox:1.37 becomes /busybox:1.37), so a mirror has to carry busybox; a test.image that already names the mirror gets it twice, so drop it there" - kind: fixed description: "In any namespace but kafka, the platform's Kates API reached the brokers but its chaos, cluster topology, cluster alerts and conformity checks still looked for Kafka in namespace kafka: chaos refused every plan with No broker pods found, and every conformity check returned 409. The kates floor moves to 0.12.1, which sets those settings from kates.networkPolicy.kafka.namespace, already the release namespace here, and their cluster name from kates.kafka.clusterName (krafter). kafka-cluster.clusterName other than krafter needs kates.kafka.clusterName and kates.kafka.bootstrapServers set to match. Run `make platform-chart-deps` before upgrading" - kind: fixed diff --git a/charts/kates-platform/README.md b/charts/kates-platform/README.md index 4de0efd8..a27f0248 100644 --- a/charts/kates-platform/README.md +++ b/charts/kates-platform/README.md @@ -36,11 +36,15 @@ Any namespace works: Kafka and Kates both run in the release namespace, and the | `apicurio-registry.enabled` | `false` | Deploy Apicurio Registry (KafkaSQL storage against the `kafka-cluster` subchart) | | `mirror-maker2.enabled` | `false` | Deploy MirrorMaker 2 (needs a real `mirrors[].source`) | | `connect-cluster.enabled` | `false` | Deploy Kafka Connect next to Kafka (`kates deploy` installs it as its own release instead) | +| `test.image` | `busybox:1.37` | Image of the umbrella's Helm test pod (needs `nc`; runs as uid 65534, Pod Security "restricted"); `global.imageRegistry`, when set, is prepended with a slash | +| `test.resources` | 25m/32Mi requests, 100m/64Mi limits | Resources of the Helm test pod | +| `podLabels` | `{}` | Labels added to the Helm test pod; its own labels win a clash. Each subchart's pods take theirs from its own `podLabels` (`kates.podLabels`, `kafka-cluster.podLabels`, …) | +| `imagePullSecrets` | `[]` | Pull secrets of the Helm test pod, after `global.imagePullSecrets` (which kafka-cluster and kates read too; the other subcharts take their own imagePullSecrets); an entry is a name or `{name: …}` | Subchart values are validated by each subchart's own `values.schema.json`. ## Version pinning -`Chart.yaml` pins the subcharts with ranges (`>=1.0.10 <2.0.0` for kafka-cluster, `>=0.11.6 <1.0.0` for mirror-maker2, `>=2.1.8 <3.0.0` for connect-cluster, `>=0.12.1 <1.0.0` for kates, `>=0.4.0 <1.0.0` for apicurio-registry); each floor is the version the umbrella was last rendered with. The `file://` repository always resolves to the local working tree, and `Chart.lock` records the exact version at build time. +`Chart.yaml` pins the subcharts with ranges (`>=1.0.12 <2.0.0` for kafka-cluster, `>=0.11.8 <1.0.0` for mirror-maker2, `>=2.1.10 <3.0.0` for connect-cluster, `>=0.13.0 <1.0.0` for kates, `>=0.4.1 <1.0.0` for apicurio-registry); each floor is the version the umbrella was last rendered with. The `file://` repository always resolves to the local working tree, and `Chart.lock` records the exact version at build time. Helm installs whatever sits in `charts/` without checking it against those ranges, so the umbrella checks itself: rendering fails, naming the subchart, while `charts/` holds an enabled subchart outside its range. A `charts/` left from an older build would otherwise install its old subcharts, such as kates 0.9.1, whose ClusterRole read every Secret in the cluster. Run `make platform-chart-deps` to refresh it. diff --git a/charts/kates-platform/templates/_helpers.tpl b/charts/kates-platform/templates/_helpers.tpl index 5f0a806a..e18c5e95 100644 --- a/charts/kates-platform/templates/_helpers.tpl +++ b/charts/kates-platform/templates/_helpers.tpl @@ -14,3 +14,21 @@ app.kubernetes.io/managed-by: {{ .Release.Service }} app.kubernetes.io/part-of: kates-platform app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} + +{{/* +imagePullSecrets for the umbrella's own pod: global.imagePullSecrets, which +the subcharts read too, then imagePullSecrets. An entry is a name or +{name: …}, as kafka-cluster takes global.imagePullSecrets. +*/}} +{{- define "kates-platform.imagePullSecrets" -}} +{{- $names := list }} +{{- range concat ((.Values.global | default dict).imagePullSecrets | default list) (.Values.imagePullSecrets | default list) }} +{{- if kindIs "map" . }}{{ $names = append $names .name }}{{ else }}{{ $names = append $names . }}{{ end }} +{{- end }} +{{- with $names | uniq -}} +imagePullSecrets: + {{- range . }} + - name: {{ . | quote }} + {{- end }} +{{- end }} +{{- end }} diff --git a/charts/kates-platform/templates/tests/test-connection.yaml b/charts/kates-platform/templates/tests/test-connection.yaml index e2b9ea7d..1ee61ad3 100644 --- a/charts/kates-platform/templates/tests/test-connection.yaml +++ b/charts/kates-platform/templates/tests/test-connection.yaml @@ -1,41 +1,40 @@ {{- if (index .Values "kafka-cluster").enabled }} {{- $clusterName := (index .Values "kafka-cluster").clusterName | default "krafter" }} +{{- /* kates.io/test-pod is the label kafka-cluster's broker NetworkPolicy + admits test pods by. These win a clash with podLabels. */}} +{{- $labels := dict "app.kubernetes.io/name" "kates-platform" "app.kubernetes.io/instance" .Release.Name "app.kubernetes.io/managed-by" .Release.Service "helm.sh/chart" (printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_") "kates.io/test-pod" "true" }} apiVersion: v1 kind: Pod metadata: name: {{ .Release.Name }}-platform-test namespace: {{ .Release.Namespace }} labels: - app.kubernetes.io/name: kates-platform - # The label kafka-cluster's broker NetworkPolicy admits test pods by. - kates.io/test-pod: "true" - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} - helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} + {{- toYaml (merge $labels (.Values.podLabels | default dict)) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with include "kates-platform.imagePullSecrets" . }}{{ . | nindent 2 }}{{ end }} + # busybox runs as root; 65534 is its nobody user. securityContext: runAsNonRoot: true runAsUser: 65534 + seccompProfile: + type: RuntimeDefault containers: - name: platform-test - image: {{ .Values.test.image }} + image: "{{ with (.Values.global | default dict).imageRegistry }}{{ . }}/{{ end }}{{ .Values.test.image }}" securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL + {{- with .Values.test.resources }} resources: - requests: - memory: 32Mi - cpu: 25m - limits: - memory: 64Mi - cpu: 100m + {{- toYaml . | nindent 8 }} + {{- end }} command: - sh - -c diff --git a/charts/kates-platform/values.schema.json b/charts/kates-platform/values.schema.json index 2e1dc752..a5b77a8b 100644 --- a/charts/kates-platform/values.schema.json +++ b/charts/kates-platform/values.schema.json @@ -25,9 +25,29 @@ "properties": { "image": { "type": "string" + }, + "resources": { + "type": "object" } } }, + "imagePullSecrets": { + "type": "array", + "items": { + "oneOf": [ + { "type": "string", "minLength": 1 }, + { + "type": "object", + "properties": { "name": { "type": "string", "minLength": 1 } }, + "required": ["name"] + } + ] + } + }, + "podLabels": { + "type": "object", + "additionalProperties": { "type": "string" } + }, "mirror-maker2": { "type": "object", "properties": { diff --git a/charts/kates-platform/values.yaml b/charts/kates-platform/values.yaml index 7ff04b31..9b6a0445 100644 --- a/charts/kates-platform/values.yaml +++ b/charts/kates-platform/values.yaml @@ -46,6 +46,25 @@ mirror-maker2: connect-cluster: enabled: false -# Image used by the helm test connectivity pod (needs `nc`) +# The umbrella's own pod is the helm test connectivity pod below. Its pull +# secrets are global.imagePullSecrets, which kafka-cluster and kates read too +# (the other subcharts take their own imagePullSecrets), then imagePullSecrets; +# an entry is a name (regcred) or {name: regcred}. +# global.imageRegistry, when set, is prepended to its image with a slash. +imagePullSecrets: [] +# Labels added to the helm test pod. Its own labels win a clash. The +# subcharts' pods take theirs from kates.podLabels, kafka-cluster.podLabels +# and so on. +podLabels: {} + test: + # Image of the helm test connectivity pod (needs `nc`). It runs as busybox's + # nobody user (65534). image: busybox:1.37 + resources: + requests: + memory: 32Mi + cpu: 25m + limits: + memory: 64Mi + cpu: 100m diff --git a/charts/kates/Chart.yaml b/charts/kates/Chart.yaml index de81945c..321bace6 100644 --- a/charts/kates/Chart.yaml +++ b/charts/kates/Chart.yaml @@ -2,9 +2,9 @@ apiVersion: v2 name: kates description: Kates — Kafka Advanced Testing & Engineering Suite type: application -# Patch: appVersion and the image tag move to 1.28.0, the tag the v1.28.0 -# release publishes. Templates are otherwise unchanged. -version: 0.12.2 +# Minor: global.imageRegistry now reaches every image the chart renders, so an +# image value that already names the mirror gets it twice. +version: 0.13.0 # The non-blocking health endpoints this chart's probes now point at come from # a BUILD-TIME property (quarkus.messaging.health.enabled=false), so they only # exist in the image built from this branch. Keep appVersion and @@ -30,6 +30,18 @@ annotations: artifacthub.io/changes: | - kind: changed description: "appVersion 1.28.0 and the image tag move to 1.28.0. The Kates API changes since 1.27.0: a conformity check runs with only the namespace-scoped Role that rbac.scope=namespace grants, taking the zones from the EndpointSlices of Strimzi's headless Service when it may not read nodes, and answers 503 when Kubernetes refuses it the Kafka CR (#439); a fault that names no targetNamespace is aimed at kates.chaos.kafka.namespace instead of the literal kafka, so the safety guard no longer refuses every such step with Kafka elsewhere (#442). Templates are otherwise unchanged." + - kind: fixed + description: "Under Pod Security Standards restricted, enforced by a namespace label or a Kyverno policy, the helm test pods and PostgreSQL were refused (no seccomp profile), and the Kates API pod's wait-for-postgres init container had no CPU or memory requests or memory limit. With default values, and with backup, cleanup and migration enabled, every pod the chart renders now passes PSS restricted with requests and a memory limit on every container. The test pods run with securityContext and containerSecurityContext, as the API pod does, and the backup job's container takes containerSecurityContext, which it lacked. New values: helmTests.image and helmTests.resources (not tests, which holds the benchmark defaults), cleanup.resources, initContainers.waitForPostgres.securityContext and initContainers.waitForPostgres.resources, the init container's in the migration Job too; postgresql.securityContext gains seccompProfile RuntimeDefault" + - kind: changed + description: "ACTION REQUIRED for a release that sets global.imageRegistry — it is now prepended to every image the chart renders, not only the Kates API's: PostgreSQL, the wait-for-postgres init container, the backup, cleanup and migration jobs, the helm test pods and chaos.stressImage. An image value that already names the mirror gets it twice: drop the mirror from it. global.imagePullSecrets, which nothing read, and imagePullSecrets now reach every pod the chart renders, jobs and tests included. An entry of either list may be a Secret name or {name: ...}, as kafka-cluster takes global.imagePullSecrets, so a kates-platform global of [regcred] reaches the Kates pods as {name: regcred}" + - kind: added + description: "chaos.stressImage (default docker.io/polinux/stress:1.0.4, the backend's own) sets KATES_CHAOS_STRESS_IMAGE, the image CPU_STRESS and IO_STRESS run in the target pod, with global.imageRegistry prepended, so a cluster that pulls through a mirror pulls it there too. It needs a Kates API image newer than 1.28.0: 1.28.0 and earlier ignore it and pull from Docker Hub" + - kind: added + description: "podLabels reach every pod the chart renders: the Kates API, PostgreSQL, the backup, cleanup and migration jobs and the helm test pods. The chart's own labels win a clash, so no selector changes. The PostgreSQL pod also carries app.kubernetes.io/instance now, so it restarts once on upgrade" + - kind: added + description: "kyvernoPolicy.exceptions, a list of {policyName, ruleNames}, renders one kyverno.io/v2 PolicyException in the release namespace exempting this release's pods, Deployment, ReplicaSets, StatefulSet, Jobs and CronJobs from those rules of any Kyverno policy, the cluster's own included, along with the autogen- and autogen-cronjob- rules Kyverno generates from each. It is namespaced, so rbac.scope=namespace renders it, and it does not need kyvernoPolicy.enabled. The CronJobs' Jobs now carry the chart's labels so it matches them. Kyverno must accept exceptions from the release namespace" + - kind: fixed + description: "With rbac.scope=namespace, kyvernoPolicy.enabled still rendered the chart's ClusterPolicies, and kyvernoPolicy.policyExceptions PolicyExceptions in other namespaces, which an account with edit in the release namespace cannot create. Under scope namespace they render nothing; kyvernoPolicy.exceptions covers the release's own pods there. Under scope cluster they are unchanged" - kind: fixed description: "With Kafka outside namespace kafka, or named other than krafter, chaos, the cluster topology, cluster alerts and conformity checks looked for Kafka in kafka/krafter: the configmap never set kates.chaos.kafka.namespace, kates.chaos.kafka.cluster or kates.topology.kafka-cluster, and set kates.topology.kafka-namespace only with rbac.scope=namespace. Chaos refused every plan with No broker pods found, and every conformity check returned 409. The configmap now sets KATES_CHAOS_KAFKA_NAMESPACE and KATES_TOPOLOGY_KAFKA_NAMESPACE from networkPolicy.kafka.namespace (through tpl, with the policy on or off), whatever rbac.scope says, and KATES_CHAOS_KAFKA_CLUSTER and KATES_TOPOLOGY_KAFKA_CLUSTER from kafka.clusterName (new, default krafter). The defaults are the backend's own, so a release that keeps them renders the same settings; an extraEnv entry for these variables still wins. Upgrading: a release that set networkPolicy.kafka.namespace to anything but the namespace Kafka runs in now points chaos and topology there too" - kind: added diff --git a/charts/kates/README.md b/charts/kates/README.md index ea48efe1..9f511b88 100644 --- a/charts/kates/README.md +++ b/charts/kates/README.md @@ -35,6 +35,8 @@ All configuration is in [values.yaml](values.yaml). Key sections: | `image.repository` | `ghcr.io/bmscomp/kates` | Container image name | | `image.tag` | `1.28.0` | Container image tag, pinned in `values.yaml` to match the chart's `appVersion`. Keep it at 1.22.0 or newer: the probes use health endpoints older images lack | | `image.pullPolicy` | `IfNotPresent` | Image pull policy | +| `global.imageRegistry` | `""` | Prepended, with a slash, to every image the chart renders: the API, PostgreSQL, the init container, the jobs, the helm test pods and `chaos.stressImage` | +| `imagePullSecrets`, `global.imagePullSecrets` | `[]` | Pull secrets, both lists, on every pod the chart renders. An entry is a Secret name (`regcred`) or `{name: regcred}` | | `replicaCount` | `1` | Number of Kates API pods. Must be 1; see [One replica](#one-replica) | | `kafka.bootstrapServers` | `krafter-kafka-bootstrap.kafka.svc:9092` | Kafka bootstrap address. This, `kafka.topicNamespace`, `trogdor.coordinatorUrl` and `networkPolicy.kafka.namespace` go through `tpl`, so a value may name `{{ .Release.Namespace }}`, as kates-platform does | | `kafka.topicNamespace` | `""` | Namespace for CDC test `KafkaTopic` CRs (`""` = auto-detect) | @@ -44,6 +46,7 @@ All configuration is in [values.yaml](values.yaml). Key sections: | `engine.reaperGraceMs` | `300000` | Grace after a `RUNNING` or `STOPPING` run's planned duration before the reaper fails it | | `engine.submissionTimeoutMs` | `120000` | Maximum time a run may stay `PENDING` before submission is considered stranded | | `chaos.provider` | `""` | Chaos provider every fault goes through: `litmus-crd`, `kubernetes`, `hybrid` or `noop`. Empty keeps the backend's default, `litmus-crd`. `kates deploy --with-chaos=false` sets `noop` when no chaos release is installed. See [Chaos permissions](#chaos-permissions) | +| `chaos.stressImage` | `docker.io/polinux/stress:1.0.4` | Image `CPU_STRESS` and `IO_STRESS` run in the target pod (`KATES_CHAOS_STRESS_IMAGE`), with `global.imageRegistry` prepended. Needs a Kates API image newer than 1.28.0 | | `prometheus.url` | `http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090` | Prometheus the backend queries for a disruption's Kafka metrics: kube-prometheus-stack's Service for `charts/monitoring` installed as release `monitoring` in namespace `monitoring`, as `kates deploy` does. After `make monitoring`, which uses namespace `kafka`, set `http://monitoring-kube-prometheus-prometheus.kafka.svc:9090` and `networkPolicy.prometheus.namespace=kafka` | ### Networking @@ -86,10 +89,13 @@ that namespace (`kates/docs/deployment.md` runs one in `kates`) needs a | Parameter | Default | Description | |-----------|---------|-------------| -| `securityContext.runAsNonRoot` | `true` | Run pod as non-root | -| `securityContext.runAsUser` | `1000` | Pod UID | +| `securityContext.runAsNonRoot` | `true` | Run pod as non-root. `securityContext` and `containerSecurityContext` apply to the API pod, the jobs and the helm test pods | +| `securityContext.runAsUser` | `100` | Pod UID | +| `securityContext.seccompProfile.type` | `RuntimeDefault` | Seccomp profile, which Pod Security Standards restricted requires | | `containerSecurityContext.readOnlyRootFilesystem` | `true` | Read-only root FS | | `containerSecurityContext.allowPrivilegeEscalation` | `false` | Block privilege escalation | +| `postgresql.securityContext` | UID and `fsGroup` 999, seccomp `RuntimeDefault` | PostgreSQL pod security context | +| `kyvernoPolicy.exceptions` | `[]` | `{policyName, ruleNames}` entries: one Kyverno `PolicyException` in the release namespace exempts this release's pods and controllers from those rules of any policy, `autogen-` and `autogen-cronjob-` variants included. Needs no `kyvernoPolicy.enabled`, and Kyverno must accept exceptions from the namespace | | `serviceAccount.create` | `true` | Create a ServiceAccount | | `rbac.create` | `true` | Create the RBAC objects `rbac.scope` calls for | | `rbac.scope` | `cluster` | `cluster`: the backend's ClusterRole/ClusterRoleBinding (Litmus CRDs, Strimzi resources, cluster reads; no Secrets). `namespace`: nothing cluster-scoped; see [Namespace-only install](#namespace-only-install) | @@ -100,7 +106,7 @@ that namespace (`kates/docs/deployment.md` runs one in `kates`) needs a ### Namespace-only install -`rbac.scope=namespace` installs Kates with no cluster-scoped object. Performance and `INTEGRITY` tests make no Kubernetes call. Chaos is off: `KATES_CHAOS_PROVIDER` is `noop` unless `chaos.provider` is set. A conformity check reads the Kafka namespace, `networkPolicy.kafka.namespace` (the chart sets `KATES_TOPOLOGY_KAFKA_NAMESPACE` from it), through one read-only Role: +`rbac.scope=namespace` installs Kates with no cluster-scoped object: the chart's Kyverno ClusterPolicies and `kyvernoPolicy.policyExceptions` render nothing, while `kyvernoPolicy.exceptions` renders in the release namespace. Performance and `INTEGRITY` tests make no Kubernetes call. Chaos is off: `KATES_CHAOS_PROVIDER` is `noop` unless `chaos.provider` is set. A conformity check reads the Kafka namespace, `networkPolicy.kafka.namespace` (the chart sets `KATES_TOPOLOGY_KAFKA_NAMESPACE` from it), through one read-only Role: | Resource | Verbs | |----------|-------| @@ -242,6 +248,9 @@ that — `image.tag` does. | `cleanup.retentionDays` | `30` | Delete finished runs created more than this many days ago | | `cleanup.statuses` | `[DONE, FAILED]` | Which finished runs to delete: `DONE`, `FAILED` or both. A cancelled run is stored `FAILED` | | `cleanup.image` | `curlimages/curl:8.7.1` | Image the job runs `curl` from | +| `cleanup.resources` | `25m`/`32Mi`, limit `64Mi` | Cleanup job resources | +| `helmTests.image` | `curlimages/curl:8.7.1` | Image the `helm test` pods run `curl` from | +| `helmTests.resources` | `25m`/`32Mi`, limit `64Mi` | Resources of each `helm test` pod | ### Test run cleanup @@ -258,9 +267,11 @@ Upgrading from 0.10.9 or earlier: the job there never deleted a run. It asked fo | `extraEnv` | `[]` | Extra environment variables | | `extraVolumes` | `[]` | Extra volumes | | `extraVolumeMounts` | `[]` | Extra volume mounts | -| `podLabels` | `{}` | Extra pod labels | +| `podLabels` | `{}` | Extra labels on every pod the chart renders, jobs and test pods included; the chart's own labels win a clash | | `podAnnotations` | `{}` | Extra pod annotations | | `initContainers.waitForPostgres.image` | `busybox:1.36` | Init container image | +| `initContainers.waitForPostgres.securityContext` | UID 65534, no privilege escalation, `ALL` dropped | Init container security context (`null`: `containerSecurityContext`; `{}` keeps these defaults) | +| `initContainers.waitForPostgres.resources` | `10m`/`16Mi`, limit `32Mi` | Init container resources, in the API pod and the migration Job | ## Example: Local Kind Cluster diff --git a/charts/kates/templates/_helpers.tpl b/charts/kates/templates/_helpers.tpl index adcafeb7..fce636f6 100644 --- a/charts/kates/templates/_helpers.tpl +++ b/charts/kates/templates/_helpers.tpl @@ -107,9 +107,40 @@ Resolve the full image reference, prepending global.imageRegistry if set. Usage: {{ include "kates.image" . }} */}} {{- define "kates.image" -}} -{{- if and .Values.global .Values.global.imageRegistry -}} -{{ .Values.global.imageRegistry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }} -{{- else -}} -{{ .Values.image.repository }}:{{ .Values.image.tag }} +{{- include "kates.imageRef" (dict "image" (printf "%s:%s" .Values.image.repository .Values.image.tag) "ctx" .) -}} +{{- end }} + +{{/* +Any image the chart renders, with global.imageRegistry prepended if set. +Usage: {{ include "kates.imageRef" (dict "image" .Values.backup.image "ctx" .) }} +*/}} +{{- define "kates.imageRef" -}} +{{- with (.ctx.Values.global | default dict).imageRegistry }}{{ . }}/{{ end }}{{ .image }} +{{- end }} + +{{/* +imagePullSecrets for every pod: global.imagePullSecrets and imagePullSecrets. +Each entry may be a name (regcred) or {name: regcred}, as kafka-cluster takes +global.imagePullSecrets; kates-platform hands the same global to both. +*/}} +{{- define "kates.imagePullSecrets" -}} +{{- $names := list -}} +{{- range concat ((.Values.global | default dict).imagePullSecrets | default list) (.Values.imagePullSecrets | default list) -}} +{{- if kindIs "map" . }}{{ $names = append $names .name }}{{ else }}{{ $names = append $names . }}{{ end -}} {{- end -}} +{{- with $names | uniq -}} +imagePullSecrets: +{{- range . }} + - name: {{ . | quote }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +A pod template's labels: the chart's own (.labels, as YAML) plus podLabels. +The chart's own win a clash, so no selector changes. +Usage: {{ include "kates.podLabels" (dict "labels" (include "kates.selectorLabels" .) "ctx" .) }} +*/}} +{{- define "kates.podLabels" -}} +{{- toYaml (merge (fromYaml .labels) (.ctx.Values.podLabels | default dict)) }} {{- end }} diff --git a/charts/kates/templates/backup-cronjob.yaml b/charts/kates/templates/backup-cronjob.yaml index d06fed60..7eee74c0 100644 --- a/charts/kates/templates/backup-cronjob.yaml +++ b/charts/kates/templates/backup-cronjob.yaml @@ -13,21 +13,30 @@ spec: successfulJobsHistoryLimit: {{ .Values.backup.retention }} failedJobsHistoryLimit: 3 jobTemplate: + metadata: + labels: + {{- include "kates.labels" . | nindent 8 }} + app.kubernetes.io/component: backup spec: template: metadata: labels: - {{- include "kates.selectorLabels" . | nindent 12 }} - app.kubernetes.io/component: backup + {{- $own := printf "%s\napp.kubernetes.io/component: backup" (include "kates.selectorLabels" .) }} + {{- include "kates.podLabels" (dict "labels" $own "ctx" .) | nindent 12 }} spec: restartPolicy: OnFailure + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 10 }}{{ end }} {{- with .Values.securityContext }} securityContext: {{- toYaml . | nindent 12 }} {{- end }} containers: - name: pg-backup - image: {{ .Values.backup.image | quote }} + image: {{ include "kates.imageRef" (dict "image" .Values.backup.image "ctx" .) | quote }} + {{- with .Values.containerSecurityContext }} + securityContext: + {{- toYaml . | nindent 14 }} + {{- end }} command: - sh - -c diff --git a/charts/kates/templates/cleanup-cronjob.yaml b/charts/kates/templates/cleanup-cronjob.yaml index 583d5f6a..625d46bf 100644 --- a/charts/kates/templates/cleanup-cronjob.yaml +++ b/charts/kates/templates/cleanup-cronjob.yaml @@ -18,6 +18,10 @@ spec: successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: + metadata: + labels: + {{- include "kates.labels" . | nindent 8 }} + app.kubernetes.io/component: cleanup spec: # A missing or wrong key, or an API without DELETE /api/tests, fails # every attempt the same way. Two retries cover a restarting API; the @@ -31,17 +35,18 @@ spec: # egress to port 8080, so a job pod carrying them could not reach # the API at all. The Service and the PodDisruptionBudget select # on them too. - {{- include "kates.labels" . | nindent 12 }} - app.kubernetes.io/component: cleanup + {{- $own := printf "%s\napp.kubernetes.io/component: cleanup" (include "kates.labels" .) }} + {{- include "kates.podLabels" (dict "labels" $own "ctx" .) | nindent 12 }} spec: restartPolicy: OnFailure + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 10 }}{{ end }} {{- with .Values.securityContext }} securityContext: {{- toYaml . | nindent 12 }} {{- end }} containers: - name: cleanup - image: {{ .Values.cleanup.image | quote }} + image: {{ include "kates.imageRef" (dict "image" .Values.cleanup.image "ctx" .) | quote }} {{- with .Values.containerSecurityContext }} securityContext: {{- toYaml . | nindent 14 }} @@ -55,13 +60,10 @@ spec: key: {{ $keyName }} optional: true {{- end }} + {{- with .Values.cleanup.resources }} resources: - requests: - memory: "32Mi" - cpu: "25m" - limits: - memory: "64Mi" - cpu: "100m" + {{- toYaml . | nindent 14 }} + {{- end }} command: - sh - -c diff --git a/charts/kates/templates/configmap.yaml b/charts/kates/templates/configmap.yaml index 8f575c9d..f2b5c8dc 100644 --- a/charts/kates/templates/configmap.yaml +++ b/charts/kates/templates/configmap.yaml @@ -186,6 +186,9 @@ data: {{- with .Values.chaos.provider | default (ternary "noop" "" $namespaced) }} KATES_CHAOS_PROVIDER: {{ . | quote }} {{- end }} + {{- with .Values.chaos.stressImage }} + KATES_CHAOS_STRESS_IMAGE: {{ include "kates.imageRef" (dict "image" . "ctx" $) | quote }} + {{- end }} {{- with .Values.faultTolerance }} COM_BMSCOMP_KATES_SERVICE_TOPICSERVICE_CREATETOPIC_TIMEOUT_VALUE: {{ .topicService.timeoutMs | default "35000" | quote }} diff --git a/charts/kates/templates/deployment.yaml b/charts/kates/templates/deployment.yaml index 9f957730..99ff9096 100644 --- a/charts/kates/templates/deployment.yaml +++ b/charts/kates/templates/deployment.yaml @@ -27,10 +27,7 @@ spec: template: metadata: labels: - {{- include "kates.selectorLabels" . | nindent 8 }} - {{- with .Values.podLabels }} - {{- toYaml . | nindent 8 }} - {{- end }} + {{- include "kates.podLabels" (dict "labels" (include "kates.selectorLabels" .) "ctx" .) | nindent 8 }} annotations: checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }} @@ -43,10 +40,7 @@ spec: {{- end }} serviceAccountName: {{ include "kates.serviceAccountName" . }} terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds | default 60 }} - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 6 }}{{ end }} {{- if .Values.dnsPolicy }} dnsPolicy: {{ .Values.dnsPolicy }} {{- end }} @@ -66,17 +60,17 @@ spec: initContainers: {{- if .Values.postgresql.enabled }} - name: wait-for-postgres - image: {{ .Values.initContainers.waitForPostgres.image | quote }} + image: {{ include "kates.imageRef" (dict "image" .Values.initContainers.waitForPostgres.image "ctx" .) | quote }} imagePullPolicy: {{ .Values.initContainers.waitForPostgres.pullPolicy | default "IfNotPresent" }} command: ['sh', '-c', 'until nc -z {{ include "kates.postgresql.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }} 5432; do echo "Waiting for PostgreSQL..."; sleep 2; done'] + {{- with .Values.initContainers.waitForPostgres.securityContext | default .Values.containerSecurityContext }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - runAsNonRoot: true - runAsUser: 65534 - capabilities: - drop: - - ALL + {{- toYaml . | nindent 10 }} + {{- end }} + {{- with .Values.initContainers.waitForPostgres.resources }} + resources: + {{- toYaml . | nindent 10 }} + {{- end }} volumeMounts: - name: tmp mountPath: /tmp diff --git a/charts/kates/templates/kyverno-image-verification.yaml b/charts/kates/templates/kyverno-image-verification.yaml index 2ac8b966..2244cd89 100644 --- a/charts/kates/templates/kyverno-image-verification.yaml +++ b/charts/kates/templates/kyverno-image-verification.yaml @@ -1,4 +1,4 @@ -{{- if and .Values.kyvernoPolicy.enabled .Values.kyvernoPolicy.cosign.enabled (.Capabilities.APIVersions.Has "kyverno.io/v1") }} +{{- if and .Values.kyvernoPolicy.enabled .Values.kyvernoPolicy.cosign.enabled (ne .Values.rbac.scope "namespace") (.Capabilities.APIVersions.Has "kyverno.io/v1") }} {{- if .Values.kyvernoPolicy.cosign.publicKey }} apiVersion: kyverno.io/v1 kind: ClusterPolicy diff --git a/charts/kates/templates/kyverno-network-policies.yaml b/charts/kates/templates/kyverno-network-policies.yaml index 69c94752..b3a23194 100644 --- a/charts/kates/templates/kyverno-network-policies.yaml +++ b/charts/kates/templates/kyverno-network-policies.yaml @@ -1,4 +1,4 @@ -{{- if and .Values.kyvernoPolicy.enabled .Values.kyvernoPolicy.networkPolicyGeneration.enabled (.Capabilities.APIVersions.Has "kyverno.io/v1") }} +{{- if and .Values.kyvernoPolicy.enabled .Values.kyvernoPolicy.networkPolicyGeneration.enabled (ne .Values.rbac.scope "namespace") (.Capabilities.APIVersions.Has "kyverno.io/v1") }} apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: diff --git a/charts/kates/templates/kyverno-policies.yaml b/charts/kates/templates/kyverno-policies.yaml index 619e3b98..ac3d8f14 100644 --- a/charts/kates/templates/kyverno-policies.yaml +++ b/charts/kates/templates/kyverno-policies.yaml @@ -1,4 +1,4 @@ -{{- if and .Values.kyvernoPolicy.enabled (.Capabilities.APIVersions.Has "kyverno.io/v1") }} +{{- if and .Values.kyvernoPolicy.enabled (ne .Values.rbac.scope "namespace") (.Capabilities.APIVersions.Has "kyverno.io/v1") }} apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: @@ -200,7 +200,7 @@ spec: {{- end }} {{- end }} --- -{{- if and .Values.kyvernoPolicy.enabled (.Capabilities.APIVersions.Has "kyverno.io/v1") }} +{{- if and .Values.kyvernoPolicy.enabled (ne .Values.rbac.scope "namespace") (.Capabilities.APIVersions.Has "kyverno.io/v1") }} apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: diff --git a/charts/kates/templates/kyverno-policy-exceptions.yaml b/charts/kates/templates/kyverno-policy-exceptions.yaml index 58d91824..edf49cf4 100644 --- a/charts/kates/templates/kyverno-policy-exceptions.yaml +++ b/charts/kates/templates/kyverno-policy-exceptions.yaml @@ -1,4 +1,6 @@ -{{- if and .Values.kyvernoPolicy.enabled .Values.kyvernoPolicy.policyExceptions.enabled (.Capabilities.APIVersions.Has "kyverno.io/v2") }} +{{- /* Exceptions to this chart's own ClusterPolicy, in other namespaces: like + the policy, nothing under rbac.scope=namespace. */}} +{{- if and .Values.kyvernoPolicy.enabled .Values.kyvernoPolicy.policyExceptions.enabled (ne .Values.rbac.scope "namespace") (.Capabilities.APIVersions.Has "kyverno.io/v2") }} {{- range $ns := .Values.kyvernoPolicy.policyExceptions.namespaces }} {{- range $policy := $.Values.kyvernoPolicy.policyExceptions.exemptPolicies }} --- @@ -35,3 +37,41 @@ spec: {{- end }} {{- end }} {{- end }} +{{- /* kyvernoPolicy.exceptions: this release's own workloads, in its own + namespace, exempt from rules of any policy, the cluster's included. */}} +{{- with .Values.kyvernoPolicy.exceptions }} +{{- if $.Capabilities.APIVersions.Has "kyverno.io/v2" }} +--- +apiVersion: kyverno.io/v2 +kind: PolicyException +metadata: + name: {{ printf "%s-exceptions" (include "kates.fullname" $) | trunc 63 | trimSuffix "-" }} + namespace: {{ $.Release.Namespace }} + labels: + {{- include "kates.labels" $ | nindent 4 }} +spec: + exceptions: + {{- range . }} + - policyName: {{ .policyName | quote }} + ruleNames: + {{- range .ruleNames }} + {{- /* with the rules Kyverno generates from it for controllers and CronJobs */}} + - {{ . | quote }} + - {{ printf "autogen-%s" . | quote }} + - {{ printf "autogen-cronjob-%s" . | quote }} + {{- end }} + {{- end }} + match: + any: + - resources: + kinds: [Pod, Deployment, ReplicaSet, StatefulSet, Job, CronJob] + namespaces: [{{ $.Release.Namespace | quote }}] + selector: + matchLabels: + app.kubernetes.io/instance: {{ $.Release.Name | quote }} + matchExpressions: + - key: app.kubernetes.io/name + operator: In + values: [{{ include "kates.name" $ | quote }}, {{ include "kates.postgresql.fullname" $ | quote }}] +{{- end }} +{{- end }} diff --git a/charts/kates/templates/migration-job.yaml b/charts/kates/templates/migration-job.yaml index faa907cb..2b04fe1f 100644 --- a/charts/kates/templates/migration-job.yaml +++ b/charts/kates/templates/migration-job.yaml @@ -16,10 +16,11 @@ spec: template: metadata: labels: - {{- include "kates.selectorLabels" . | nindent 8 }} - app.kubernetes.io/component: migration + {{- $own := printf "%s\napp.kubernetes.io/component: migration" (include "kates.selectorLabels" .) }} + {{- include "kates.podLabels" (dict "labels" $own "ctx" .) | nindent 8 }} spec: restartPolicy: OnFailure + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 6 }}{{ end }} {{- with .Values.securityContext }} securityContext: {{- toYaml . | nindent 8 }} @@ -27,27 +28,24 @@ spec: {{- if .Values.postgresql.enabled }} initContainers: - name: wait-for-postgres - image: {{ .Values.initContainers.waitForPostgres.image | quote }} + image: {{ include "kates.imageRef" (dict "image" .Values.initContainers.waitForPostgres.image "ctx" .) | quote }} imagePullPolicy: {{ .Values.initContainers.waitForPostgres.pullPolicy | default "IfNotPresent" }} command: ['sh', '-c', 'until nc -z {{ include "kates.postgresql.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }} 5432; do echo "Waiting for PostgreSQL..."; sleep 2; done'] - {{- with .Values.containerSecurityContext }} + {{- with .Values.initContainers.waitForPostgres.securityContext | default .Values.containerSecurityContext }} securityContext: {{- toYaml . | nindent 10 }} {{- end }} + {{- with .Values.initContainers.waitForPostgres.resources }} resources: - requests: - memory: "16Mi" - cpu: "10m" - limits: - memory: "32Mi" - cpu: "50m" + {{- toYaml . | nindent 10 }} + {{- end }} volumeMounts: - name: tmp mountPath: /tmp {{- end }} containers: - name: migrate - image: {{ .Values.migration.image | default (printf "%s:%s" .Values.image.repository .Values.image.tag) | quote }} + image: {{ include "kates.imageRef" (dict "image" (.Values.migration.image | default (printf "%s:%s" .Values.image.repository .Values.image.tag)) "ctx" .) | quote }} env: {{- include "kates.dbEnv" . | nindent 8 }} {{- if .Values.migration.command }} diff --git a/charts/kates/templates/postgresql.yaml b/charts/kates/templates/postgresql.yaml index b8521137..b42c67fa 100644 --- a/charts/kates/templates/postgresql.yaml +++ b/charts/kates/templates/postgresql.yaml @@ -19,9 +19,11 @@ spec: template: metadata: labels: - app.kubernetes.io/name: {{ include "kates.postgresql.fullname" . }} - app.kubernetes.io/component: database + {{- /* instance is not in the selector: it lets kyvernoPolicy.exceptions match this pod */}} + {{- $own := printf "app.kubernetes.io/name: %s\napp.kubernetes.io/component: database\napp.kubernetes.io/instance: %s" (include "kates.postgresql.fullname" .) .Release.Name }} + {{- include "kates.podLabels" (dict "labels" $own "ctx" .) | nindent 8 }} spec: + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 6 }}{{ end }} {{- with .Values.postgresql.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} @@ -40,7 +42,7 @@ spec: {{- end }} containers: - name: postgresql - image: "{{ .Values.postgresql.image.repository }}:{{ .Values.postgresql.image.tag }}" + image: {{ include "kates.imageRef" (dict "image" (printf "%s:%s" .Values.postgresql.image.repository .Values.postgresql.image.tag) "ctx" .) | quote }} {{- with .Values.postgresql.containerSecurityContext }} securityContext: {{- toYaml . | nindent 10 }} diff --git a/charts/kates/templates/tests/test-api.yaml b/charts/kates/templates/tests/test-api.yaml index 0e709b04..4b83a4ce 100644 --- a/charts/kates/templates/tests/test-api.yaml +++ b/charts/kates/templates/tests/test-api.yaml @@ -3,32 +3,29 @@ kind: Pod metadata: name: {{ include "kates.fullname" . }}-test-api labels: - {{- include "kates.labels" . | nindent 4 }} + {{- include "kates.podLabels" (dict "labels" (include "kates.labels" .) "ctx" .) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 2 }}{{ end }} + {{- with .Values.securityContext }} securityContext: - runAsNonRoot: true - runAsUser: 100 - fsGroup: 100 + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: api-check - image: curlimages/curl:8.7.1 + {{- $t := .Values.helmTests | default dict }} + image: {{ include "kates.imageRef" (dict "image" ($t.image | default "curlimages/curl:8.7.1") "ctx" .) | quote }} + {{- with .Values.containerSecurityContext }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: - - ALL + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with $t.resources }} resources: - requests: - memory: "32Mi" - cpu: "25m" - limits: - memory: "64Mi" - cpu: "100m" + {{- toYaml . | nindent 6 }} + {{- end }} command: - sh - -c diff --git a/charts/kates/templates/tests/test-health.yaml b/charts/kates/templates/tests/test-health.yaml index b0bb0f7a..34bd4b68 100644 --- a/charts/kates/templates/tests/test-health.yaml +++ b/charts/kates/templates/tests/test-health.yaml @@ -3,32 +3,29 @@ kind: Pod metadata: name: {{ include "kates.fullname" . }}-test labels: - {{- include "kates.labels" . | nindent 4 }} + {{- include "kates.podLabels" (dict "labels" (include "kates.labels" .) "ctx" .) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 2 }}{{ end }} + {{- with .Values.securityContext }} securityContext: - runAsNonRoot: true - runAsUser: 100 - fsGroup: 100 + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: health-check - image: curlimages/curl:8.7.1 + {{- $t := .Values.helmTests | default dict }} + image: {{ include "kates.imageRef" (dict "image" ($t.image | default "curlimages/curl:8.7.1") "ctx" .) | quote }} + {{- with .Values.containerSecurityContext }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: - - ALL + {{- toYaml . | nindent 6 }} + {{- end }} + {{- with $t.resources }} resources: - requests: - memory: "32Mi" - cpu: "25m" - limits: - memory: "64Mi" - cpu: "100m" + {{- toYaml . | nindent 6 }} + {{- end }} command: - sh - -c diff --git a/charts/kates/templates/tests/test-kafka.yaml b/charts/kates/templates/tests/test-kafka.yaml index d2f62812..84263f43 100644 --- a/charts/kates/templates/tests/test-kafka.yaml +++ b/charts/kates/templates/tests/test-kafka.yaml @@ -3,25 +3,25 @@ kind: Pod metadata: name: {{ include "kates.fullname" . }}-test-kafka labels: - {{- include "kates.labels" . | nindent 4 }} + {{- include "kates.podLabels" (dict "labels" (include "kates.labels" .) "ctx" .) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with include "kates.imagePullSecrets" . }}{{ . | nindent 2 }}{{ end }} + {{- with .Values.securityContext }} securityContext: - runAsNonRoot: true - runAsUser: 100 - fsGroup: 100 + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: kafka-check - image: curlimages/curl:8.7.1 + {{- $t := .Values.helmTests | default dict }} + image: {{ include "kates.imageRef" (dict "image" ($t.image | default "curlimages/curl:8.7.1") "ctx" .) | quote }} + {{- with .Values.containerSecurityContext }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: - - ALL + {{- toYaml . | nindent 6 }} + {{- end }} {{- if .Values.apiKey.enabled }} env: - name: KATES_API_KEY @@ -31,13 +31,10 @@ spec: key: {{ .Values.apiKey.secretKey | default "api-key" }} optional: true {{- end }} + {{- with $t.resources }} resources: - requests: - memory: "32Mi" - cpu: "25m" - limits: - memory: "64Mi" - cpu: "100m" + {{- toYaml . | nindent 6 }} + {{- end }} volumeMounts: - name: tmp mountPath: /tmp diff --git a/charts/kates/values-corporate.yaml b/charts/kates/values-corporate.yaml index b316d9c5..69280324 100644 --- a/charts/kates/values-corporate.yaml +++ b/charts/kates/values-corporate.yaml @@ -9,8 +9,11 @@ global: imagePullSecrets: [] # ── Security (OpenShift / PSA restricted) ───────────────────────────────────── -# Let the platform assign UIDs (required for OpenShift SCCs) +# On OpenShift, uncomment the two nulls so its SCCs assign the UID; left out, +# values.yaml's runAsUser and fsGroup (100) stay, as Helm merges this file. securityContext: + # runAsUser: null + # fsGroup: null runAsNonRoot: true seccompProfile: type: RuntimeDefault diff --git a/charts/kates/values.schema.json b/charts/kates/values.schema.json index 019fff1c..6efca52f 100644 --- a/charts/kates/values.schema.json +++ b/charts/kates/values.schema.json @@ -39,10 +39,15 @@ "imagePullSecrets": { "type": "array", "items": { - "type": "object", - "properties": { - "name": { "type": "string" } - } + "anyOf": [ + { "type": "string" }, + { + "type": "object", + "properties": { + "name": { "type": "string" } + } + } + ] } }, "nameOverride": { "type": "string" }, @@ -328,7 +333,9 @@ "pullPolicy": { "type": "string", "enum": [ "Always", "Never", "IfNotPresent" ] - } + }, + "securityContext": { "type": "object" }, + "resources": { "type": "object" } } } } @@ -448,7 +455,8 @@ "provider": { "type": "string", "enum": [ "", "litmus-crd", "kubernetes", "hybrid", "noop" ] - } + }, + "stressImage": { "type": "string" } } }, "faultTolerance": { @@ -748,7 +756,37 @@ "uniqueItems": true, "items": { "type": "string", "enum": [ "DONE", "FAILED" ] } }, - "image": { "type": "string", "minLength": 1 } + "image": { "type": "string", "minLength": 1 }, + "resources": { "type": "object" } + } + }, + "helmTests": { + "type": "object", + "properties": { + "image": { "type": "string", "minLength": 1 }, + "resources": { "type": "object" } + } + }, + "kyvernoPolicy": { + "type": "object", + "properties": { + "exceptions": { + "type": "array", + "description": "PolicyException entries for this release's pods: a Kyverno policy and rule names", + "items": { + "type": "object", + "properties": { + "policyName": { "type": "string", "minLength": 1 }, + "ruleNames": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 } + } + }, + "required": [ "policyName", "ruleNames" ], + "additionalProperties": false + } + } } } }, diff --git a/charts/kates/values.yaml b/charts/kates/values.yaml index 468ebf3c..83b35ba7 100644 --- a/charts/kates/values.yaml +++ b/charts/kates/values.yaml @@ -1,9 +1,11 @@ # -- Global configuration global: - # -- Override the image registry for all images in the chart - # Useful for air-gapped clusters that mirror images to an internal registry + # -- Prepended, with a slash, to every image the chart renders, the chaos + # stress image included. Useful for air-gapped clusters that mirror images + # to an internal registry imageRegistry: "" - # -- Override image pull secrets globally + # -- Pull secrets added to imagePullSecrets on every pod the chart renders, + # each a Secret name or {name: ...}: [regcred] or [{name: regcred}] imagePullSecrets: [] # -- Kubernetes cluster DNS domain clusterDomain: "cluster.local" @@ -31,7 +33,9 @@ image: # -- Image pull policy pullPolicy: IfNotPresent -# -- Secrets for pulling images from private registries +# -- Secrets for pulling images from private registries, on every pod the +# chart renders, each a Secret name or {name: ...}: [regcred] or +# [{name: regcred}] imagePullSecrets: [] # -- Override the chart name nameOverride: "" @@ -87,7 +91,8 @@ rbac: # -- Extra annotations to add to pods podAnnotations: {} -# -- Extra labels to add to pods +# -- Extra labels on every pod the chart renders: the API, PostgreSQL, the +# jobs and the helm test pods. The chart's own labels win a clash. podLabels: {} # -- Kubernetes Service configuration @@ -180,10 +185,11 @@ jvm: # -- JVM options passed via JAVA_TOOL_OPTIONS (ignored by the native image) options: "-Xms512m -Xmx2560m -XX:+UseZGC -XX:+ZGenerational" -# -- Pod-level security context (PSS restricted compliant) -# NOTE: runAsUser and fsGroup are intentionally unset so that platforms -# like OpenShift can assign random UIDs via their Security Context Constraints. -# Set them explicitly in your environment values file if needed. +# -- Pod-level security context (PSS restricted compliant), for the API pod, +# the backup, cleanup and migration jobs and the helm test pods. +# On OpenShift, set runAsUser and fsGroup to null so its Security Context +# Constraints assign the UID. Leaving them out of a values file, as +# values-corporate.yaml does, keeps 100: Helm merges a file over these. securityContext: runAsNonRoot: true runAsUser: 100 @@ -191,7 +197,8 @@ securityContext: seccompProfile: type: RuntimeDefault -# -- Container-level security context (PSS restricted compliant) +# -- Container-level security context (PSS restricted compliant), for the +# same pods as securityContext containerSecurityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true @@ -348,6 +355,25 @@ initContainers: waitForPostgres: image: busybox:1.36 pullPolicy: IfNotPresent + # -- Its container security context (null: containerSecurityContext; {} + # keeps these defaults, as Helm merges it over them). + # busybox runs as root unless told otherwise; 65534 is nobody. + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65534 + capabilities: + drop: + - ALL + # -- Its resources, in the API pod and the migration Job + resources: + requests: + memory: "16Mi" + cpu: "10m" + limits: + memory: "32Mi" + cpu: "50m" # -- Kafka connection configuration kafka: @@ -586,6 +612,8 @@ postgresql: runAsNonRoot: true runAsUser: 999 fsGroup: 999 + seccompProfile: + type: RuntimeDefault # -- PostgreSQL container security context containerSecurityContext: allowPrivilegeEscalation: false @@ -699,6 +727,10 @@ chaos: # `kates deploy --with-chaos=false` sets noop when no Litmus release is # installed. provider: "" + # -- The image CPU_STRESS and IO_STRESS run in the target pod + # (KATES_CHAOS_STRESS_IMAGE), with global.imageRegistry prepended. Empty + # keeps the backend's default, the same image. + stressImage: "docker.io/polinux/stress:1.0.4" # -- PostgreSQL backup CronJob backup: @@ -767,11 +799,48 @@ cleanup: - FAILED # -- Curl image for API calls image: curlimages/curl:8.7.1 + # -- Cleanup job resources + resources: + requests: + memory: "32Mi" + cpu: "25m" + limits: + memory: "64Mi" + cpu: "100m" + +# -- The helm test pods (`helm test`). They run with securityContext and +# containerSecurityContext, like the API pod. Not `tests`, which holds the +# benchmark defaults per test type. +helmTests: + # -- Curl image the tests call the API with + image: curlimages/curl:8.7.1 + # -- Resources of each test pod + resources: + requests: + memory: "32Mi" + cpu: "25m" + limits: + memory: "64Mi" + cpu: "100m" # -- Kyverno pod security policies (requires Kyverno controller in cluster) kyvernoPolicy: - # -- Enable Kyverno ClusterPolicy for the kates namespace + # -- Enable Kyverno ClusterPolicy for the kates namespace. ClusterPolicies + # are cluster-scoped: with rbac.scope=namespace they, the network policy + # generation, the cosign check and policyExceptions render nothing enabled: false + # -- Exceptions for this release's own pods, in its namespace, to rules of + # any Kyverno policy (the cluster's own included), whatever enabled says: + # a list of {policyName, ruleNames}. Rendered, where the kyverno.io/v2 API + # exists, as one PolicyException matching the API, PostgreSQL, the jobs and the test pods + # (by app.kubernetes.io/instance and app.kubernetes.io/name) as Pod, + # Deployment, ReplicaSet, StatefulSet, Job and CronJob, each rule with the + # autogen- and autogen-cronjob- Kyverno generates from it. A + # namespaced Policy is named /. Kyverno must accept + # exceptions from this namespace (its exceptionNamespace setting), e.g. + # - policyName: require-requests-limits + # ruleNames: [requests-and-memory-limit] + exceptions: [] # -- Policy action: Audit (observe only) or Enforce (block non-compliant) action: Audit # -- Enable mutation rules to auto-inject security contexts @@ -815,7 +884,9 @@ kyvernoPolicy: - cert-manager - litmus - "kates-detect-*" - # -- PolicyExceptions for dev/test namespaces + # -- PolicyExceptions to this chart's ClusterPolicy for whole dev/test + # namespaces (needs enabled; scope cluster only). For this release's own + # pods, in its namespace, use exceptions above policyExceptions: # -- Enable PolicyException resources enabled: false diff --git a/charts/legacy-kafka/Chart.yaml b/charts/legacy-kafka/Chart.yaml index 3593861d..1c925f4a 100644 --- a/charts/legacy-kafka/Chart.yaml +++ b/charts/legacy-kafka/Chart.yaml @@ -4,7 +4,7 @@ description: A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to mig type: application # Minor bump: mode and image derive from kafka.version when left empty, so any # version from 2.1 up deploys with no overlay. -version: 0.2.0 +version: 0.2.1 # Tracks the default kafka.version — the newest 3.x line, whose official image # needs no build. Any other version is `--set kafka.version=`; the mode and # the image follow (2.x is ZooKeeper on the built image, see values-kafka-2x.yaml). @@ -28,6 +28,12 @@ annotations: artifacthub.io/category: streaming-messaging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: fixed + description: "The Helm test pod has CPU and memory requests and a memory limit, testResources (100m/256Mi requests, 1000m/768Mi limits, what the topics Job gets: it runs two Kafka CLI JVMs), and uses imagePullSecrets like every other pod" + - kind: added + description: "global.imageRegistry (default empty) is prepended to the derived official image, apache/kafka:, for clusters that pull from a mirror; the built image keeps legacyImageRegistry, and an explicit kafka.image or testImage is used as written" + - kind: added + description: "podLabels (default empty) adds labels to the broker, ZooKeeper, topics Job and Helm test pods; the chart's own labels and extraLabels win on a clash, so selectors never change" - kind: changed description: mode defaults from kafka.version (zookeeper below 3.3.0, kraft from 3.3.0) and kafka.image from the version too (apache/kafka: from 3.7.0, /kates-legacy-kafka: below), so `--set kafka.version=3.5.2` needs no overlay; explicit values still win - kind: added diff --git a/charts/legacy-kafka/README.md b/charts/legacy-kafka/README.md index 0927fa70..aa9bdfb2 100644 --- a/charts/legacy-kafka/README.md +++ b/charts/legacy-kafka/README.md @@ -36,7 +36,11 @@ helm upgrade --install legacy charts/legacy-kafka -n kafka-legacy --create-names ``` Explicit `mode` and `kafka.image` values still win, which is what the two era -overlays below do. +overlays below do. To pull from a mirror, `global.imageRegistry` is prepended +to the derived official image (`/apache/kafka:`); the built +image takes `legacyImageRegistry`, and an explicit `kafka.image` is used as +written. `imagePullSecrets` reach every pod, the topics Job and the Helm test +included. ZooKeeper runs from the **same image** as the broker — the Kafka tarball ships ZooKeeper, so there is no second version pin that can drift away from the @@ -172,6 +176,12 @@ See [`values.yaml`](values.yaml) — every key is commented — and [`values.schema.json`](values.schema.json), which is strict (`additionalProperties: false`), so a typo fails the install instead of being silently ignored. +Every pod (brokers, ZooKeeper, the topics Job, the Helm test) passes Pod +Security Standards `restricted` with the defaults (`podSecurityContext`, +`containerSecurityContext`) and has CPU and memory requests and a memory limit; +the Helm test's are `testResources`. `podLabels` adds labels to every pod, and +the chart's own labels win on a clash. + | Overlay | For | |:---|:---| | `values-kafka-2x.yaml` | Kafka 2.8.2, ZooKeeper mode, the built image — a documented example of the 2.x shape (the same result as `--set kafka.version=2.8.2`, plus era config and topics) | diff --git a/charts/legacy-kafka/templates/_helpers.tpl b/charts/legacy-kafka/templates/_helpers.tpl index 01274f76..cbb859bb 100644 --- a/charts/legacy-kafka/templates/_helpers.tpl +++ b/charts/legacy-kafka/templates/_helpers.tpl @@ -78,17 +78,29 @@ the same image). An explicit `kafka.image` wins; otherwise: the official multi-arch apache/kafka: exists from 3.7.0 (KIP-975), and below that the image Dockerfile.legacy-kafka builds, /kates-legacy- kafka: — `scripts/build-legacy-kafka-image.sh --version --load`. +global.imageRegistry, when set, is prepended to the official image only: the +built one has legacyImageRegistry, and an explicit kafka.image is used as written. */}} {{- define "legacy-kafka.image" -}} {{- if .Values.kafka.image -}} {{- .Values.kafka.image -}} {{- else if semverCompare ">=3.7.0" (include "legacy-kafka.semver" .Values.kafka.version) -}} +{{- with .Values.global.imageRegistry }}{{ trimSuffix "/" . }}/{{ end -}} {{- printf "apache/kafka:%s" (toString .Values.kafka.version) -}} {{- else -}} {{- printf "%s/kates-legacy-kafka:%s" (.Values.legacyImageRegistry | default "ghcr.io/bmscomp" | trimSuffix "/") (toString .Values.kafka.version) -}} {{- end -}} {{- end }} +{{/* +Labels for a pod template: the chart's own labels (pass them as .labels) +plus podLabels. The chart's own win on a clash, so a selector never changes. +Usage: include "legacy-kafka.podLabels" (dict "ctx" . "labels" (include "legacy-kafka.labels" .)) +*/}} +{{- define "legacy-kafka.podLabels" -}} +{{- toYaml (merge (fromYaml .labels) .ctx.Values.podLabels) }} +{{- end }} + {{- define "legacy-kafka.brokerSelectorLabels" -}} {{ include "legacy-kafka.selectorLabels" . }} app.kubernetes.io/component: broker diff --git a/charts/legacy-kafka/templates/job-topics.yaml b/charts/legacy-kafka/templates/job-topics.yaml index 82b8e71f..5db14b20 100644 --- a/charts/legacy-kafka/templates/job-topics.yaml +++ b/charts/legacy-kafka/templates/job-topics.yaml @@ -21,8 +21,7 @@ spec: template: metadata: labels: - {{- include "legacy-kafka.labels" . | nindent 8 }} - app.kubernetes.io/component: topics + {{- include "legacy-kafka.podLabels" (dict "ctx" . "labels" (printf "%s\napp.kubernetes.io/component: topics" (include "legacy-kafka.labels" .))) | nindent 8 }} spec: restartPolicy: Never serviceAccountName: {{ include "legacy-kafka.serviceAccountName" . }} diff --git a/charts/legacy-kafka/templates/statefulset-kafka.yaml b/charts/legacy-kafka/templates/statefulset-kafka.yaml index a2d3c56b..de0a2bc7 100644 --- a/charts/legacy-kafka/templates/statefulset-kafka.yaml +++ b/charts/legacy-kafka/templates/statefulset-kafka.yaml @@ -20,8 +20,7 @@ spec: template: metadata: labels: - {{- include "legacy-kafka.labels" . | nindent 8 }} - app.kubernetes.io/component: broker + {{- include "legacy-kafka.podLabels" (dict "ctx" . "labels" (printf "%s\napp.kubernetes.io/component: broker" (include "legacy-kafka.labels" .))) | nindent 8 }} annotations: # Roll the brokers when the generated config changes — otherwise a # values edit produces a ConfigMap nobody re-reads. diff --git a/charts/legacy-kafka/templates/statefulset-zookeeper.yaml b/charts/legacy-kafka/templates/statefulset-zookeeper.yaml index 450ecc20..c11afc95 100644 --- a/charts/legacy-kafka/templates/statefulset-zookeeper.yaml +++ b/charts/legacy-kafka/templates/statefulset-zookeeper.yaml @@ -20,8 +20,7 @@ spec: template: metadata: labels: - {{- include "legacy-kafka.labels" . | nindent 8 }} - app.kubernetes.io/component: zookeeper + {{- include "legacy-kafka.podLabels" (dict "ctx" . "labels" (printf "%s\napp.kubernetes.io/component: zookeeper" (include "legacy-kafka.labels" .))) | nindent 8 }} annotations: checksum/scripts: {{ include (print $.Template.BasePath "/configmap-scripts.yaml") . | sha256sum }} spec: diff --git a/charts/legacy-kafka/templates/tests/test-broker.yaml b/charts/legacy-kafka/templates/tests/test-broker.yaml index 57758fa2..400d4341 100644 --- a/charts/legacy-kafka/templates/tests/test-broker.yaml +++ b/charts/legacy-kafka/templates/tests/test-broker.yaml @@ -6,13 +6,16 @@ metadata: name: {{ $fullname }}-test-broker namespace: {{ include "legacy-kafka.namespace" . }} labels: - {{- include "legacy-kafka.labels" . | nindent 4 }} - app.kubernetes.io/component: test + {{- include "legacy-kafka.podLabels" (dict "ctx" . "labels" (printf "%s\napp.kubernetes.io/component: test" (include "legacy-kafka.labels" .))) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 4 }} containers: @@ -23,6 +26,8 @@ spec: image: {{ .Values.testImage | default (include "legacy-kafka.image" .) | quote }} securityContext: {{- toYaml .Values.containerSecurityContext | nindent 8 }} + resources: + {{- toYaml .Values.testResources | nindent 8 }} command: ["/bin/sh", "-c"] env: - name: LOG_DIR diff --git a/charts/legacy-kafka/values.schema.json b/charts/legacy-kafka/values.schema.json index e066501b..3b56c406 100644 --- a/charts/legacy-kafka/values.schema.json +++ b/charts/legacy-kafka/values.schema.json @@ -9,6 +9,16 @@ "fullnameOverride": { "type": "string" }, "namespaceOverride": { "type": "string" }, "extraLabels": { "type": "object" }, + "podLabels": { "type": "object", "additionalProperties": { "type": "string" } }, + "global": { + "type": "object", + "properties": { + "imageRegistry": { + "type": "string", + "description": "Prepended to the derived official image (apache/kafka:, 3.7.0 and up) only" + } + } + }, "clusterDomain": { "type": "string" }, "mode": { "description": "zookeeper for Kafka 2.x; kraft for Kafka 3.3+; empty derives it from kafka.version (zookeeper below 3.3.0, kraft from 3.3.0)", @@ -136,6 +146,7 @@ "extraEgress": { "type": "array" } } }, - "testImage": { "type": "string" } + "testImage": { "type": "string" }, + "testResources": { "type": "object" } } } diff --git a/charts/legacy-kafka/values.yaml b/charts/legacy-kafka/values.yaml index 147568ff..bf6958c5 100644 --- a/charts/legacy-kafka/values.yaml +++ b/charts/legacy-kafka/values.yaml @@ -17,6 +17,17 @@ fullnameOverride: "" namespaceOverride: "" # -- Extra labels added to every resource extraLabels: {} +# -- Extra labels for every pod the chart renders (brokers, ZooKeeper, the +# topics Job and the Helm test). The chart's own labels win on a clash, so a +# selector never changes. +podLabels: {} + +global: + # -- Registry prepended to the derived official image, apache/kafka: + # (Kafka 3.7.0 and up), for clusters that pull from a mirror. The built image + # below 3.7.0 takes legacyImageRegistry instead, and an explicit kafka.image + # or testImage is used as written. + imageRegistry: "" # -- Kubernetes cluster domain (advertised-listener FQDNs) clusterDomain: cluster.local @@ -157,6 +168,7 @@ nodeSelector: {} tolerations: [] affinity: {} priorityClassName: "" +# -- Pull secrets for every pod, the topics Job and the Helm test included imagePullSecrets: [] serviceAccount: @@ -176,3 +188,9 @@ networkPolicy: # -- Image used by the Helm test pod. Defaults to the broker image, which is the # only client guaranteed to speak to a 2.x broker. testImage: "" +# -- Resources of the Helm test pod. It runs two Kafka CLI tools, each a JVM, +# so it gets what the topics Job gets. The pod runs with podSecurityContext +# and containerSecurityContext, like every other pod here. +testResources: + requests: { memory: 256Mi, cpu: 100m } + limits: { memory: 768Mi, cpu: 1000m } diff --git a/charts/mirror-maker2/Chart.yaml b/charts/mirror-maker2/Chart.yaml index 4d8579a9..f5096121 100644 --- a/charts/mirror-maker2/Chart.yaml +++ b/charts/mirror-maker2/Chart.yaml @@ -16,7 +16,10 @@ type: application # Patch: the pre-flight, data-test and secret-sync image moves to # kates-tester 1.28.0, the tag the v1.28.0 release publishes; every Strimzi # chart pins the same one. Templates and defaults are otherwise unchanged. -version: 0.11.7 +# Patch: test-mm2 and test-connectors get requests and limits, so every pod +# passes Pod Security "restricted" with default values; podLabels reach every +# pod and global.imageRegistry every image. +version: 0.11.8 # Tracks the Kafka line the MM2 Connect workers run (STRIMZI_KAFKA_VERSION). appVersion: "4.3.1" kubeVersion: ">=1.27.0-0" @@ -37,12 +40,20 @@ keywords: icon: https://raw.githubusercontent.com/bmscomp/kates/main/docs/icon.png dependencies: - name: kafka-common - version: 0.1.0 + version: 0.1.1 repository: file://../kafka-common annotations: artifacthub.io/category: streaming-messaging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: fixed + description: "test-mm2 and test-connectors had no requests or limits, so a namespace that enforces a requests-and-limits policy refused them. They get tests.resources (10m and 64Mi, a 128Mi limit). Every Helm test pod and the secret-sync Job also take imagePullSecrets now, which only the workers and the preflight Job had" + - kind: changed + description: "With rack.enabled the operator adds an init container to every worker, and an operator without Strimzi's restricted security provider leaves its security context empty, which Pod Security \"restricted\" refuses. The KafkaMirrorMaker2's spec.template.initContainer now gets no privilege escalation and no capabilities; templateExtra.initContainer still merges over it" + - kind: added + description: "podLabels: labels added to the workers' pod template, the Helm test pods and the secret-sync and preflight Jobs. The chart's own labels win on a clash, so no selector changes" + - kind: added + description: "global.imageRegistry: a mirror registry put in front of every image the chart names (image, rack.clientRackInitImage, testImages.kubectl, secretSync.image, the preflight and data-test images), as charts/kates does" - kind: changed description: "testImages.kubectl (and testImages.kafka where present) move to kates-tester 1.28.0, the tag the v1.28.0 release publishes; every Strimzi chart in the repository pins the same one (scripts/check-versions.sh holds them to the kates chart's appVersion). The image is unchanged since 1.27.0 — nothing under tester/ moved — so this is the pin, not the tooling." - kind: changed diff --git a/charts/mirror-maker2/README.md b/charts/mirror-maker2/README.md index 92a3ed3e..115d88d3 100644 --- a/charts/mirror-maker2/README.md +++ b/charts/mirror-maker2/README.md @@ -853,7 +853,8 @@ The blocks this chart adds on top of the `connect-cluster` worker surface: | Scheduling | Where the workers run | `nodeSelector` (rendered as required node affinity — Strimzi's pod template has no nodeSelector), `nodeAffinity`, `tolerations`, `podAntiAffinity`, `topologySpreadConstraints`, `priorityClassName` (empty by default; prod sets it), `terminationGracePeriodSeconds` | | `*.authentication` | Target and per-source credentials | `type: scram-sha-512\|scram-sha-256\|plain\|tls\|custom`. **`custom`** is the Strimzi v1 form of OAuth and other SASL mechanisms: `custom.sasl: true` plus `custom.config` (`sasl.mechanism`, `sasl.jaas.config`, … — or `ssl.keystore.*` for a client certificate). The preflight probe and the Helm tests report a custom-authenticated cluster as not verifiable | | `logging` | Worker log levels | `type: inline\|external`, `mirrorLevel`, `rootLevel`, `extraLoggers` | -| `tests` | The data-moving Helm tests | `replication.topic`, **`replication.topicByAlias`**, `replication.group`, `replication.sourceAlias`, `offsetTranslation.group`, **`failover.{group,messages,commitAfter}`** | +| `tests` | The data-moving Helm tests | `replication.topic`, **`replication.topicByAlias`**, `replication.group`, `replication.sourceAlias`, `offsetTranslation.group`, **`failover.{group,messages,commitAfter}`**; `resources` sizes test-mm2 and test-connectors (10m and 64Mi, a 128Mi limit) | +| `podLabels` / `global.imageRegistry` | Labels and a mirror registry | `podLabels` go on the workers' pod template, the test pods and the Jobs, under the chart's own labels; `global.imageRegistry` goes in front of every image the chart names (`image`, `rack.clientRackInitImage`, `testImages.kubectl`, `secretSync.image`, the preflight and data-test images) | **Bold** keys are new in 0.4.0; the scheduling row and `custom` authentication in 0.8.0. diff --git a/charts/mirror-maker2/templates/_helpers.tpl b/charts/mirror-maker2/templates/_helpers.tpl index a82ab4f0..4c77637c 100644 --- a/charts/mirror-maker2/templates/_helpers.tpl +++ b/charts/mirror-maker2/templates/_helpers.tpl @@ -549,3 +549,8 @@ workers would be worse than none. {{- define "mirror-maker2.imagePullPolicy" -}} {{- include "kafka-common.imagePullPolicy" .Values.imagePullPolicy }} {{- end }} + +{{/* An image this chart names, behind global.imageRegistry. Call with (list $ ). */}} +{{- define "mirror-maker2.image" -}} +{{- include "kafka-common.image" (dict "image" (index . 1) "registry" ((index . 0).Values.global).imageRegistry) }} +{{- end }} diff --git a/charts/mirror-maker2/templates/kafka-mirror-maker2.yaml b/charts/mirror-maker2/templates/kafka-mirror-maker2.yaml index 14e0995d..c363591b 100644 --- a/charts/mirror-maker2/templates/kafka-mirror-maker2.yaml +++ b/charts/mirror-maker2/templates/kafka-mirror-maker2.yaml @@ -152,6 +152,7 @@ built as one map and merged with templateExtra, never written beside it. "namespace" (include "mirror-maker2.namespace" .) "version" .Values.version "image" .Values.image + "imageRegistry" (.Values.global).imageRegistry "replicas" .Values.replicas "minReplicasDefault" 2 "autoscaling" .Values.autoscaling @@ -173,6 +174,7 @@ built as one map and merged with templateExtra, never written beside it. "podSelector" (include "mirror-maker2.podSelectorLabels" . | fromYaml) "template" (dict "pdb" .Values.podDisruptionBudget + "podMetadata" (dict "labels" .Values.podLabels) "podSecurityContext" .Values.podSecurityContext "imagePullSecrets" .Values.imagePullSecrets "priorityClassName" .Values.priorityClassName diff --git a/charts/mirror-maker2/templates/preflight-job.yaml b/charts/mirror-maker2/templates/preflight-job.yaml index daecd918..dffb973b 100644 --- a/charts/mirror-maker2/templates/preflight-job.yaml +++ b/charts/mirror-maker2/templates/preflight-job.yaml @@ -2,7 +2,7 @@ {{- $fullname := include "mirror-maker2.fullname" . }} {{- $ns := include "mirror-maker2.namespace" . }} {{- $pf := .Values.preflight }} -{{- $image := $pf.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version) }} +{{- $image := include "mirror-maker2.image" (list . ($pf.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version))) }} {{- /* Preflight: probe every source with the SAME client the workers will use. @@ -39,7 +39,7 @@ spec: template: metadata: labels: - {{- include "mirror-maker2.componentLabels" (dict "component" "preflight" "ctx" .) | nindent 8 }} + {{- include "kafka-common.podLabels" (dict "labels" (include "mirror-maker2.componentLabels" (dict "component" "preflight" "ctx" .)) "podLabels" .Values.podLabels) | nindent 8 }} spec: restartPolicy: Never {{- /* Deliberately NOT the chart's ServiceAccount. That SA is an ordinary diff --git a/charts/mirror-maker2/templates/secret-sync.yaml b/charts/mirror-maker2/templates/secret-sync.yaml index 28b67514..a7a9b314 100644 --- a/charts/mirror-maker2/templates/secret-sync.yaml +++ b/charts/mirror-maker2/templates/secret-sync.yaml @@ -32,8 +32,10 @@ CR simply references the Secrets and the operator retries until they appear. "namespace" (include "mirror-maker2.namespace" .) "labels" (include "mirror-maker2.labels" .) "secrets" .Values.secretSync.secrets - "image" (.Values.secretSync.image | default .Values.testImages.kubectl) + "image" (include "mirror-maker2.image" (list . (.Values.secretSync.image | default .Values.testImages.kubectl))) "pullPolicy" .Values.imagePullPolicy + "imagePullSecrets" .Values.imagePullSecrets + "podLabels" .Values.podLabels "timeoutSeconds" .Values.secretSync.timeoutSeconds "watch" .Values.secretSync.watch "schedule" .Values.secretSync.schedule) }} diff --git a/charts/mirror-maker2/templates/tests/test-connectors.yaml b/charts/mirror-maker2/templates/tests/test-connectors.yaml index d5f037ab..82cf2565 100644 --- a/charts/mirror-maker2/templates/tests/test-connectors.yaml +++ b/charts/mirror-maker2/templates/tests/test-connectors.yaml @@ -1,4 +1,4 @@ -{{- $kubectl := .Values.testImages.kubectl }} +{{- $kubectl := include "mirror-maker2.image" (list . .Values.testImages.kubectl) }} {{- $sa := printf "%s-test" (include "mirror-maker2.fullname" .) }} {{- $targetAlias := .Values.target.alias | default "target" }} {{- /* @@ -28,13 +28,17 @@ metadata: name: {{ include "mirror-maker2.fullname" . }}-test-connectors namespace: {{ include "mirror-maker2.namespace" . }} labels: - {{- include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .) | nindent 4 }} + {{- include "kafka-common.podLabels" (dict "labels" (include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .)) "podLabels" .Values.podLabels) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-weight": "10" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with $.Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} serviceAccountName: {{ $sa }} securityContext: runAsNonRoot: true @@ -50,6 +54,8 @@ spec: readOnlyRootFilesystem: true capabilities: drop: ["ALL"] + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} command: ['sh', '-c'] args: - | diff --git a/charts/mirror-maker2/templates/tests/test-failover.yaml b/charts/mirror-maker2/templates/tests/test-failover.yaml index 39ff4a23..f41de8d5 100644 --- a/charts/mirror-maker2/templates/tests/test-failover.yaml +++ b/charts/mirror-maker2/templates/tests/test-failover.yaml @@ -28,7 +28,7 @@ {{- if include "mirror-maker2.testAuthUnsupported" (dict "c" $target) }} {{- fail "tests.failover.enabled: the target uses TLS, mTLS or custom authentication, which this test cannot configure. Disable it or use scripts/test-mm2-migration.sh." }} {{- end }} -{{- $image := $t.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version) }} +{{- $image := include "mirror-maker2.image" (list . ($t.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version))) }} {{- /* THE ASSERTION test-offset-translation.yaml STOPS SHORT OF. @@ -85,16 +85,19 @@ metadata: name: {{ $fullname }}-test-failover namespace: {{ include "mirror-maker2.namespace" . }} labels: - {{- include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .) | nindent 4 }} - # The selector the kafka-cluster broker NetworkPolicy admits. Without it + # kates.io/test-pod is the selector the kafka-cluster broker NetworkPolicy admits. Without it # this pod is denied at the network layer and reports a replication failure. - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .)) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-weight": "40" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with $.Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} securityContext: runAsNonRoot: true seccompProfile: diff --git a/charts/mirror-maker2/templates/tests/test-mm2.yaml b/charts/mirror-maker2/templates/tests/test-mm2.yaml index ea26577c..db143d16 100644 --- a/charts/mirror-maker2/templates/tests/test-mm2.yaml +++ b/charts/mirror-maker2/templates/tests/test-mm2.yaml @@ -1,4 +1,4 @@ -{{- $kubectl := .Values.testImages.kubectl }} +{{- $kubectl := include "mirror-maker2.image" (list . .Values.testImages.kubectl) }} {{- $sa := printf "%s-test" (include "mirror-maker2.fullname" .) }} apiVersion: v1 kind: ServiceAccount @@ -54,12 +54,16 @@ metadata: name: {{ include "mirror-maker2.fullname" . }}-test-ready namespace: {{ include "mirror-maker2.namespace" . }} labels: - {{- include "mirror-maker2.labels" . | nindent 4 }} + {{- include "kafka-common.podLabels" (dict "labels" (include "mirror-maker2.labels" .) "podLabels" .Values.podLabels) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with $.Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} serviceAccountName: {{ $sa }} securityContext: runAsNonRoot: true @@ -75,6 +79,8 @@ spec: readOnlyRootFilesystem: true capabilities: drop: ["ALL"] + resources: + {{- toYaml $.Values.tests.resources | nindent 8 }} command: ['sh', '-c'] args: - | diff --git a/charts/mirror-maker2/templates/tests/test-offset-translation.yaml b/charts/mirror-maker2/templates/tests/test-offset-translation.yaml index 61a5b8e1..699ab10b 100644 --- a/charts/mirror-maker2/templates/tests/test-offset-translation.yaml +++ b/charts/mirror-maker2/templates/tests/test-offset-translation.yaml @@ -25,7 +25,7 @@ {{- if include "mirror-maker2.testAuthUnsupported" (dict "c" $target) }} {{- fail "tests.offsetTranslation.enabled: the target uses TLS, mTLS or custom authentication, which this test cannot configure. Disable it or use scripts/test-mm2-migration.sh." }} {{- end }} -{{- $image := $t.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version) }} +{{- $image := include "mirror-maker2.image" (list . ($t.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version))) }} {{- /* Offset translation is the half of a migration that people discover they skipped on the morning of the cutover: all the data is on the target, and no consumer @@ -54,16 +54,19 @@ metadata: name: {{ $fullname }}-test-offsets namespace: {{ include "mirror-maker2.namespace" . }} labels: - {{- include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .) | nindent 4 }} - # The selector the kafka-cluster broker NetworkPolicy admits. Without it + # kates.io/test-pod is the selector the kafka-cluster broker NetworkPolicy admits. Without it # this pod is denied at the network layer and reports a replication failure. - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .)) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-weight": "30" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with $.Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} securityContext: runAsNonRoot: true seccompProfile: diff --git a/charts/mirror-maker2/templates/tests/test-replication.yaml b/charts/mirror-maker2/templates/tests/test-replication.yaml index c1654d31..7c970a22 100644 --- a/charts/mirror-maker2/templates/tests/test-replication.yaml +++ b/charts/mirror-maker2/templates/tests/test-replication.yaml @@ -99,7 +99,7 @@ tells you nothing about replication. {{- if include "mirror-maker2.testAuthUnsupported" (dict "c" $target) }} {{- fail "tests.replication.enabled: the target uses TLS, mTLS or custom authentication, which this test cannot configure. Disable it or use scripts/test-mm2-migration.sh." }} {{- end }} -{{- $image := $t.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version) }} +{{- $image := include "mirror-maker2.image" (list . ($t.image | default (printf "quay.io/strimzi/kafka:%s-kafka-%s" (.Values.strimziVersion | default "1.1.0") .Values.version))) }} {{- /* THE test. Everything else in this chart's test suite asks Kubernetes whether things look right; this one asks Kafka whether the data arrived. @@ -130,16 +130,19 @@ metadata: name: {{ $fullname }}-test-replication namespace: {{ $ns }} labels: - {{- include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .) | nindent 4 }} - # The selector the kafka-cluster broker NetworkPolicy admits. Without it + # kates.io/test-pod is the selector the kafka-cluster broker NetworkPolicy admits. Without it # this pod is denied at the network layer and reports a replication failure. - kates.io/test-pod: "true" + {{- include "kafka-common.podLabels" (dict "labels" (include "mirror-maker2.componentLabels" (dict "component" "test" "ctx" .)) "podLabels" .Values.podLabels "testPod" true) | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-weight": "20" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with $.Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 4 }} + {{- end }} securityContext: runAsNonRoot: true seccompProfile: diff --git a/charts/mirror-maker2/values.schema.json b/charts/mirror-maker2/values.schema.json index d3cfa4c1..53d47f64 100644 --- a/charts/mirror-maker2/values.schema.json +++ b/charts/mirror-maker2/values.schema.json @@ -41,6 +41,22 @@ "extraLabels": { "type": "object" }, + "podLabels": { + "type": "object", + "description": "Labels added to every pod: the workers, the test pods and the Jobs; the chart labels win on a clash", + "additionalProperties": { + "type": "string" + } + }, + "global": { + "type": "object", + "properties": { + "imageRegistry": { + "type": "string", + "description": "Registry put in front of every image this chart names (empty: as written)" + } + } + }, "clusterDomain": { "type": "string" }, @@ -322,6 +338,10 @@ "minimum": 1 } } + }, + "resources": { + "type": "object", + "description": "Kubernetes resource requirements of test-mm2 and test-connectors" } } }, diff --git a/charts/mirror-maker2/values.yaml b/charts/mirror-maker2/values.yaml index 6462ea37..ed96144d 100644 --- a/charts/mirror-maker2/values.yaml +++ b/charts/mirror-maker2/values.yaml @@ -18,6 +18,20 @@ keepOnDelete: true # -- Extra labels added to every resource extraLabels: {} +# -- Labels added to every pod: the workers (through the KafkaMirrorMaker2's +# pod template), the Helm test pods and the secret-sync and preflight Jobs. +# The chart's own labels win on a clash, so no selector changes. +podLabels: {} + +# -- Settings shared with a parent chart +global: + # -- Registry put in front of every image this chart names, for a mirror: + # `image`, rack.clientRackInitImage, testImages.kubectl, secretSync.image, + # the preflight image and tests.replication.image (`mirror.example.com` + # gives mirror.example.com/quay.io/strimzi/kafka:…). Empty: as written. An + # empty `image` or clientRackInitImage is the operator's own, which the + # operator's registry setting redirects. + imageRegistry: "" # -- Images used by the Helm test pods testImages: @@ -638,6 +652,14 @@ alerts: # or target uses either REFUSES TO RENDER them rather than shipping a test that # fails on its own configuration. tests: + # -- Requests and limits of test-mm2 and test-connectors (kubectl and a + # shell). The data tests below size their own Kafka-client pods. + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi # -- End-to-end round trip, once per mirror: produce a corpus to that source, # consume it back from the target under the policy-correct replicated name, # and diff the corpus against what arrived (a count would let a duplicated diff --git a/charts/monitoring/Chart.yaml b/charts/monitoring/Chart.yaml index fb875380..b1c37ef4 100644 --- a/charts/monitoring/Chart.yaml +++ b/charts/monitoring/Chart.yaml @@ -1,10 +1,9 @@ apiVersion: v2 name: kates-monitoring -# Minor: kube-state-metrics now publishes the ChaosEngine's status as a -# state set, with the read access that takes — a series the chaos board's -# namespace picker and four of this chart's own alerts were built on and -# nothing produced. Defaults change (a CRD is watched); nothing else moves. -version: 1.6.0 +# Minor: every container of the stack gets requests and a memory limit, +# Prometheus included (2Gi), and a limit is a new way for a busy install to +# be OOM-killed; Grafana's root chown init container is off. +version: 1.7.0 appVersion: "82.4.3" description: Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration home: https://github.com/bmscomp/kates @@ -23,6 +22,16 @@ annotations: artifacthub.io/category: monitoring-logging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: changed + description: "Every container the stack renders or the Prometheus operator creates has CPU and memory requests and a memory limit, set through the subchart's values: Grafana (100m/128Mi, limit 512Mi) and its sidecars (10m/64Mi, 256Mi), the Prometheus operator (50m/128Mi, 512Mi), kube-state-metrics (50m/128Mi, 512Mi), the admission create and patch Jobs (10m/32Mi, 64Mi), node-exporter (10m/32Mi, 128Mi), Prometheus (200m/512Mi, 2Gi), Alertmanager (20m/64Mi, 256Mi) and the config reloader (10m/32Mi, 128Mi). A Prometheus that needs more than 2Gi is OOM-killed: raise kube-prometheus-stack.prometheus.prometheusSpec.resources.limits.memory (values-generic.yaml sets 4Gi)" + - kind: changed + description: "kube-prometheus-stack.grafana.initChownData.enabled is false. That init container runs as root with CAP_CHOWN, which the Restricted Pod Security Standard refuses, and it only runs with persistence on; Grafana's fsGroup (472) makes the volume writable on storage that honours fsGroup. Set it back to true for storage that does not" + - kind: changed + description: "The Helm test pod sets seccompProfile RuntimeDefault and runAsGroup 65534, and takes its security contexts and resources from test.podSecurityContext, test.securityContext and test.resources" + - kind: added + description: "global.imageRegistry and global.imagePullSecrets, passed down to kube-prometheus-stack and its subcharts and applied to the Helm test pod too; podLabels, added to the test pod's labels (the chart's own win on a clash)" + - kind: changed + description: "The node-exporter DaemonSet still runs by default and is the one workload that cannot pass the Restricted (or Baseline) standard: it reads the host. Everything else about it now follows Restricted: the RuntimeDefault seccomp profile, no privilege escalation and every capability dropped, merged into the subchart's non-root defaults. kube-prometheus-stack.nodeExporter.enabled=false switches it off, which kates deploy already does when the namespace enforces baseline or restricted; the values header and README say so" - kind: fixed description: "kube-state-metrics publishes kube_customresource_chaosengine_status_engine_status (status.engineStatus as a state set: one series per engine and state, initialized, completed, stopped) and may list and watch chaosengines.litmuschaos.io. The chaos board's namespace picker, its 'experiment in progress' tile and this chart's four chaos alerts all read that series, and nothing produced it: the picker resolved to nothing and KafkaChaosExperimentActive could not fire. Both work from the first ChaosEngine on. The three alerts guarded by 'and on() count(... == 1) == 0' still cannot fire, series or no series — count() over nothing is empty, not zero — and that guard is a separate fix. kube-state-metrics keeps running when the CRD is absent and discovers it when the chaos plane arrives, so installing this chart first is fine" - kind: fixed diff --git a/charts/monitoring/README.md b/charts/monitoring/README.md index 2257efb8..f2d274f2 100644 --- a/charts/monitoring/README.md +++ b/charts/monitoring/README.md @@ -35,9 +35,44 @@ Grafana is exposed as a NodePort on `30080` by default (dev convenience). | `kube-prometheus-stack.grafana.sidecar.dashboards.searchNamespace` | `ALL` | Which namespaces the sidecar watches for boards | | `kube-prometheus-stack.kube-state-metrics.customResourceState` | ChaosEngine `status.engineStatus` as a state set | What makes `kube_customresource_chaosengine_status_engine_status` exist — see [below](#kube-state-metrics-and-the-chaosengine). `enabled: false` switches it off | | `kube-prometheus-stack.kube-state-metrics.rbac.extraRules` | `list`/`watch` on `chaosengines.litmuschaos.io` | The read access that state set needs | +| `global.imageRegistry` | `""` | Registry prefixed to every image, the wrapped stack's and the test pod's | +| `global.imagePullSecrets` | `[]` | Pull secrets for every pod (`- name: s` or `- s`) | +| `podLabels` | `{}` | Extra labels on the Helm test pod; the stack's pods take theirs from each component's `podLabels` (`podMetadata.labels` for Prometheus and Alertmanager), and the admission-create and admission-patch Jobs only from `kube-prometheus-stack.commonLabels`, which labels every resource kube-prometheus-stack's own templates render | +| `test.image` | `curlimages/curl:8.7.1` | The Helm test image | +| `test.podSecurityContext` / `test.securityContext` | non-root 65534, `RuntimeDefault` seccomp / no privilege escalation, all capabilities dropped | The test pod's security contexts | +| `test.resources` | requests `25m`/`32Mi`, limits `100m`/`64Mi` | The test pod's resources | +| `kube-prometheus-stack.*.resources` | requests and a memory limit on every container | See [Pod security](#pod-security-and-resources) | +| `kube-prometheus-stack.nodeExporter.enabled` | `true` | node-exporter; host-level, see [Pod security](#pod-security-and-resources) | Own keys are validated by `values.schema.json`; subchart keys are validated by kube-prometheus-stack itself. +## Pod security and resources + +Every pod this chart and the wrapped stack render passes the Restricted Pod +Security Standard, and every container has CPU and memory requests and a +memory limit, the Prometheus and Alertmanager pods the operator creates +included. The defaults are in `values.yaml`; Prometheus is limited to 2Gi, +which a busy cluster outgrows (`values-generic.yaml` sets 4Gi). + +One workload cannot pass: the **node-exporter** DaemonSet reads the host (host +network, PID namespace and root filesystem), so no namespace that enforces +`baseline` or `restricted` admits it. Every other Restricted setting is on +(non-root, `RuntimeDefault` seccomp, no privilege escalation, all capabilities +dropped), so only those host controls need a namespace that allows them. It +stays on by default. Switch it off with + +```bash +--set kube-prometheus-stack.nodeExporter.enabled=false +``` + +`kates deploy` does this itself when the monitoring namespace enforces +`baseline` or `restricted`. The node panels and node alerts go with it. + +Grafana's `initChownData` init container (root, `CAP_CHOWN`) is off: with +persistence on, Grafana's fsGroup makes the volume writable on storage that +honours fsGroup. Turn it back on only for storage that does not, in a +namespace that allows it. + ## kube-state-metrics and the ChaosEngine Since 1.6.0 the wrapped kube-state-metrics publishes diff --git a/charts/monitoring/templates/tests/test-connection.yaml b/charts/monitoring/templates/tests/test-connection.yaml index 42e6d71a..9f2fcd64 100644 --- a/charts/monitoring/templates/tests/test-connection.yaml +++ b/charts/monitoring/templates/tests/test-connection.yaml @@ -5,31 +5,27 @@ metadata: name: {{ .Release.Name }}-monitoring-test namespace: {{ .Release.Namespace }} labels: - {{- include "kates-monitoring.labels" . | nindent 4 }} + {{- toYaml (merge (include "kates-monitoring.labels" . | fromYaml) (.Values.podLabels | default dict)) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with (.Values.global).imagePullSecrets }} + imagePullSecrets: + {{- range . }} + - name: {{ if kindIs "string" . }}{{ . }}{{ else }}{{ .name }}{{ end }} + {{- end }} + {{- end }} securityContext: - runAsNonRoot: true - runAsUser: 65534 + {{- toYaml .Values.test.podSecurityContext | nindent 4 }} containers: - name: connection-test - image: {{ .Values.test.image }} + image: {{ with (.Values.global).imageRegistry }}{{ . }}/{{ end }}{{ .Values.test.image }} securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: - - ALL + {{- toYaml .Values.test.securityContext | nindent 8 }} resources: - requests: - memory: 32Mi - cpu: 25m - limits: - memory: 64Mi - cpu: 100m + {{- toYaml .Values.test.resources | nindent 8 }} command: - sh - -c diff --git a/charts/monitoring/values.schema.json b/charts/monitoring/values.schema.json index 9e663ff7..f3135cf7 100644 --- a/charts/monitoring/values.schema.json +++ b/charts/monitoring/values.schema.json @@ -48,8 +48,27 @@ "test": { "type": "object", "properties": { - "image": { "type": "string" } + "image": { "type": "string" }, + "podSecurityContext": { "type": "object" }, + "securityContext": { "type": "object" }, + "resources": { "type": "object" } } + }, + "global": { + "type": "object", + "description": "Passed down to kube-prometheus-stack and its subcharts; also used by this chart's test pod", + "properties": { + "imageRegistry": { "type": "string" }, + "imagePullSecrets": { + "type": "array", + "items": { "type": ["string", "object"] } + } + } + }, + "podLabels": { + "type": "object", + "description": "Extra labels on this chart's own pod (the Helm test); the chart's labels win on a clash", + "additionalProperties": { "type": "string" } } } } diff --git a/charts/monitoring/values.yaml b/charts/monitoring/values.yaml index 02bbd925..c81aa10d 100644 --- a/charts/monitoring/values.yaml +++ b/charts/monitoring/values.yaml @@ -1,5 +1,35 @@ # Kates Monitoring — Default Values # Wraps kube-prometheus-stack with Kates-specific dashboards and config +# +# Every pod this chart and the wrapped stack render passes the Restricted Pod +# Security Standard and carries CPU and memory requests and a memory limit, +# except one: the node-exporter DaemonSet reads the host (host network, PID +# namespace and root filesystem) and cannot run where a namespace enforces +# baseline or restricted. It stays on by default; switch it off with +# `kube-prometheus-stack.nodeExporter.enabled=false`, which `kates deploy` +# already does when the monitoring namespace enforces baseline or restricted. +# The node-level panels and alerts go with it. + +# -- Global settings, passed down to kube-prometheus-stack and its subcharts +# as well as used by this chart's own test pod +global: + # -- Registry prefixed to every image (empty: each image's own registry) + imageRegistry: "" + # -- Pull secrets for every pod; `- name: secret` or `- secret` + imagePullSecrets: [] + +# -- Extra labels on this chart's own pod (the Helm test). The chart's labels +# win on a clash. The wrapped stack's pods take theirs from each component's +# key: kube-prometheus-stack.grafana.podLabels, +# kube-prometheus-stack.prometheusOperator.podLabels, +# kube-prometheus-stack.kube-state-metrics.podLabels, +# kube-prometheus-stack.prometheus-node-exporter.podLabels, and +# podMetadata.labels under prometheus.prometheusSpec and +# alertmanager.alertmanagerSpec. The admission-create and admission-patch +# Jobs take no pod label value: kube-prometheus-stack.commonLabels labels +# them, with every other resource kube-prometheus-stack's own templates +# render (not the grafana, kube-state-metrics or node-exporter subcharts). +podLabels: {} kube-prometheus-stack: enabled: true @@ -12,11 +42,32 @@ kube-prometheus-stack: service: type: NodePort nodePort: 30080 + # -- Grafana's own container + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + memory: 512Mi + # -- The init container that chowns the data volume runs as root with + # CAP_CHOWN, which the Restricted standard refuses. Off: Grafana's pod + # fsGroup (472) already makes a persistent volume writable on storage that + # honours fsGroup. Turn it back on only for storage that does not, in a + # namespace that allows it. + initChownData: + enabled: false sidecar: image: tag: "2.2.3" pullPolicy: IfNotPresent imagePullPolicy: IfNotPresent + # -- Each sidecar (dashboards, datasources) + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 256Mi dashboards: enabled: true # -- The annotation the sidecar reads to file a board into a Grafana @@ -93,6 +144,13 @@ kube-prometheus-stack: image: tag: v3.9.1 pullPolicy: IfNotPresent + # -- Prometheus's container in the StatefulSet the operator creates + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + memory: 2Gi storageSpec: volumeClaimTemplate: spec: @@ -108,10 +166,23 @@ kube-prometheus-stack: image: tag: v0.79.2 pullPolicy: IfNotPresent + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + memory: 512Mi prometheusConfigReloader: image: tag: v0.79.2 pullPolicy: IfNotPresent + # -- The config-reloader the operator adds to Prometheus and Alertmanager + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi # kube-prometheus-stack reads this under prometheusOperator, not at the top # level of the subchart — a second copy used to sit beside kube-state-metrics # where the chart never looked at it, so its enabled and tag settings did @@ -127,17 +198,37 @@ kube-prometheus-stack: image: tag: 1.7.8 pullPolicy: IfNotPresent + # -- The admission-create and admission-patch hook Jobs + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 64Mi alertmanager: alertmanagerSpec: image: tag: v0.28.1 pullPolicy: IfNotPresent + # -- Alertmanager's container in the StatefulSet the operator creates + resources: + requests: + cpu: 20m + memory: 64Mi + limits: + memory: 256Mi kube-state-metrics: image: tag: v2.14.0 pullPolicy: IfNotPresent + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + memory: 512Mi # The chaos board and the chaos rules below read # kube_customresource_chaosengine_status_engine_status — is an experiment # in progress? kube-state-metrics publishes nothing about a custom @@ -181,10 +272,28 @@ kube-prometheus-stack: path: [status, engineStatus] list: [initialized, completed, stopped] + # Host-level: cannot pass the Restricted (or Baseline) standard. See the + # header for the switch, kube-prometheus-stack.nodeExporter.enabled=false. prometheus-node-exporter: image: tag: v1.8.2 pullPolicy: IfNotPresent + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi + # -- Merged into the subchart's own (non-root 65534, read-only root + # filesystem), so only the host controls stay outside Restricted. + securityContext: + seccompProfile: + type: RuntimeDefault + containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL defaultRules: create: true @@ -253,3 +362,24 @@ dashboards: # Image used by the helm test connectivity pod test: image: curlimages/curl:8.7.1 + # -- The test pod's security context (Restricted Pod Security Standard) + podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + # -- The test container's security context + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + memory: 32Mi + cpu: 25m + limits: + memory: 64Mi + cpu: 100m diff --git a/charts/strimzi-operator/Chart.yaml b/charts/strimzi-operator/Chart.yaml index 1eecc109..edb6ac87 100644 --- a/charts/strimzi-operator/Chart.yaml +++ b/charts/strimzi-operator/Chart.yaml @@ -1,9 +1,9 @@ apiVersion: v2 name: strimzi-operator -# Patch: the CRD-hook and Helm-test image moves to kates-tester 1.28.0, the -# tag the v1.28.0 release publishes; every Strimzi chart pins the same one. -# Templates and defaults are otherwise unchanged. -version: 0.3.7 +# Minor: the operator runs Strimzi's restricted pod security provider by +# default, which rolls every Kafka pod it manages once and refuses a Kaniko +# Connect build; the Helm tests get resources, and podLabels is new. +version: 0.4.0 # Tracks the Strimzi operator version this chart deploys. The CLI rewrites this # (and the dependency version) in a generated copy when another version is # selected; the repository copy is the pin. @@ -30,6 +30,12 @@ annotations: artifacthub.io/category: streaming-messaging artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: changed + description: "strimzi-kafka-operator.extraEnvs sets STRIMZI_POD_SECURITY_PROVIDER_CLASS=restricted, so every pod the operator creates (brokers, controllers, Connect, MirrorMaker 2, the Entity Operator, Cruise Control, the exporter) carries the Restricted Pod Security Standard's settings and starts in a namespace that enforces restricted. The operator rolls the pods it manages once after the upgrade. A KafkaConnect build with Kaniko needs root and is refused under this provider: set strimzi-kafka-operator.extraEnvs to a list without the entry to go back to upstream's baseline. Helm replaces the list whole, so keep the entry when adding env of your own" + - kind: added + description: "tests.resources: each Helm test pod requests 10m CPU and 64Mi and is limited to 128Mi, so the tests run where a policy requires requests and a memory limit" + - kind: added + description: "podLabels, added to the labels of the pods this chart renders itself (the CRD-upgrade Job, the drain cleaner, the Helm tests); the chart's own labels win on a clash. strimzi-kafka-operator.image.imagePullSecrets now reaches those pods too" - kind: changed description: "testImages.kubectl (and testImages.kafka where present) move to kates-tester 1.28.0, the tag the v1.28.0 release publishes; every Strimzi chart in the repository pins the same one (scripts/check-versions.sh holds them to the kates chart's appVersion). The image is unchanged since 1.27.0 — nothing under tester/ moved — so this is the pin, not the tooling." - kind: changed diff --git a/charts/strimzi-operator/README.md b/charts/strimzi-operator/README.md index 55ff121f..805c77a0 100644 --- a/charts/strimzi-operator/README.md +++ b/charts/strimzi-operator/README.md @@ -93,6 +93,8 @@ kubectl get deploy strimzi-cluster-operator -n strimzi-operator \ | `alerts.enabled` | `true` | `StrimziOperatorDown`, `StrimziReconciliationsFailing`, `KafkaCertificateExpiringSoon` and `KafkaCertificateExpiryCritical` (on `strimzi_certificate_expiration_timestamp_ms`, for every cluster the operator manages). Thresholds: `alerts.thresholds.certificate{Warning,Critical}Days` (30, 7). Runbook: `docs/kafka-cluster-runbook.md`. | | `nameOverride` / `fullnameOverride` | `""` | Standard name overrides | | `commonLabels` / `commonAnnotations` | `{}` | Applied to chart-owned resources | +| `podLabels` | `{}` | Extra labels on the pods this chart renders itself (CRD-upgrade Job, drain cleaner, Helm tests); the chart's own labels win on a clash. The operator's pod takes upstream's `strimzi-kafka-operator.labels`. | +| `tests.resources` | requests `10m`/`64Mi`, memory limit `128Mi` | Resources of each Helm test pod | ### Operator keys (`strimzi-kafka-operator.*`) @@ -110,8 +112,9 @@ Only these deviate from stock upstream. Everything else in the [upstream values] | `strimzi-kafka-operator.createGlobalResources` | `true` | `false` is the additional-operator shape: the fixed-name ClusterRoles and bindings come from the primary's release. On its own it leaves the ServiceAccount unbound from three of them — pair it with `globalBindings.enabled: true`. | | `strimzi-kafka-operator.watchNamespaces` | `[]` | With `watchAnyNamespace: false`, the namespaces to watch. The release namespace is always included, so `[]` means "only my own namespace". | | `strimzi-kafka-operator.dashboards.enabled` | `true` | The operator's nine Grafana dashboards (Kafka, KRaft, Cruise Control, Kafka Exporter, Connect, MirrorMaker 2, Kafka Bridge, Kafka OAuth, operators). They read exactly the series kafka-cluster's vendored exporter rules produce; kafka-cluster 1.0 has no dashboards of its own. Names are fixed: enable on one operator release. | +| `strimzi-kafka-operator.extraEnvs` | `STRIMZI_POD_SECURITY_PROVIDER_CLASS=restricted` | Every pod the operator creates gets the Restricted Pod Security Standard's settings, so Kafka starts in a namespace that enforces `restricted`. Helm replaces the list whole: keep this entry when adding your own. Leave it out for upstream's baseline provider, which a KafkaConnect `build` with Kaniko (root) needs. | -> **There is no `global` block.** Upstream contains zero `.Values.global` references and ignores `global.imageRegistry` entirely. Declaring it to satisfy the schema would convert a loud failure into a silent one, so `--set global.*` is rejected. For registry redirection use `strimzi-kafka-operator.defaultImageRegistry`. +> **There is no `global` block.** Upstream contains zero `.Values.global` references and ignores `global.imageRegistry` entirely. Declaring it to satisfy the schema would convert a loud failure into a silent one, so `--set global.*` is rejected. For registry redirection use `strimzi-kafka-operator.defaultImageRegistry`. This chart's own images are `testImages.kubectl` and `drainCleaner.image`; set them to your mirror too. `strimzi-kafka-operator.image.imagePullSecrets` (a name or a list, as upstream reads it) also reaches this chart's own pods. ## Examples diff --git a/charts/strimzi-operator/templates/_helpers.tpl b/charts/strimzi-operator/templates/_helpers.tpl index febacf02..14833a5b 100644 --- a/charts/strimzi-operator/templates/_helpers.tpl +++ b/charts/strimzi-operator/templates/_helpers.tpl @@ -158,6 +158,30 @@ watchNamespaces plus the release namespace. Returns "*" or a comma list. {{- end -}} {{- end }} +{{/* +Labels for a pod this chart renders itself: podLabels under the given labels, +which win on a clash so no selector moves. Call with (dict "ctx" $ "labels" ). +*/}} +{{- define "strimzi-operator.podLabels" -}} +{{- toYaml (merge (.labels | fromYaml) (.ctx.Values.podLabels | default dict)) -}} +{{- end }} + +{{/* +imagePullSecrets for this chart's own pods, from the operator's +strimzi-kafka-operator.image.imagePullSecrets read as upstream reads it: a +string names one Secret, a list is used as given. +*/}} +{{- define "strimzi-operator.imagePullSecrets" -}} +{{- $s := ((index .Values "strimzi-kafka-operator").image | default dict).imagePullSecrets -}} +{{- if and $s (kindIs "string" $s) }} +imagePullSecrets: + - name: {{ $s }} +{{- else if and $s (kindIs "slice" $s) }} +imagePullSecrets: + {{- toYaml $s | nindent 2 }} +{{- end }} +{{- end }} + {{/* Labels for a component other than the operator itself. */}} {{- define "strimzi-operator.componentLabels" -}} {{- $l := include "strimzi-operator.labels" .ctx | fromYaml -}} diff --git a/charts/strimzi-operator/templates/crd-upgrade.yaml b/charts/strimzi-operator/templates/crd-upgrade.yaml index 78e94646..a9c16e33 100644 --- a/charts/strimzi-operator/templates/crd-upgrade.yaml +++ b/charts/strimzi-operator/templates/crd-upgrade.yaml @@ -127,10 +127,10 @@ spec: template: metadata: labels: - {{- include "strimzi-operator.selectorLabels" . | nindent 8 }} - app: crd-upgrade + {{- include "strimzi-operator.podLabels" (dict "ctx" . "labels" (printf "%s\napp: crd-upgrade" (include "strimzi-operator.selectorLabels" .))) | nindent 8 }} spec: serviceAccountName: {{ include "strimzi-operator.fullname" . }}-crd-upgrade + {{- include "strimzi-operator.imagePullSecrets" . | nindent 6 }} restartPolicy: OnFailure securityContext: {{- include "strimzi-operator.podSecurityContext" . | nindent 8 }} diff --git a/charts/strimzi-operator/templates/drain-cleaner.yaml b/charts/strimzi-operator/templates/drain-cleaner.yaml index 7f7f2bb8..6756ec08 100644 --- a/charts/strimzi-operator/templates/drain-cleaner.yaml +++ b/charts/strimzi-operator/templates/drain-cleaner.yaml @@ -129,9 +129,10 @@ spec: template: metadata: labels: - {{- toYaml $labels | nindent 8 }} + {{- include "strimzi-operator.podLabels" (dict "ctx" . "labels" (toYaml $labels)) | nindent 8 }} spec: serviceAccountName: {{ $name }} + {{- include "strimzi-operator.imagePullSecrets" . | nindent 6 }} {{- with $dc.priorityClassName }} priorityClassName: {{ . }} {{- end }} diff --git a/charts/strimzi-operator/templates/tests/test-operator.yaml b/charts/strimzi-operator/templates/tests/test-operator.yaml index c76b8faf..267861e8 100644 --- a/charts/strimzi-operator/templates/tests/test-operator.yaml +++ b/charts/strimzi-operator/templates/tests/test-operator.yaml @@ -8,6 +8,7 @@ releases (the primary plus an additional operator) are both named be deleted from under each other's `helm test`. */}} {{- $testRole := printf "%s-%s" $testSA .Release.Namespace }} +{{- $podLabels := include "strimzi-operator.podLabels" (dict "ctx" . "labels" (include "strimzi-operator.labels" .)) }} apiVersion: v1 kind: ServiceAccount metadata: @@ -68,13 +69,14 @@ metadata: name: {{ include "strimzi-operator.fullname" . }}-test-operator namespace: {{ .Release.Namespace }} labels: - {{- include "strimzi-operator.labels" . | nindent 4 }} + {{- $podLabels | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never serviceAccountName: {{ $testSA }} + {{- include "strimzi-operator.imagePullSecrets" . | nindent 2 }} securityContext: {{- include "strimzi-operator.podSecurityContext" . | nindent 4 }} containers: @@ -82,6 +84,8 @@ spec: image: {{ $kubectl }} securityContext: {{- include "strimzi-operator.containerSecurityContext" . | nindent 8 }} + resources: + {{- toYaml .Values.tests.resources | nindent 8 }} command: ['sh', '-c'] args: - | @@ -104,13 +108,14 @@ metadata: name: {{ include "strimzi-operator.fullname" . }}-test-crds namespace: {{ .Release.Namespace }} labels: - {{- include "strimzi-operator.labels" . | nindent 4 }} + {{- $podLabels | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never serviceAccountName: {{ $testSA }} + {{- include "strimzi-operator.imagePullSecrets" . | nindent 2 }} securityContext: {{- include "strimzi-operator.podSecurityContext" . | nindent 4 }} containers: @@ -118,6 +123,8 @@ spec: image: {{ $kubectl }} securityContext: {{- include "strimzi-operator.containerSecurityContext" . | nindent 8 }} + resources: + {{- toYaml .Values.tests.resources | nindent 8 }} command: ['sh', '-c'] args: - | @@ -154,13 +161,14 @@ metadata: name: {{ include "strimzi-operator.fullname" . }}-test-watch-scope namespace: {{ .Release.Namespace }} labels: - {{- include "strimzi-operator.labels" . | nindent 4 }} + {{- $podLabels | nindent 4 }} annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: restartPolicy: Never serviceAccountName: {{ $testSA }} + {{- include "strimzi-operator.imagePullSecrets" . | nindent 2 }} securityContext: {{- include "strimzi-operator.podSecurityContext" . | nindent 4 }} containers: @@ -168,6 +176,8 @@ spec: image: {{ $kubectl }} securityContext: {{- include "strimzi-operator.containerSecurityContext" . | nindent 8 }} + resources: + {{- toYaml .Values.tests.resources | nindent 8 }} command: ['sh', '-c'] args: - | diff --git a/charts/strimzi-operator/values.schema.json b/charts/strimzi-operator/values.schema.json index f2ee2d62..08951a58 100644 --- a/charts/strimzi-operator/values.schema.json +++ b/charts/strimzi-operator/values.schema.json @@ -9,6 +9,18 @@ "fullnameOverride": { "type": "string" }, "commonLabels": { "type": "object" }, "commonAnnotations": { "type": "object" }, + "podLabels": { + "type": "object", + "additionalProperties": { "type": "string" }, + "description": "Extra labels on the pods this chart renders itself (CRD-upgrade Job, drain cleaner, Helm tests). The chart's own labels win on a clash." + }, + "tests": { + "type": "object", + "additionalProperties": false, + "properties": { + "resources": { "type": "object", "description": "Resources of each Helm test pod." } + } + }, "strimziVersion": { "type": "string", "minLength": 1, @@ -330,7 +342,12 @@ } }, "podSecurityContext": { "type": "object" }, - "securityContext": { "type": "object" } + "securityContext": { "type": "object" }, + "extraEnvs": { + "type": "array", + "description": "Extra operator env. The default sets STRIMZI_POD_SECURITY_PROVIDER_CLASS=restricted; Helm replaces the list whole.", + "items": { "type": "object", "required": ["name"], "properties": { "name": { "type": "string", "minLength": 1 } } } + } } } } diff --git a/charts/strimzi-operator/values.yaml b/charts/strimzi-operator/values.yaml index b31503ba..cbf7db79 100644 --- a/charts/strimzi-operator/values.yaml +++ b/charts/strimzi-operator/values.yaml @@ -31,6 +31,11 @@ fullnameOverride: "" commonLabels: {} # -- Extra annotations added to every resource this chart renders commonAnnotations: {} +# -- Extra labels on every pod this chart renders itself: the CRD-upgrade Job, +# the drain cleaner and the Helm tests. The chart's own labels win on a clash, +# so no selector moves. The operator's pod is upstream's: use +# `strimzi-kafka-operator.labels` for it. +podLabels: {} # ── Strimzi version ────────────────────────────────────────────────────────── # -- Strimzi version used to build the CRD bundle URL for the crdUpgrade hook. @@ -46,6 +51,16 @@ testImages: # -- kubectl image used by the CRD-upgrade hook Job and the Helm tests kubectl: "ghcr.io/bmscomp/kates-tester:1.28.0" +# ── Helm tests ─────────────────────────────────────────────────────────────── +tests: + # -- Resources for each Helm test pod (a kubectl client) + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi + # ── CRD lifecycle ──────────────────────────────────────────────────────────── # Closes the gap Helm leaves: crds/ is applied on install and never on upgrade. # This hook runs pre-install AND pre-upgrade, so CRDs are current before the @@ -200,7 +215,7 @@ alerts: # The ONLY place operator settings take effect. Full key reference: # https://github.com/strimzi/strimzi-kafka-operator/blob/main/helm-charts/helm3/strimzi-kafka-operator/values.yaml # -# Only FOUR keys below deviate from stock upstream; everything else is pinned +# Only FIVE keys below deviate from stock upstream; everything else is pinned # for safety or is an explicit restatement of an upstream default. strimzi-kafka-operator: # -- Operator replicas. All three previous call sites set 1. @@ -224,6 +239,17 @@ strimzi-kafka-operator: # 60000 in config/kafka/strimzi-values.yaml, which was never applied. fullReconciliationIntervalMs: 120000 + # -- Extra operator env. STRIMZI_POD_SECURITY_PROVIDER_CLASS=restricted + # gives every pod the operator creates (brokers, controllers, Connect, + # MirrorMaker 2, the Entity Operator, Cruise Control, the exporter) the + # Restricted Pod Security Standard's settings, so they start in a namespace + # that enforces it. Helm replaces a list whole: keep this entry when you add + # your own, or leave it out for upstream's baseline provider (which a + # KafkaConnect `build` with Kaniko needs: Kaniko runs as root). + extraEnvs: + - name: STRIMZI_POD_SECURITY_PROVIDER_CLASS + value: restricted + # -- Operator resources. MEMORY ONLY: 768Mi is what runs live (upstream # default is 384Mi). cpu is deliberately NOT set — upstream already defaults # it to limits 1000m / requests 200m, and duplicating that here would create diff --git a/charts/velero/Chart.yaml b/charts/velero/Chart.yaml index 8b5f607a..45efcd44 100644 --- a/charts/velero/Chart.yaml +++ b/charts/velero/Chart.yaml @@ -14,4 +14,14 @@ maintainers: name: velero sources: - https://github.com/vmware-tanzu/velero -version: 11.3.3 +version: 11.3.4 +annotations: + artifacthub.io/changes: | + - kind: changed + description: "Kates defaults (upstream ships these empty): podSecurityContext runs as 65534:65534, non-root, with the RuntimeDefault seccomp profile; containerSecurityContext and kubectl.containerSecurityContext forbid privilege escalation and drop every capability; resources (100m/128Mi, limit 512Mi), upgradeJobResources (50m/128Mi, limit 256Mi) and kubectl.resources (10m/32Mi, limit 128Mi) are set. The Deployment and the upgrade-crds and cleanup-crds Jobs pass the Restricted Pod Security Standard. Plugin initContainers are written by the installer and need the same securityContext and resources" + - kind: changed + description: "The Helm test pod (a Kates addition) sets the RuntimeDefault seccomp profile, takes its image, security contexts and resources from tests.*, and carries image.imagePullSecrets and podLabels (the chart's own labels win on a clash)" + - kind: changed + description: "deployNodeAgent stays false and is documented as host-level: the node-agent mounts the kubelet's pod and plugin directories from the host and runs as root with the default capabilities, which the Baseline and Restricted standards refuse, so it runs only in a namespace that enforces privileged. nodeAgent.podSecurityContext adds the RuntimeDefault seccomp profile (privileged containers run unconfined regardless) and nodeAgent.resources requests 100m/512Mi with a 1024Mi memory limit. nodeAgent.containerSecurityContext stays empty, allowPrivilegeEscalation unset: a default of false would be merged into a privileged true setting, which the API server refuses" + - kind: changed + description: "namespace.labels (empty by default) is documented as the one switch whose Job cannot pass the Restricted standard: upstream's label-namespace Job takes no security context or resources from any value. Label the namespace yourself instead" diff --git a/charts/velero/README.md b/charts/velero/README.md index 088b7f7c..6cdec74f 100644 --- a/charts/velero/README.md +++ b/charts/velero/README.md @@ -4,6 +4,45 @@ Velero is an open source tool to safely backup and restore, perform disaster rec Velero has two main components: a CLI, and a server-side Kubernetes deployment. +## Kates defaults + +This copy of the upstream chart changes defaults in `values.yaml` only, plus +the Helm test pod (`templates/tests/test-connection.yaml`, configured by +`tests.*`). With them, the Velero Deployment, the upgrade-crds and +cleanup-crds Jobs and the test pod pass the Restricted Pod Security Standard +and carry CPU and memory requests and a memory limit: + +| Key | Kates default | +|-----|---------------| +| `podSecurityContext` | `runAsNonRoot`, `runAsUser`/`runAsGroup` 65534, seccomp `RuntimeDefault` | +| `containerSecurityContext`, `kubectl.containerSecurityContext` | `allowPrivilegeEscalation: false`, capabilities drop `ALL` | +| `resources` | requests `100m`/`128Mi`, memory limit `512Mi` | +| `upgradeJobResources` | requests `50m`/`128Mi`, memory limit `256Mi` | +| `kubectl.resources` | requests `10m`/`32Mi`, memory limit `128Mi` | +| `tests.image` | `curlimages/curl:8.7.1` | +| `tests.podSecurityContext` / `tests.containerSecurityContext` / `tests.resources` | the test pod's, Restricted | + +Plugin `initContainers` are yours to write: give each +`allowPrivilegeEscalation: false`, capabilities drop `ALL` and resources, or +the pod is refused where `restricted` is enforced. + +The **node-agent** (`deployNodeAgent`, off) cannot pass: it mounts the +kubelet's pod and plugin directories from the host and runs as root with the +runtime's default capabilities, which the Baseline and Restricted standards +refuse. Turn it on only in a namespace labelled +`pod-security.kubernetes.io/enforce: privileged`. `nodeAgent.podSecurityContext` +adds the `RuntimeDefault` seccomp profile (a privileged container runs +unconfined regardless), and `nodeAgent.resources` requests `100m`/`512Mi` with +a `1024Mi` memory limit. `nodeAgent.containerSecurityContext` stays empty, as +upstream ships it: `allowPrivilegeEscalation` is not set, because a default of +`false` would be merged into a `privileged: true` setting, a pair the API +server refuses. + +The **label-namespace** Job (rendered when `namespace.labels` is not empty; +`{}` by default) cannot pass either: upstream's template gives it no security +context and no resources, and no value reaches them. Leave `namespace.labels` +empty and label the namespace yourself (`kubectl label namespace ...`). + ## Installing the Velero CLI See the different options for installing the [Velero CLI](https://velero.io/docs/v1.13/basic-install/#install-the-cli). diff --git a/charts/velero/templates/tests/test-connection.yaml b/charts/velero/templates/tests/test-connection.yaml index b20125de..ddf59951 100644 --- a/charts/velero/templates/tests/test-connection.yaml +++ b/charts/velero/templates/tests/test-connection.yaml @@ -7,34 +7,28 @@ metadata: name: {{ include "velero.fullname" . }}-test namespace: {{ .Release.Namespace }} labels: - app.kubernetes.io/name: {{ include "velero.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} - helm.sh/chart: {{ include "velero.chart" . }} + {{- $labels := dict "app.kubernetes.io/name" (include "velero.name" .) "app.kubernetes.io/instance" .Release.Name "app.kubernetes.io/managed-by" .Release.Service "helm.sh/chart" (include "velero.chart" .) }} + {{- toYaml (merge $labels (.Values.podLabels | default dict)) | nindent 4 }} annotations: helm.sh/hook: test helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: restartPolicy: Never + {{- with .Values.image.imagePullSecrets }} + imagePullSecrets: + {{- range . }} + - name: {{ . }} + {{- end }} + {{- end }} securityContext: - runAsNonRoot: true - runAsUser: 65534 + {{- toYaml .Values.tests.podSecurityContext | nindent 4 }} containers: - name: velero-metrics-test - image: curlimages/curl:8.7.1 + image: "{{ .Values.tests.image.repository }}:{{ .Values.tests.image.tag }}" securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: - - ALL + {{- toYaml .Values.tests.containerSecurityContext | nindent 8 }} resources: - requests: - memory: 32Mi - cpu: 25m - limits: - memory: 64Mi - cpu: 100m + {{- toYaml .Values.tests.resources | nindent 8 }} command: - sh - -c diff --git a/charts/velero/values.schema.json b/charts/velero/values.schema.json index 455844ff..8527f705 100644 --- a/charts/velero/values.schema.json +++ b/charts/velero/values.schema.json @@ -173,6 +173,22 @@ "type": "array", "items": {} }, + "tests": { + "type": "object", + "description": "Kates addition: the Helm test pod", + "properties": { + "image": { + "type": "object", + "properties": { + "repository": { "type": "string" }, + "tag": { "type": "string" } + } + }, + "podSecurityContext": { "type": "object" }, + "containerSecurityContext": { "type": "object" }, + "resources": { "type": "object" } + } + }, "metrics": { "type": "object", "properties": { diff --git a/charts/velero/values.yaml b/charts/velero/values.yaml index b3859e9b..49cbf263 100644 --- a/charts/velero/values.yaml +++ b/charts/velero/values.yaml @@ -4,6 +4,10 @@ # Labels settings in namespace namespace: + # Kates: a non-empty map renders the label-namespace Job, which takes no + # securityContext or resources from any value and so cannot pass the + # Restricted Pod Security Standard. Keep it empty and label the namespace + # yourself where restricted is enforced. labels: {} # Enforce Pod Security Standards with Namespace Labels # https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/ @@ -67,13 +71,14 @@ podLabels: {} # Resource requests/limits to specify for the Velero deployment. # https://velero.io/docs/v1.6/customize-installation/#customize-resource-requests-and-limits -resources: {} - # requests: - # cpu: 500m - # memory: 128Mi - # limits: - # cpu: 1000m - # memory: 512Mi +# Kates: set by default (upstream: {}), so the pod passes a policy that +# requires requests and a memory limit. +resources: + requests: + cpu: 100m + memory: 128Mi + limits: + memory: 512Mi # Container resize policy for the Velero deployment. # See: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/ @@ -92,13 +97,13 @@ hostAliases: [] # - "bar.local" # Resource requests/limits to specify for the upgradeCRDs job pod. Need to be adjusted by user accordingly. -upgradeJobResources: {} -# requests: -# cpu: 50m -# memory: 128Mi -# limits: -# cpu: 100m -# memory: 256Mi +# Kates: set by default (upstream: {}). +upgradeJobResources: + requests: + cpu: 50m + memory: 128Mi + limits: + memory: 256Mi upgradeCRDsJob: # Extra volumes for the Upgrade CRDs Job. Optional. extraVolumes: [] @@ -139,16 +144,27 @@ initContainers: # SecurityContext to use for the Velero deployment. Optional. # Set fsGroup for `AWS IAM Roles for Service Accounts` # see more informations at: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html -podSecurityContext: {} +# Kates: the Restricted Pod Security Standard by default (upstream: {}), for +# the Deployment and the upgrade-crds/cleanup-crds Jobs. The image's user is +# named, not numbered, so runAsNonRoot needs a numeric runAsUser; Velero +# writes only to its emptyDirs (/plugins, /scratch). +podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault # fsGroup: 1337 # Container Level Security Context for the 'velero' container of the Velero deployment. Optional. # See: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container -containerSecurityContext: {} - # allowPrivilegeEscalation: false - # capabilities: - # drop: ["ALL"] - # add: [] +# Kates: set by default (upstream: {}). Plugin initContainers are yours to +# write: give each the same securityContext and resources, or the pod is +# refused where restricted is enforced. +containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] # readOnlyRootFilesystem: true # Container Lifecycle Hooks to use for the Velero deployment. Optional. @@ -231,6 +247,32 @@ extraObjects: [] # secretName: velero-secrets-store # type: Opaque +# Kates addition (not upstream): the Helm test pod +# (templates/tests/test-connection.yaml), rendered with metrics.enabled. It +# takes image.imagePullSecrets and podLabels like the Velero pod. +tests: + image: + repository: curlimages/curl + tag: 8.7.1 + podSecurityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + containerSecurityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + memory: 32Mi + cpu: 25m + limits: + memory: 64Mi + cpu: 100m + # Settings for Velero's prometheus metrics. Enabled by default. metrics: enabled: true @@ -351,9 +393,19 @@ kubectl: # tag: 1.16.15 # Container Level Security Context for the 'kubectl' container of the crd jobs. Optional. # See: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container - containerSecurityContext: {} + # Kates: set by default (upstream: {}). + containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] # Resource requests/limits to specify for the upgrade/cleanup job. Optional - resources: {} + # Kates: set by default (upstream: {}). + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi # Annotations to set for the upgrade/cleanup job. Optional. annotations: {} # Labels to set for the upgrade/cleanup job. Optional. @@ -651,6 +703,10 @@ backupsEnabled: true snapshotsEnabled: true # Whether to deploy the node-agent daemonset. +# Kates: the node-agent is host-level (hostPath mounts of the kubelet's pod and +# plugin directories) and cannot pass the Baseline or Restricted Pod Security +# Standard. Leave it off where the namespace enforces either; turn it on only +# in a namespace labelled pod-security.kubernetes.io/enforce: privileged. deployNodeAgent: false nodeAgent: @@ -664,13 +720,14 @@ nodeAgent: runtimeClassName: "" # Resource requests/limits to specify for the node-agent daemonset deployment. Optional. # https://velero.io/docs/v1.6/customize-installation/#customize-resource-requests-and-limits - resources: {} - # requests: - # cpu: 500m - # memory: 512Mi - # limits: - # cpu: 1000m - # memory: 1024Mi + # Kates: set by default (upstream: {}), upstream's suggested memory with a + # smaller CPU request. + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + memory: 1024Mi # Container resize policy for the node-agent daemonset. # See: https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/ resizePolicy: [] @@ -736,10 +793,17 @@ nodeAgent: # see more informations at: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html podSecurityContext: runAsUser: 0 + # Kates: added (upstream: unset). + seccompProfile: + type: RuntimeDefault # fsGroup: 1337 # Container Level Security Context for the 'node-agent' container of the node-agent daemonset. Optional. # See: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container + # Kates: left empty like upstream, allowPrivilegeEscalation included. The + # node-agent cannot pass Restricted anyway, and a default of false would be + # merged into a privileged: true set here or in nodeAgent.securityContext, + # a pair the API server refuses. containerSecurityContext: {} # Container Lifecycle Hooks to use for the node-agent daemonset. Optional. diff --git a/cli/cmd/mcp_caveats.go b/cli/cmd/mcp_caveats.go index ba64175f..04eeaed7 100644 --- a/cli/cmd/mcp_caveats.go +++ b/cli/cmd/mcp_caveats.go @@ -201,7 +201,7 @@ var mcpCaveatsCore = []mcpCaveat{ mcpAnchoredRef("kates/src/main/resources/application.properties:329-334", "# Prometheus server (for disruption metrics capture)", "kates.prometheus.url=http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090"), - mcpAnchoredRef("charts/kates/values.yaml:411-421", + mcpAnchoredRef("charts/kates/values.yaml:437-447", "# -- Prometheus the backend queries for a disruption's Kafka metrics", `url: "http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090"`), mcpAnchoredRef("charts/kates/templates/configmap.yaml:43-45", @@ -211,7 +211,7 @@ var mcpCaveatsCore = []mcpCaveat{ `KATES_PROMETHEUS_URL: "http://monitoring-kube-prometheus-prometheus.kafka.svc:9090"`), mcpJava + "disruption/PrometheusMetricsCapture.java:31-34", mcpJava + "disruption/SlaGrader.java:26,140,153,170-178,183,191-192", - "charts/monitoring/Chart.yaml:18", + "charts/monitoring/Chart.yaml:17", }, }, { diff --git a/cli/cmd/mcp_tools_scenario.go b/cli/cmd/mcp_tools_scenario.go index d7a51715..f1a2f59a 100644 --- a/cli/cmd/mcp_tools_scenario.go +++ b/cli/cmd/mcp_tools_scenario.go @@ -1258,7 +1258,7 @@ var mcpCaveatsScenario = []mcpCaveat{ mcpAnchoredRef(mcpJava+"engine/TestOrchestrator.java:1045-1091", "TestSpec applyTypeDefaults(", "return merged;"), mcpAnchoredRef("kates/src/main/resources/application.properties:73-114", "# Per-type overrides: STRESS", "kates.tests.roundtrip.throughput=10000"), - mcpAnchoredRef("charts/kates/values.yaml:624-692", "defaults:", "tests:", `throughput: "10000"`), + mcpAnchoredRef("charts/kates/values.yaml:652-720", "defaults:", "tests:", `throughput: "10000"`), mcpAnchoredRef("charts/kates/templates/configmap.yaml:88-184", "KATES_TESTS_LOAD_REPLICATION_FACTOR:", "KATES_TESTS_ROUNDTRIP_NUM_CONSUMERS:"), }, diff --git a/config/velero/velero-values-offline.yaml b/config/velero/velero-values-offline.yaml index 731457a3..4234a938 100644 --- a/config/velero/velero-values-offline.yaml +++ b/config/velero/velero-values-offline.yaml @@ -11,6 +11,18 @@ initContainers: volumeMounts: - mountPath: /target name: plugins + # The chart's defaults cover its own containers only: without these the + # Restricted Pod Security Standard refuses the Velero pod. + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 64Mi configuration: backupStorageLocation: diff --git a/config/velero/velero-values.yaml b/config/velero/velero-values.yaml index 3996b004..88eb6867 100644 --- a/config/velero/velero-values.yaml +++ b/config/velero/velero-values.yaml @@ -11,6 +11,18 @@ initContainers: volumeMounts: - mountPath: /target name: plugins + # The chart's defaults cover its own containers only: without these the + # Restricted Pod Security Standard refuses the Velero pod. + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 64Mi configuration: backupStorageLocation: diff --git a/docs/book/07-chaos-practice.md b/docs/book/07-chaos-practice.md index 724a3135..c23066bf 100644 --- a/docs/book/07-chaos-practice.md +++ b/docs/book/07-chaos-practice.md @@ -142,8 +142,8 @@ The `kubernetes` provider implements these disruptions against the Kubernetes AP | `LEADER_ELECTION` | Force-delete the target pod; a step that names `targetTopic` targets that partition's leader | Forces [leader election](appendix-a-glossary.md#gl-leader-election) for targeted partition | | `SCALE_DOWN` | Lower `spec.replicas` by one on the [KafkaNodePool](appendix-a-glossary.md#gl-kafkanodepool) of each matching broker, then wait for the Strimzi Cluster Operator to remove a broker; on a Kafka that Strimzi doesn't run, scale its StatefulSet down by one | One broker fewer per node pool, until rollback puts it back | | `NETWORK_PARTITION` | Create a NetworkPolicy with no allow rules that selects the target pod | Adds a NetworkPolicy with no allow rules to the pod, removed after `chaosDurationSec` | -| `CPU_STRESS` | Stress ephemeral container injected into the pod | Saturates CPU on the broker pod | -| `IO_STRESS` | Stress ephemeral container injected into the pod | Injects disk I/O pressure on broker storage | +| `CPU_STRESS` | A non-root stress ephemeral container, `chaos.stressImage`, injected into the pod | Saturates CPU on the broker pod | +| `IO_STRESS` | The same container, with I/O workers | Injects disk I/O pressure on broker storage | The provider injects each fault once, and doesn't try it again when a call fails. The API server may have carried out the call and lost only its answer. A second try would then act on what the first changed: it would kill another pod picked at random, or scale a StatefulSet down from the count the first try left. The step fails instead. A partition that fails or is interrupted removes its NetworkPolicies all the same, as at the end of its duration, and a failed node pool scale-down gives the pools their replicas back. Removing a partition is the one call the provider retries, up to three more times, because doing it again removes nothing more. diff --git a/docs/book/17-security.md b/docs/book/17-security.md index ce3d3c3a..f03f24a6 100644 --- a/docs/book/17-security.md +++ b/docs/book/17-security.md @@ -640,34 +640,149 @@ kubectl run np-probe -n default --rm -i --restart=Never --image="$IMAGE" -- bash ## Container Security -The `kafka-cluster` chart hardens the broker and controller pods by default: every `KafkaNodePool` it renders carries the security contexts below in its pod template, and a pool's `podSecurityContext` and `containerSecurityContext` values merge over them. +The Kates charts hold their pods to the Kubernetes Pod Security Standards (PSS) at the **restricted** level. With default values, every pod they render passes it, Helm test pods and jobs included, and every container carries CPU and memory requests and a memory limit. The pods the Strimzi operator creates from the charts' custom resources get the restricted settings too, from the operator and from the resources' templates. The exceptions are a handful of host-level agents and upstream subcharts that take no security context from values, which [Workloads That Cannot Pass](#workloads-that-cannot-pass) names with their switches. ### Security Contexts -Every container in the Kates platform runs with a hardened security context. These settings follow the Kubernetes Pod Security Standards (PSS) at the **restricted** level: +Beyond the baseline level's bans on host namespaces, privileged containers and `hostPath` volumes, the restricted level asks four settings of every pod and container. The charts set those four, and some also make the root filesystem read-only, which the standard doesn't ask for. Each setting takes away one thing an attacker who gets into the container could use: + +| Setting | Value | What it takes away | +|---------|-------|--------------------| +| `runAsNonRoot` | `true` | UID 0, even when the image's `USER` is root | +| `seccompProfile.type` | `RuntimeDefault` | System calls outside the container runtime's default allow list | +| `allowPrivilegeEscalation` | `false` | `setuid` and `setgid` binaries that raise privileges inside the container | +| `capabilities.drop` | `["ALL"]` | Every Linux capability: no raw sockets, no network admin, no mounts | +| `readOnlyRootFilesystem` | `true`, where a chart sets it | Writes outside mounted volumes, such as a dropped binary | + +The `kafka-cluster` chart writes these into the pod template of every [`KafkaNodePool`](appendix-a-glossary.md#gl-kafkanodepool) it renders, with a fixed user ID and `fsGroup` besides. A pool's `podSecurityContext` and `containerSecurityContext` values merge over them: ```yaml template: + pod: + securityContext: + runAsNonRoot: true + runAsUser: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault kafkaContainer: securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] - pod: - securityContext: - runAsNonRoot: true - fsGroup: 1001 ``` -Each setting takes away one thing an attacker who gets into the container could use: +### What Each Chart Sets + +Each chart puts the restricted settings, with requests and a memory limit, on the pods this table names. The last column gives the values that hold their settings, and each chart's README lists the defaults: + +| Chart | Pods it renders | Values that hold the settings | +|-------|-----------------|-------------------------------| +| `kates` | The Kates API, PostgreSQL, the backup, cleanup and migration jobs, the Helm test pods | `securityContext` and `containerSecurityContext`; PostgreSQL's under `postgresql`, the init container's under `initContainers.waitForPostgres` | +| `kafka-cluster` | The Helm test pods; through Strimzi, every Kafka component | Each component's `template` and each pool's contexts; `tests.resources`, `tests.jvmResources` | +| `connect-cluster`, `mirror-maker2` | The test pods and the secret-sync and pre-flight jobs; through Strimzi, the workers | `podSecurityContext`; `connectContainer` for the worker container; `tests.resources` | +| `strimzi-operator` | The operator, the CRD-upgrade job, the Drain Cleaner, the Helm test pods | `strimzi-kafka-operator.extraEnvs` for the pods the operator creates; `tests.resources` | +| `kates-chaos` | The experiment installer job, the Game Day job, the Helm test pods | `podSecurityContext`, `securityContext` and `resources` under `experiments.installer`, `gameday` and `tests` | +| `monitoring` | The kube-prometheus-stack pods but node-exporter, the Prometheus and Alertmanager pods its operator creates, the Helm test pod | Each component's keys under `kube-prometheus-stack`; `test.*` | +| `velero` | The Velero server, the upgrade-crds and cleanup-crds jobs, the Helm test pod | `podSecurityContext`, `containerSecurityContext`, `kubectl.containerSecurityContext`, `tests.*` | +| `apicurio-registry`, `headlamp`, `kafka-ui`, `legacy-kafka` | The application, its UI, brokers or jobs, and the Helm test pod | Each chart's pod and container contexts; `tests.*` (`test.*` in `headlamp`, `testResources` in `legacy-kafka`) | +| `kates-platform` | Its Helm test pod; each subchart's pods as that chart's row says | `test.resources`, beside fixed security contexts; each subchart's keys under its name, such as `kates.securityContext` | + +A key the table names merges over the chart's defaults, so setting one field of a security context keeps the others. + +### The Strimzi Restricted Provider + +Strimzi, not Helm, creates the pods of a Kafka cluster, a Connect cluster and a MirrorMaker 2, so their security contexts come from the operator's pod security provider. The `strimzi-operator` chart turns on Strimzi's restricted provider with one environment variable: + +```yaml +strimzi-kafka-operator: + extraEnvs: + - name: STRIMZI_POD_SECURITY_PROVIDER_CLASS + value: restricted +``` + +The operator then gives every pod it creates the restricted settings. Helm replaces a list whole, so keep this entry when you add environment variables of your own. + +An operator you share with other teams may run Strimzi's default baseline provider instead. So the `kafka-cluster`, `connect-cluster` and `mirror-maker2` charts write the restricted settings into their custom resources' templates as well, and their pods pass on either provider. The templates cover every component's pod and containers, and the init container that rack awareness adds. In `kafka-cluster`, `kafka.template`, `entityOperator.template`, `cruiseControl.template`, `kafkaExporter.template` and each pool's `template` merge over them. + +### Chaos Stress Containers + +A `CPU_STRESS` or `IO_STRESS` fault on the `kubernetes` chaos provider adds an ephemeral container to the target pod (see [Direct Kubernetes API](07-chaos-practice.md#direct-kubernetes-api)). The container joins the target, usually a broker, so the target namespace's admission rules apply to it, not the Kates API's. It runs as the user and group the target pod sets, at pod level or else on one of its containers, as OpenShift does, which that namespace already admits. Where neither sets a user other than 0, the container runs as user 65534 (nobody), and where neither sets a group, as group 65534. It has no privilege escalation, every capability dropped and the `RuntimeDefault` seccomp profile, so it passes restricted. + +Its image is pinned, too. The `kates` chart's `chaos.stressImage`, `docker.io/polinux/stress:1.0.4` by default, reaches the Kates API as `KATES_CHAOS_STRESS_IMAGE`, with `global.imageRegistry` put in front of it. The setting and that security context need a Kates API image newer than 1.28.0. Version 1.28.0 and earlier pull an untagged `polinux/stress` from Docker Hub and add it with no security context of its own. A namespace that enforces restricted refuses it, and it runs as root wherever the target pod sets no user. + +### Workloads That Cannot Pass + +A few workloads can't pass restricted, or carry no requests and limits, with the values the charts ship. They read the host, or they come from an upstream template that takes no security context or resources from values. Each is off by default or has a switch: + +| Chart | Workload | Why it fails | Switch | +|-------|----------|--------------|--------| +| `kates-chaos` | Deployment `litmus`, the [LitmusChaos](appendix-a-glossary.md#gl-litmuschaos) operator, and `litmus-monitor`, its exporter, with `litmus-core.exporter.enabled` | The litmus-core subchart sets no security context and reads none from values | `kyvernoPolicy.enabled` with `mutate`, as `values-generic.yaml` sets them; `kyvernoPolicy.exceptions`; or `litmus-core.enabled=false` to bring your own Litmus | +| `monitoring` | DaemonSet `monitoring-prometheus-node-exporter`, on by default | Host network and PID namespace, `hostPath` mounts of `/proc`, `/sys` and `/`, a host port | `kube-prometheus-stack.nodeExporter.enabled=false`; the node panels and alerts go with it | +| `velero` | DaemonSet `node-agent`, with `deployNodeAgent=true`; off by default | `hostPath` mounts of the kubelet's pod and plugin directories; root with the runtime's default capabilities; no `allowPrivilegeEscalation: false` | Keep `deployNodeAgent=false`, or run it in a namespace that enforces `privileged` | +| `velero` | Job `velero-label-namespace`, when `namespace.labels` isn't empty; `{}` by default | Upstream's template sets no security context and no resources, and no value reaches them | Keep `namespace.labels` empty and label the namespace yourself | +| `connect-cluster` | The operator's build pod, with `build` mode | Strimzi's Kaniko build runs as root | `image`, the default, or `plugins` | +| `kafka-cluster` | SeaweedFS's master, volume and filer StatefulSets, and with `createBuckets` Job `-bucket-hook`; `seaweedfs.enabled` is off by default | With the values this chart ships: `hostPath` data and log volumes, root, no requests or limits. The bucket hook takes no resources, labels or pull secrets from any value | Keep `seaweedfs.enabled: false` and use an object store outside the namespace, or install SeaweedFS from the start with the subchart's own security contexts, resources and claim-backed storage ([SeaweedFS](20-installation-guide.md#seaweedfs)) | + +The two Litmus pods pass the baseline level. Kyverno can exempt them through a [`PolicyException`](appendix-a-glossary.md#gl-policyexception), which `kyvernoPolicy.exceptions` renders (see [Policy Exceptions](#policy-exceptions)). It can also patch them to restricted at admission: the chart's own `kyvernoPolicy`, enabled with `mutate: true` as `values-generic.yaml` sets it, gives them a non-root user, the `RuntimeDefault` seccomp profile, no privilege escalation and no capabilities. Pod Security Admission, the controller built into Kubernetes that reads a namespace's `pod-security.kubernetes.io/enforce` label, has no exception for a single workload, so a namespace labelled `restricted` refuses them unless those mutate rules patched them first. + +`kates deploy` sets `kube-prometheus-stack.nodeExporter.enabled=false` itself when the monitoring namespace enforces baseline or restricted. The [Velero](appendix-a-glossary.md#gl-velero) node agent is what file-system backups and the data mover need, so the `kafka-cluster` chart's `backup.volumes: fs-backup` needs a `velero` namespace that admits it. Velero's plugins are `initContainers` you write in its values, and restricted refuses the Velero pod unless each has `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, and CPU and memory requests with a memory limit. `config/velero/velero-values.yaml`, the file `make velero` installs with, and its offline twin `velero-values-offline.yaml` give the AWS plugin all three; give a plugin you add the same. + +Two more settings trade restricted away. A Connect `build` needs an operator without the restricted provider's `extraEnvs` entry, besides a namespace that admits a root pod. SeaweedFS, which the `kafka-cluster` chart's `values-prod.yaml` turns on, keeps its data in `hostPath` directories, as [SeaweedFS](20-installation-guide.md#seaweedfs) describes. In a namespace that enforces restricted, point tiered storage and Velero at an object store outside it instead. + +### Pod Labels + +A cluster that requires a label on every pod, such as a team or a cost center, sets it through each chart's `podLabels` value. The value reaches every pod the chart renders, Helm test pods and jobs included, and the pod templates of the Strimzi resources. The chart's own labels win a clash, so no selector changes, except in `velero`'s server Deployment, where the vendored chart puts `podLabels` after its own labels, so a key that clashes with one of them breaks the Deployment. The pods in this table take their labels from a key of their own instead: + +| Chart | Pods | Their key | +|-------|------|-----------| +| `kates-chaos` | The chaos operator and exporter | `litmus-core.customLabels`, which labels every other resource the subchart renders too | +| `kafka-cluster` | SeaweedFS's master, volume and filer; its bucket hook takes no labels from any value | `seaweedfs.podLabels` | +| `monitoring` | Every pod but the Helm test | Each component's `podLabels` under `kube-prometheus-stack`; `podMetadata.labels` for Prometheus and Alertmanager; `kube-prometheus-stack.commonLabels` for the admission-create and admission-patch jobs | +| `strimzi-operator` | The operator | `strimzi-kafka-operator.labels` | +| `velero` | The server and Helm test; the CRD jobs; the node agent | `podLabels`; `kubectl.labels`; `nodeAgent.podLabels` | +| `kates-platform` | Every subchart's pods; the umbrella's `podLabels` reaches only its Helm test pod | Each subchart's own keys, such as `kates.podLabels` and `kafka-cluster.podLabels` | + +`kube-prometheus-stack.commonLabels` labels every resource that kube-prometheus-stack's own templates render, not only the two jobs. + +### Registries and Pull Secrets + +On a cluster that pulls through a mirror, each chart can put a registry in front of the images it names and attach pull secrets to every pod it renders. The keys differ where a chart wraps an upstream one: + +| Chart | Mirror registry | Pull secrets | +|-------|-----------------|--------------| +| `kates`, `connect-cluster`, `mirror-maker2`, `headlamp`, `kafka-ui` | `global.imageRegistry`, in front of every image the chart names; in `kates`, `chaos.stressImage` too | `imagePullSecrets`; `kates` also reads `global.imagePullSecrets`, and takes names or `{name: …}` in both | +| `legacy-kafka` | `global.imageRegistry` for the official Kafka image; `legacyImageRegistry` for the built one | `imagePullSecrets` | +| `kafka-cluster` | `tests.imageRegistry` for the test images; `seaweedfs.image.registry`, with a trailing slash | `global.imagePullSecrets`, as names or `{name: …}`; plain names such as `[regcred]` when SeaweedFS is on, which reach its master, volume and filer but not its bucket hook | +| `kates-platform` | `global.imageRegistry`, in front of the Helm test's image; each subchart reads it as its own row says | `global.imagePullSecrets`, which the `kates` and `kafka-cluster` subcharts read; `imagePullSecrets` for the Helm test pod alone | +| `kates-chaos` | `global.imageRegistry`, except the litmus-core images, which you set under `litmus-core.operator`, `.runner` and `.exporter` | `global.imagePullSecrets`, which litmus-core's pods don't read | +| `monitoring` | `global.imageRegistry`, the wrapped stack's images included | `global.imagePullSecrets` | +| `strimzi-operator` | `strimzi-kafka-operator.defaultImageRegistry`; write the mirror into `testImages.kubectl` and `drainCleaner.image` | `strimzi-kafka-operator.image.imagePullSecrets` | +| `apicurio-registry` | `image.registry`, the Helm test's image included | `imagePullSecrets` | +| `velero` | None: set `image.repository`, `kubectl.image.repository` and `tests.image.repository` | `image.imagePullSecrets` | + +The Strimzi pods of `kafka-cluster`, `connect-cluster` and `mirror-maker2` run the operator's images, so `strimzi-kafka-operator.defaultImageRegistry` mirrors them. In `kafka-cluster`, `global.imageRegistry` is the registry of the Strimzi client image the tests use when `testImages.kafka` is empty. + +### Checking the Charts Against the Policies + +`scripts/check-chart-policies.sh` holds every chart to four sample cluster policies, kept in `.github/kyverno/policies/`: PSS restricted, CPU and memory requests with a memory limit, an image tag or digest that isn't `latest`, and a `team` label on every pod. It renders each chart with its defaults, and once more for each set of values that turns its optional workloads on, SeaweedFS and Velero's namespace-label job aside. Every render sets `team=ci` through the chart's `podLabels` and the other label keys [Pod Labels](#pod-labels) lists, so the label policy proves those keys reach every pod. The Kyverno CLI applies all four policies to each render, together with the exceptions in `.github/kyverno/exceptions/`. Those cover node-exporter, the Velero node agent and the Litmus pods from [Workloads That Cannot Pass](#workloads-that-cannot-pass), one control at a time. -| Setting | Value | Purpose | -|---------|-------|---------| -| `runAsNonRoot` | true | Prevents running as UID 0 — even if the container image specifies `USER root` | -| `readOnlyRootFilesystem` | true | No writes outside mounted volumes — prevents attackers from dropping malware | -| `allowPrivilegeEscalation` | false | Blocks `setuid` / `setgid` binaries — prevents privilege escalation within the container | -| `drop: ALL` | — | Removes all Linux capabilities — no raw sockets, no network admin, no filesystem mounts | +The script needs `helm`, the Kyverno CLI and the Helm repositories the charts depend on. Run it from the repository root, with no argument for every chart or with the charts to check: + +```bash +helm repo add seaweedfs https://seaweedfs.github.io/seaweedfs/helm +helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/ +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update + +# Every chart +scripts/check-chart-policies.sh +# Only the charts you name +scripts/check-chart-policies.sh kates kates-chaos +``` + +It prints one `PASS` or `FAIL` line per chart, with the failing resources and rules under it, and exits 1 on any failure. The exceptions in `.github/kyverno/exceptions/` are a template for your own cluster's: each names the workload, the policy and the controls it lifts, and says why. ### Quotas as Security @@ -714,6 +829,8 @@ The kates chart ships four `ClusterPolicy` resources. It renders them only when | `kates-image-verification` | Supply Chain | Critical | Verifies Cosign image signatures from trusted registries. Off by default: needs `kyvernoPolicy.cosign.enabled` and a `publicKey` | | `kates-generate-network-policies` | Network Security | Medium | Generates default-deny NetworkPolicies in namespaces created after it. Off by default: needs `kyvernoPolicy.networkPolicyGeneration.enabled`, which `kates kyverno apply --with-netpol` sets | +A `ClusterPolicy` is cluster-scoped, so with `rbac.scope=namespace` the chart renders none of the four, nor the exceptions of `kyvernoPolicy.policyExceptions`, which live in other namespaces. An account with edit rights in the release namespace alone could create neither. `kyvernoPolicy.exceptions` still renders under that scope, because its `PolicyException` lives in the release namespace (see [Policy Exceptions](#policy-exceptions)). + ### Pod Security Standards (Mutate + Validate) The `kates-pod-security-standards` policy combines **mutation** (auto-patching) with **validation** (enforcement). This two-phase approach is deliberate: mutation silently fixes common mistakes so developers don't have to remember every security setting, while validation catches anything mutation couldn't fix. @@ -790,7 +907,67 @@ Certain system and infrastructure namespaces are excluded from automatic Network ### Policy Exceptions -For development environments where strict enforcement would impede workflows, Kates supports `PolicyException` CRDs (Kyverno `v2` API). These temporarily relax specific validation rules for designated namespaces without disabling the entire policy: +A `PolicyException` (Kyverno `v2` API) exempts named resources from named rules without turning the whole policy off. The charts render two kinds, from two values. + +The first, `kyvernoPolicy.exceptions` on the `kates` and `kates-chaos` charts, exempts some of the release's own workloads from rules of any Kyverno policy, the cluster's own included. In `kates` those are every pod whose name label is `kates` or `kates-postgresql`: the API, PostgreSQL, the backup, cleanup and migration jobs and the Helm test pods. In `kates-chaos` they are the Litmus operator and exporter. The value is a list of `{policyName, ruleNames}` entries and needs no `kyvernoPolicy.enabled`. No other chart has it: to exempt another chart's pod, write a `PolicyException` of your own, like those in `.github/kyverno/exceptions/`. A cluster that enforces restricted through a policy named `pss-restricted` admits the Litmus operator with these values on the `kates-chaos` release: + +```yaml +kyvernoPolicy: + exceptions: + - policyName: pss-restricted # your cluster's PSS policy + ruleNames: [restricted] # the rule to exempt from +``` + +For the `chaos` release that `kates deploy` installs in `litmus`, the chart renders one `PolicyException` in the release namespace, here with its labels left out: + +```yaml +apiVersion: kyverno.io/v2 +kind: PolicyException +metadata: + name: chaos-kates-chaos-exceptions + namespace: litmus +spec: + exceptions: + - policyName: "pss-restricted" + ruleNames: + - "restricted" + - "autogen-restricted" + - "autogen-cronjob-restricted" + match: + any: + - resources: + kinds: [Pod, Deployment, ReplicaSet, StatefulSet, Job, CronJob] + namespaces: ["litmus"] + selector: + matchLabels: + app.kubernetes.io/instance: "chaos" + matchExpressions: + - key: app.kubernetes.io/name + operator: In + values: ["litmus"] +``` + +Reading the render: + +1. Each rule comes with the `autogen-` and `autogen-cronjob-` rules Kyverno generates from it for controllers and CronJobs, so the Deployment is exempt as well as its pods. +2. The selector matches the release's Litmus operator and exporter, which carry the release and litmus-core's name, `litmus`. The chart's own jobs and test pods pass on their own and stay under the policy, and so do the Litmus runner and experiment pods, which carry no `app.kubernetes.io/instance` label. In the `kates` chart the selector matches every pod whose `app.kubernetes.io/name` is `kates` or `kates-postgresql` instead, its jobs and Helm test pods included. +3. A namespaced `Policy` is named `/` in `policyName`. + +Kyverno must accept exceptions from the release namespace, which its `exceptionNamespace` setting decides. + +The second kind relaxes the `kates` chart's own `kates-pod-security-standards` policy for whole namespaces, such as development ones. It comes from `kyvernoPolicy.policyExceptions`, needs `kyvernoPolicy.enabled` and `policyExceptions.enabled`, and renders nothing under `rbac.scope=namespace`. These values exempt the `litmus` namespace from two of the policy's rules: + +```yaml +kyvernoPolicy: + enabled: true + policyExceptions: + enabled: true + namespaces: [litmus] + exemptPolicies: [kates-pod-security-standards] + exemptRules: [validate-readonly-rootfs, validate-drop-capabilities] +``` + +The chart renders one `PolicyException` per namespace and policy, in that namespace, again shown without its labels and annotations: ```yaml apiVersion: kyverno.io/v2 @@ -811,8 +988,6 @@ spec: kinds: [Pod, Deployment, StatefulSet, ReplicaSet] ``` -Exceptions are configured via Helm values under `kyvernoPolicy.policyExceptions` and deployed only when `policyExceptions.enabled=true`. - ### Enforce vs Audit Modes All Kyverno policies support two operational modes, controlled by the `kyvernoPolicy.action` Helm value: @@ -871,7 +1046,7 @@ Use this checklist when auditing your deployment. Each item links to the section - [ ] `superUsers` list contains only `kates-backend`, the Kates API's principal — see [User Permissions Matrix](#user-permissions-matrix) - [ ] Each service has its own `KafkaUser` with minimum required ACLs — see [Adding a New Service](#adding-a-new-service) - [ ] Every listener carries `networkPolicyPeers` (an external one declared in `kafka.listeners`, since the `externalAccess` preset cannot carry them), and a pod with no grant times out on the listener ports — see [Closing the Listeners](#closing-the-listeners) -- [ ] Containers run as non-root with read-only root filesystem — see [Container Security](#container-security) +- [ ] Every pod passes Pod Security restricted, and each one that can't is off or exempt by name — see [Container Security](#container-security) - [ ] The operator's certificate-expiry alerts are loaded (`kubectl get prometheusrule -n strimzi-operator`) — see [Rotation Monitoring](#rotation-monitoring) - [ ] Per-user quotas limit blast radius from runaway clients — see [Quotas as Security](#quotas-as-security) - [ ] `deleteClaim: false` on all PVCs (data survives pod deletion) — see [Kafka Deployment Engineering](15-kafka-deployment.md) @@ -921,7 +1096,7 @@ Expect a graded report organized by category — authentication, transport secur - Every listener authenticates, but only some encrypt: `plain` (9092) uses SCRAM-SHA-512 without TLS so performance baselines isolate Kafka throughput from encryption cost — production traffic belongs on `tls` (9093) or `external` (9094). - Authorization is deny-by-default: each service gets its own `KafkaUser` with `patternType: prefix` ACLs and quotas, and only `kates-backend`, the Kates API's user, holds superUser status. - SCRAM passwords live in base64-encoded — not encrypted — Kubernetes Secrets, so RBAC on Secrets in the `kafka` and `kates` namespaces is the wall around your credentials. Kates reads its copy of the `kates-backend` password once, at startup, and nothing keeps that copy in step, so a rotation is rotate, re-copy, restart. -- Kyverno enforces restricted Pod Security Standards in two phases — mutation silently patches missing settings, validation rejects what mutation can't fix — and starts in `Audit` mode so you can review violations before switching to `Enforce`. +- The charts' pods pass restricted Pod Security Standards by default, bar a few named host-level and upstream workloads; Kyverno patches, then validates, and starts in `Audit` mode. - As shipped, every pod can reach the client listeners; giving each listener `networkPolicyPeers` makes `networkPolicy.clients` the allow list wherever the chart's policies render. - `kates security audit` grades the whole posture A–F against CIS-mapped checks, while `make kafka-verify-policies` checks Kyverno, the Strimzi operator and the Kafka CR, and probes whether a pod with no grant can reach the brokers. diff --git a/docs/book/20-installation-guide.md b/docs/book/20-installation-guide.md index e4b4bd3d..6241c731 100644 --- a/docs/book/20-installation-guide.md +++ b/docs/book/20-installation-guide.md @@ -123,6 +123,8 @@ The `kates` chart (`charts/kates`), which installs the Kates API, ships addition | `kates-image-verification` | Verifies Cosign container image signatures from trusted registries | | `kates-generate-network-policies` | Automatically generates default-deny NetworkPolicies in new namespaces | +The charts' own pods pass restricted PSS with default values, apart from a few host-level and upstream workloads. [Container Security](17-security.md#container-security) lists what each chart sets and those workloads' switches, and [Policy Exceptions](17-security.md#policy-exceptions) shows how `kyvernoPolicy.exceptions` exempts the Litmus operator from your cluster's own policy. + ::: {.callout-tip} Start with `kyvernoPolicy.action: Audit` (the default) to observe policy violations without blocking deployments. Switch to `Enforce` once you're confident all workloads comply. See [Security & Compliance](17-security.md) for details on each policy. ::: @@ -135,7 +137,7 @@ Start with `kyvernoPolicy.action: Audit` (the default) to observe policy violati |---|---|---| | Velero with its CRDs, running in the `velero` namespace | `backup.enabled` renders a `velero.io/v1` `Schedule` in `backup.veleroNamespace` (`velero`), and a pre-upgrade `Backup` hook | `helm upgrade --install` fails with `no matches for kind "Schedule" in version "velero.io/v1"` — the chart does not check for the API first | | A `BackupStorageLocation` named `seaweedfs` in `velero`, whose bucket exists and whose credentials the store accepts | `backup.storageLocation: seaweedfs` | the install succeeds and every backup ends `FailedValidation` | -| Velero's node agent | `backup.volumes: fs-backup` | the backups keep the objects but not the broker and controller volume data | +| Velero's node agent, in a `velero` namespace that admits privileged pods ([it can't pass restricted](17-security.md#workloads-that-cannot-pass)) | `backup.volumes: fs-backup` | the backups keep the objects but not the broker and controller volume data | | A Secret `kafka-seaweedfs-credentials` in `kafka`, with keys `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` | `seaweedfs.s3.existingSecret`, so the chart renders no credentials Secret of its own | nothing reads it while tiered storage is off; with tiered storage on, the brokers cannot start | | amd64 nodes, or the subchart's pin to them cleared | the SeaweedFS subchart, which gives its master, volume and filer pods `nodeSelector: kubernetes.io/arch: amd64` | on a cluster with no amd64 nodes the SeaweedFS pods stay `Pending` and the `--wait` of [Step 5 — Install the Chart](#step-5--install-the-chart) times out. The SeaweedFS image is multi-arch, so setting `seaweedfs.master.nodeSelector`, `seaweedfs.volume.nodeSelector` and `seaweedfs.filer.nodeSelector` to `""` lets them run on arm64 too | | Kyverno (see [Kyverno (Optional)](#kyverno-optional)) | `kyvernoPolicy.enabled` with `action: Enforce` | no error: the chart renders its `ClusterPolicy` only where the `kyverno.io/v1` API exists, and the release notes report it `not rendered`. With Kyverno present, read the warning below first | @@ -146,16 +148,12 @@ Nothing in the repository creates the storage location for you, and `make velero `helm dependency build` downloads the SeaweedFS subchart, so the machine that installs needs the `helm repo add seaweedfs` of [Step 3 — Build the Chart Dependencies](#step-3--build-the-chart-dependencies) and a route to `https://seaweedfs.github.io/seaweedfs/helm`. -::: {.callout-warning} -With Kyverno installed, the `Enforce` policy of `values-prod.yaml` — and of `values-staging.yaml` — refuses pods that the same release depends on. Its checks are patterns on the Pod spec, and only pods labelled `strimzi.io/kind` are exempt from some of them: - -- The SeaweedFS pods `values-prod.yaml` enables — master, volume and filer, and the bucket hook of [SeaweedFS](#seaweedfs) — set no security context at all, so they fail the non-root, capabilities, seccomp and privilege-escalation checks. Their StatefulSets never get a pod, and the `--wait` of [Step 5 — Install the Chart](#step-5--install-the-chart) times out. -- `require-seccomp` exempts no pod and wants a pod-level `seccompProfile`. Of the Strimzi pods, the chart gives one to the node pools only, so the Entity Operator, Cruise Control and Kafka Exporter pods are refused — and without the Entity Operator no topic or user is ever created. -- Two `helm test` pods, `krafter-test-produce-consume` and `krafter-test-performance`, drop no capabilities and fail `restrict-capabilities`. - -Until the chart's own workloads satisfy the policy, layer a file with `kyvernoPolicy.action: Audit` after the overlay, and read `kubectl get policyreport -n kafka` for what `Enforce` would refuse. +::: {.callout-warning title="Enforce refuses the SeaweedFS pods"} +With Kyverno installed, the `Enforce` policy of `values-prod.yaml` refuses the SeaweedFS pods that overlay turns on: master, volume and filer, and the bucket hook of [SeaweedFS](#seaweedfs), which set no security context. Their StatefulSets never get a pod, and the `--wait` of [Step 5 — Install the Chart](#step-5--install-the-chart) times out. Set `seaweedfs.enabled: false` and point Velero at an object store outside the namespace, or layer a file with `kyvernoPolicy.action: Audit` after the overlay. ::: +Every other pod of `values-prod.yaml` and `values-staging.yaml` passes their `Enforce` policy: the Helm test pods, and the Entity Operator, Cruise Control and Kafka Exporter, which carry the restricted settings in their custom resources' templates. [Container Security](17-security.md#container-security) lists what the chart sets. + --- ## Understanding the Chart @@ -1632,12 +1630,19 @@ seaweedfs: existingConfigSecret: seaweedfs-s3-identities # identities that include the brokers' keys ``` +The hook's Job takes no resources, pod labels or pull secrets from any value, so a policy that requires resources or labels refuses it, and a mirror that needs a pull secret leaves it unable to pull. On such a cluster, leave `createBuckets` unset and create the buckets another way. + ::: {.callout-warning} With `filer.s3.enableAuth: true`, as in `values-prod.yaml`, the SeaweedFS S3 gateway accepts only the identities in the subchart's `seaweedfs-s3-secret`, whose keys it generates at random on install — not the keys in `seaweedfs.s3.existingSecret`, which is what the brokers and the object-store ConfigMap use. The chart does not connect the two. Give the gateway identities that include those keys with `seaweedfs.filer.s3.existingConfigSecret` (a Secret whose `seaweedfs_s3_config` key holds SeaweedFS's identities JSON), or hand the keys from `seaweedfs-s3-secret` to whatever writes to the store. ::: -::: {.callout-caution} -The SeaweedFS subchart keeps its data in `hostPath` directories — under `/ssd` for the master and volume servers, `/storage` for the filer — on whichever node each pod runs, and nothing reads `seaweedfs.volume.storage`. A pod rescheduled onto another node starts empty there, and a lost node takes its share of the offloaded segments and backups with it. For data you intend to keep, put the data directories on PersistentVolumeClaims: +With the values this chart ships, SeaweedFS fails Pod Security restricted: its master, volume and filer mount `hostPath` directories for their data and logs and run as root with no requests or limits, as upstream ships them. The subchart's own keys can make them pass: `podSecurityContext` and `containerSecurityContext` with `enabled: true`, `resources`, `data.type: persistentVolumeClaim` (`dataDirs` for the volume server) and `logs.type: emptyDir` under each of `seaweedfs.master`, `seaweedfs.volume` and `seaweedfs.filer`. Choose them before the first install, because Kubernetes refuses to give an existing StatefulSet claim templates. Otherwise, in a namespace that enforces restricted, keep `seaweedfs.enabled: false` and point tiered storage and Velero at an object store outside the namespace ([Workloads That Cannot Pass](17-security.md#workloads-that-cannot-pass)). + +::: {.callout-caution title="SeaweedFS keeps its data on the node"} +The SeaweedFS subchart keeps its data in `hostPath` directories — under `/ssd` for the master and volume servers, `/storage` for the filer — on whichever node each pod runs, and nothing reads `seaweedfs.volume.storage`. A pod rescheduled onto another node starts empty there, and a lost node takes its share of the offloaded segments and backups with it. For data you intend to keep, put the data directories on PersistentVolumeClaims with the values below. +::: + +These values give each component a claim. Kubernetes doesn't let an existing StatefulSet gain a claim template, so choose before the first install: ```yaml seaweedfs: @@ -1657,9 +1662,6 @@ seaweedfs: size: 20Gi ``` -Kubernetes does not let an existing StatefulSet gain a claim template, so choose before the first install. -::: - ### Velero Backup **Why it exists:** Even with replicated data, you need to back up the *cluster topology* — the CRDs, Secrets, and ConfigMaps that define your Kafka cluster. Without them, you'd have to recreate every topic, user, and ACL from scratch after a disaster. @@ -1741,7 +1743,7 @@ kyvernoPolicy: excludeStrimziPods: true # Don't mutate Strimzi-managed pods (operator handles them) ``` -`values-staging.yaml` and `values-prod.yaml` set `action: Enforce`, which some of the chart's own pods do not pass; [Production Prerequisites](#production-prerequisites) names them. +`values-staging.yaml` and `values-prod.yaml` set `action: Enforce`, which every pod those overlays render passes, except the SeaweedFS pods `values-prod.yaml` turns on; [Production Prerequisites](#production-prerequisites) says what to do about them. A cluster that requires a label on every pod sets it through `podLabels`, which reaches every pod the chart renders, the Strimzi components' pod templates included ([Pod Labels](17-security.md#pod-labels)). SeaweedFS's master, volume and filer take theirs from `seaweedfs.podLabels`, and its bucket hook takes none. ::: {.callout-tip} Always start with `action: Audit`. Run `kubectl get policyreport -A` to see which pods would be blocked, then fix them before switching to `Enforce`. diff --git a/docs/book/21-kafka-connect.md b/docs/book/21-kafka-connect.md index 413f7637..33d13c1b 100644 --- a/docs/book/21-kafka-connect.md +++ b/docs/book/21-kafka-connect.md @@ -345,7 +345,7 @@ While the pre-built image contains the most common CDC connectors, you may need | Way | How | When | |-----|-----|------| | `image` | A Connect image with the plugins inside | Recommended — bake them in (`make connect-build`) or layer your own image | -| `build` | The operator builds and pushes an image (`spec.build`) | A registry the operator can push to | +| `build` | The operator builds and pushes an image (`spec.build`) | A registry the operator can push to, and a namespace and operator that allow its root build pod ([Workloads That Cannot Pass](17-security.md#workloads-that-cannot-pass)) | | `plugins` | OCI images mounted as volumes at start-up (`spec.plugins`) | No custom image, and every node has the Kubernetes ImageVolume feature | The last two add plugins without you rebuilding the Docker image: diff --git a/docs/book/appendix-a-glossary.md b/docs/book/appendix-a-glossary.md index f41c7a05..3b8b2089 100644 --- a/docs/book/appendix-a-glossary.md +++ b/docs/book/appendix-a-glossary.md @@ -506,7 +506,7 @@ See [PDB](#gl-pdb). ### PolicyException {#gl-policyexception} -A Kyverno resource that exempts named resources or namespaces from specific policy rules without disabling the policy. See [Security & Compliance](17-security.md#policy-exceptions). +A Kyverno resource that exempts named resources or namespaces from specific policy rules without disabling the policy. In Kates, the `kates` and `kates-chaos` charts render one from `kyvernoPolicy.exceptions` for some of their own pods, the Litmus operator among them, against any policy the cluster runs. See [Security & Compliance](17-security.md#policy-exceptions). ### PolicyReport {#gl-policyreport} diff --git a/docs/book/appendix-b-troubleshooting.md b/docs/book/appendix-b-troubleshooting.md index efbe938b..eb08b2f7 100644 --- a/docs/book/appendix-b-troubleshooting.md +++ b/docs/book/appendix-b-troubleshooting.md @@ -66,7 +66,7 @@ Pods, images or Helm releases fail while you install the stack or roll it out. | PDB blocks rolling restart | Only 1 pod can be unavailable — intentional safety behavior | [Upgrade Playbook](18-upgrade-playbook.md#common-upgrade-issues) | | Entity Operator never starts | Kafka CR hasn't reached `Ready` — check operator logs for `UnforceableProblem` | [Kafka Deployment Engineering](15-kafka-deployment.md#strimzi-operator-cannot-determine-active-controller) | | PostgreSQL pod `CrashLoopBackOff` with `could not create lock file` | `readOnlyRootFilesystem: true` mutated by Kyverno — mount `emptyDir` at `/var/run/postgresql` and `/tmp` | [Deployment Guide](12-deployment.md#read-only-filesystem-compliance) | -| Pod admission rejected with Kyverno policy violation | Pod doesn't meet PSS standards — run `kates kyverno violations` to identify failing rules, then fix the manifest or add a `PolicyException` | [Security & Compliance](17-security.md#cluster-policies) | +| Pod admission rejected with Kyverno policy violation | Pod doesn't meet PSS standards — run `kates kyverno violations`, then fix the values, or exempt it with a `PolicyException` (`kyvernoPolicy.exceptions` in `kates` and `kates-chaos`) | [Security & Compliance](17-security.md#workloads-that-cannot-pass) | ## CLI Issues diff --git a/docs/book/appendix-c-cicd.md b/docs/book/appendix-c-cicd.md index f3c38ef1..9f6a3ef9 100644 --- a/docs/book/appendix-c-cicd.md +++ b/docs/book/appendix-c-cicd.md @@ -73,7 +73,7 @@ The filters used to be workflow-level `paths:` blocks, and the move is not cosme | Workflow | Filtered on | |----------|-------------| -| `ci.yml` | `code` (`kates/**`, `cli/**`, `images.env`, `versions.env`, the workflow and composite actions), `helm` (`charts/**`, `versions.env`, `scripts/check-versions.sh`, `config/cluster.yaml`), `yaml` (`config/**`) | +| `ci.yml` | `code` (`kates/**`, `cli/**`, `images.env`, `versions.env`, the workflow and composite actions), `helm` (`charts/**`, `versions.env`, `.github/kyverno/**`, `scripts/check-chart-policies.sh`, `scripts/check-versions.sh`, `config/cluster.yaml`, `tester/**`), `yaml` (`config/**`) | | `ci-kafka-charts.yml` | `charts/kafka-cluster/**`, `charts/connect-cluster/**`, `charts/mirror-maker2/**`, `charts/kafka-common/**`, `charts/strimzi-operator/**`, `charts/monitoring/**`, `charts/kates/**`, `charts/kates-chaos/**`, the checker scripts (`scripts/check-strimzi-crs.py`, `check-strimzi-metrics.sh`, `check-chart-matrix.py`, `check-metric-contract.sh`), `scripts/chart-matrix/**`, `scripts/metric-contract/**`, `dashboards/**`, `images.env`, `versions.env` | | `ci-mirror-maker2.yml` | `charts/mirror-maker2/**`, `charts/kafka-common/**`, `charts/legacy-kafka/**`, `charts/kafka-cluster/**` (the migration e2e stands a real source cluster up), `Dockerfile.legacy-kafka`, the migration scripts, `scripts/metric-contract/**`, `cli/cmd/migrate*.go`, `cli/pkg/migrate/**`, `cli/internal/podrun/**`, `docs/mirror-maker2-runbook.md` | | `ci-docker.yml` | `kates/**`, `cli/**`, `images.env`, `versions.env` | @@ -102,12 +102,12 @@ Every workflow also shares three conventions: a `concurrency` group per branch ( | Job | Runtime | Description | |-----|---------|-------------| -| **Detect Changes** | `dorny/paths-filter` | Splits the run into `code`, `helm` and `yaml` categories. Every job below keys its `if:` off one of them, except Kyverno Policy Validation, which always runs | +| **Detect Changes** | `dorny/paths-filter` | Splits the run into `code`, `helm` and `yaml` categories. Every job below keys its `if:` off one of them; Kyverno Policy Validation runs on `helm` or `code` | | **Build & Test** | Java 21 (Temurin) | Compiles the Kates API, a Quarkus application, and runs the test suite with `./mvnw verify`, publishes the surefire *and* failsafe JUnit reports as a check, uploads the JaCoCo report and writes its totals to the run summary | | **Dependency CVE scan** | — | Trivy over the Maven tree and `govulncheck` over the Go module, as its own job so a new CVE and a failing test are two different red checks | | **CLI Tests** | Go 1.25 | Runs `go test -race` across all CLI packages, then the terminal-compatibility and CLI style harnesses | | **Helm Lint** | Helm v3.17.0 | Lints, renders and schema-validates every chart under `charts/`, and runs the repository's cross-file consistency checks | -| **Kyverno Policy Validation** | Kyverno CLI v1.13.0 | Renders the Kyverno policy templates from the `kates`, `kafka-cluster`, and `kates-chaos` charts and validates them with `kyverno apply` | +| **Kyverno Policy Validation** | Kyverno CLI v1.13.0 | Holds every chart to the cluster policies in `.github/kyverno/`, then applies the `kates`, `kafka-cluster` and `kates-chaos` charts' own policies to their workloads | | **YAML Validation** | Python + PyYAML | Parses every YAML file under `config/` to catch syntax errors | ### Key Details @@ -133,6 +133,18 @@ This job grew well past `helm lint charts/kates`. In order: | kubeconform against each render | Rendered manifests satisfy the Kubernetes OpenAPI schemas and, for custom resources, the community CRD catalog. `-ignore-missing-schemas` keeps kinds with no published schema from failing the build | | `scripts/gen-chart-table.sh --check` | The chart table in the root `README.md` still matches `charts/*/Chart.yaml` | +### What Kyverno Policy Validation Checks + +The job holds every chart to the policies a cluster that enforces Pod Security would apply, in three steps: + +| Step | What it asserts | +|------|-----------------| +| Fetch chart dependencies (`.github/actions/helm-deps`) | `connect-cluster`, `kafka-cluster`, `kates-chaos`, `mirror-maker2`, `monitoring` and `strimzi-operator` build their dependencies; `kates-platform` updates its own in a copy | +| `scripts/check-chart-policies.sh` | Every render, with defaults and with optional workloads on, passes PSS restricted, requests with a memory limit, a tag that isn't `latest`, and a `team` label on every pod | +| Each chart's own policies | `kates`, `kafka-cluster` and `kates-chaos`, rendered with `kyvernoPolicy.enabled=true`, pass their own `ClusterPolicy`, and a run that matches no workload fails | + +Each render sets the `team` label through the chart's `podLabels` and every other label key that [Pod Labels](17-security.md#pod-labels) lists for it, so the label policy fails wherever those keys miss a pod. SeaweedFS, which can't pass with the values the chart ships, and Velero's namespace-label job, which can't pass at all, are never switched on. The only exceptions are the named ones in `.github/kyverno/exceptions/`, one file per workload, each lifting single Pod Security controls: node-exporter, Velero's node agent, and the Litmus operator and exporter. [Workloads That Cannot Pass](17-security.md#workloads-that-cannot-pass) explains each, and [Checking the Charts Against the Policies](17-security.md#checking-the-charts-against-the-policies) runs the script on your machine. + --- ## Kafka Charts CI (`ci-kafka-charts.yml`) diff --git a/docs/book/appendix-d-versions.md b/docs/book/appendix-d-versions.md index 9c29bba1..8840b94b 100644 --- a/docs/book/appendix-d-versions.md +++ b/docs/book/appendix-d-versions.md @@ -14,20 +14,20 @@ Every version the platform pins, in one place. This table is generated from the | Java (backend) | 21 | `kates/pom.xml` | | Quarkus | 3.20.6 | `kates/pom.xml` | | Go (CLI) | 1.26.0 | `cli/go.mod` | -| `apicurio-registry` chart | 0.4.0 (app 3.3.0) | `charts/apicurio-registry/Chart.yaml` | -| `connect-cluster` chart | 2.1.9 (app 4.3.1) | `charts/connect-cluster/Chart.yaml` | -| `headlamp` chart | 0.2.0 (app 0.40.1) | `charts/headlamp/Chart.yaml` | -| `kafka-cluster` chart | 1.0.11 (app 4.3.1) | `charts/kafka-cluster/Chart.yaml` | -| `kafka-common` chart | 0.1.0 (library) | `charts/kafka-common/Chart.yaml` | -| `kafka-ui` chart | 0.3.0 (app v1.5.0) | `charts/kafka-ui/Chart.yaml` | -| `kates-chaos` chart | 2.2.1 (app 3.28.0) | `charts/kates-chaos/Chart.yaml` | -| `kates-platform` chart | 0.8.2 (app 1.0.0) | `charts/kates-platform/Chart.yaml` | -| `kates` chart | 0.12.2 (app 1.28.0) | `charts/kates/Chart.yaml` | -| `legacy-kafka` chart | 0.2.0 (app 3.9.1) | `charts/legacy-kafka/Chart.yaml` | -| `mirror-maker2` chart | 0.11.7 (app 4.3.1) | `charts/mirror-maker2/Chart.yaml` | -| `monitoring` chart | 1.6.0 (app 82.4.3) | `charts/monitoring/Chart.yaml` | -| `strimzi-operator` chart | 0.3.7 (app 1.2.0) | `charts/strimzi-operator/Chart.yaml` | -| `velero` chart | 11.3.3 (app 1.17.1) | `charts/velero/Chart.yaml` | +| `apicurio-registry` chart | 0.4.1 (app 3.3.0) | `charts/apicurio-registry/Chart.yaml` | +| `connect-cluster` chart | 2.1.10 (app 4.3.1) | `charts/connect-cluster/Chart.yaml` | +| `headlamp` chart | 0.2.1 (app 0.40.1) | `charts/headlamp/Chart.yaml` | +| `kafka-cluster` chart | 1.0.12 (app 4.3.1) | `charts/kafka-cluster/Chart.yaml` | +| `kafka-common` chart | 0.1.1 (library) | `charts/kafka-common/Chart.yaml` | +| `kafka-ui` chart | 0.3.1 (app v1.5.0) | `charts/kafka-ui/Chart.yaml` | +| `kates-chaos` chart | 2.3.0 (app 3.28.0) | `charts/kates-chaos/Chart.yaml` | +| `kates-platform` chart | 0.8.3 (app 1.0.0) | `charts/kates-platform/Chart.yaml` | +| `kates` chart | 0.13.0 (app 1.28.0) | `charts/kates/Chart.yaml` | +| `legacy-kafka` chart | 0.2.1 (app 3.9.1) | `charts/legacy-kafka/Chart.yaml` | +| `mirror-maker2` chart | 0.11.8 (app 4.3.1) | `charts/mirror-maker2/Chart.yaml` | +| `monitoring` chart | 1.7.0 (app 82.4.3) | `charts/monitoring/Chart.yaml` | +| `strimzi-operator` chart | 0.4.0 (app 1.2.0) | `charts/strimzi-operator/Chart.yaml` | +| `velero` chart | 11.3.4 (app 1.17.1) | `charts/velero/Chart.yaml` | Regenerate with `scripts/gen-version-matrix.sh` and verify with `scripts/gen-version-matrix.sh --check` (CI runs the check on every book change). diff --git a/docs/book/deploying-strimzi-operator.md b/docs/book/deploying-strimzi-operator.md index 4c59c488..87bc3275 100644 --- a/docs/book/deploying-strimzi-operator.md +++ b/docs/book/deploying-strimzi-operator.md @@ -188,7 +188,9 @@ The handful of keys worth knowing: | `strimzi-kafka-operator.kubernetesServiceDnsDomain` | `cluster.local` | Upstream emits `KUBERNETES_SERVICE_DNS_DOMAIN` **only** when this differs from the default. Omitting it is invisible on a default cluster and silently breaks the operator on a custom domain — callers must inject the detected value. | | `strimzi-kafka-operator.leaderElection.enable` | `true` | `enable`, not `enabled`. See the troubleshooting section — the misspelling was live for a full release cycle. | | `strimzi-kafka-operator.resources` | memory only | The chart pins memory and lets cpu inherit upstream's defaults rather than duplicating them. | -| `strimzi-kafka-operator.defaultImageRegistry` | unset | Registry redirection. **Not** `global.imageRegistry` — upstream contains zero `.Values.global` references and ignores it entirely, so this chart does not declare `global` at all and `--set global.*` fails loudly. | +| `strimzi-kafka-operator.defaultImageRegistry` | unset | Registry redirection. **Not** `global.imageRegistry` — upstream contains zero `.Values.global` references and ignores it entirely, so this chart does not declare `global` at all and `--set global.*` fails loudly. This chart's own images, `testImages.kubectl` and `drainCleaner.image`, take your mirror as written. | +| `strimzi-kafka-operator.extraEnvs` | `STRIMZI_POD_SECURITY_PROVIDER_CLASS=restricted` | Every pod the operator creates gets the restricted Pod Security settings. Helm replaces a list whole, so keep this entry beside your own ([The Strimzi Restricted Provider](17-security.md#the-strimzi-restricted-provider)). | +| `podLabels` | `{}` | Labels on the pods this chart renders itself: the CRD-upgrade job, the Drain Cleaner and the Helm tests. The operator's pod takes `strimzi-kafka-operator.labels`. | | `crdUpgrade.enabled` | `true` | Disable it and your CRDs freeze. Only do this if another applier demonstrably owns them. | | `crdUpgrade.url` | derived | Override for airgapped mirrors. | | `operatorPolicy.enabled` | `true` | The operator's NetworkPolicy. While a `kafka-cluster` 0.4 release still owns the object, it is skipped and NOTES name the owner. | diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java b/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java index 0c4b0e3f..310aa969 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java @@ -513,7 +513,7 @@ private void executeCpuStress(FaultSpec spec) { spec.targetNamespace(), podName, "chaos-cpu-stress", - "polinux/stress", + stressImage, "stress", "--cpu", String.valueOf(spec.cpuCores()), @@ -529,7 +529,7 @@ private void executeIoStress(FaultSpec spec) { spec.targetNamespace(), podName, "chaos-io-stress", - "polinux/stress", + stressImage, "stress", "--io", String.valueOf(spec.ioWorkers()), @@ -538,6 +538,18 @@ private void executeIoStress(FaultSpec spec) { } } + static final String DEFAULT_STRESS_IMAGE = "docker.io/polinux/stress:1.0.4"; + + /** + * The image CPU_STRESS and IO_STRESS run as an ephemeral container. Written + * twice, like FaultLimits: in the annotation for the application, and as + * the field's value for a provider made with new, as tests make it. + */ + @org.eclipse.microprofile.config.inject.ConfigProperty( + name = "kates.chaos.stress-image", + defaultValue = DEFAULT_STRESS_IMAGE) + String stressImage = DEFAULT_STRESS_IMAGE; + private void injectEphemeralContainer( String namespace, String podName, String containerName, String image, String... command) { var podResource = client.pods().inNamespace(namespace).withName(podName); @@ -546,10 +558,44 @@ private void injectEphemeralContainer( throw new IllegalStateException("Pod not found: " + podName); } + // stress needs no privileges: run as the target's own non-root user and group, which its + // namespace already admits (OpenShift's restricted-v2 allows only the namespace's uid range + // and writes it on each container, not the pod), else as nobody, which polinux/stress is + // not by default. Either way the container passes Pod Security "restricted" there. + var podSc = pod.getSpec().getSecurityContext(); + Long uid = podSc == null || podSc.getRunAsUser() == null || podSc.getRunAsUser() == 0 + ? null + : podSc.getRunAsUser(); + Long gid = podSc == null ? null : podSc.getRunAsGroup(); + for (var container : pod.getSpec().getContainers()) { + var sc = container.getSecurityContext(); + if (sc == null) { + continue; + } + if (uid == null && sc.getRunAsUser() != null && sc.getRunAsUser() != 0) { + uid = sc.getRunAsUser(); + } + if (gid == null && sc.getRunAsGroup() != null) { + gid = sc.getRunAsGroup(); + } + } + EphemeralContainer ec = new EphemeralContainerBuilder() .withName(containerName) .withImage(image) .withCommand(command) + .withNewSecurityContext() + .withRunAsNonRoot(true) + .withRunAsUser(uid == null ? 65534L : uid) + .withRunAsGroup(gid == null ? 65534L : gid) + .withAllowPrivilegeEscalation(false) + .withNewCapabilities() + .withDrop("ALL") + .endCapabilities() + .withNewSeccompProfile() + .withType("RuntimeDefault") + .endSeccompProfile() + .endSecurityContext() .build(); pod.getSpec().getEphemeralContainers().add(ec); diff --git a/kates/src/main/resources/application.properties b/kates/src/main/resources/application.properties index 964a959f..29f7938e 100644 --- a/kates/src/main/resources/application.properties +++ b/kates/src/main/resources/application.properties @@ -384,6 +384,11 @@ kates.chaos.kafka.namespace=kafka kates.chaos.kafka.cluster=krafter kates.chaos.kafka.label=strimzi.io/component-type=kafka +# The image CPU_STRESS and IO_STRESS inject into the target pod as an ephemeral +# container (kubernetes provider). Set it to a mirror on a cluster that cannot +# pull from Docker Hub; the kates chart sets it from chaos.stressImage. +kates.chaos.stress-image=docker.io/polinux/stress:1.0.4 + # Test data cleanup (archive runs older than this many days) kates.cleanup.retention-days=90 diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java index 89df9368..92ededcd 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java @@ -10,6 +10,7 @@ import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; +import io.fabric8.kubernetes.api.model.EphemeralContainer; import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.api.model.PodBuilder; import io.fabric8.kubernetes.api.model.apps.StatefulSet; @@ -117,9 +118,23 @@ void testCpuStress() throws ExecutionException, InterruptedException, TimeoutExc assertEquals(subresource, write.getPath()); Pod sent = client.getKubernetesSerialization().unmarshal(write.getUtf8Body(), Pod.class); assertEquals(1, sent.getSpec().getEphemeralContainers().size()); - assertEquals( - "chaos-cpu-stress", - sent.getSpec().getEphemeralContainers().get(0).getName()); + EphemeralContainer stress = sent.getSpec().getEphemeralContainers().get(0); + assertEquals("chaos-cpu-stress", stress.getName()); + assertEquals("docker.io/polinux/stress:1.0.4", stress.getImage()); + // The pod names no user: stress runs as nobody. + assertRestricted(stress, 65534L, 65534L); + } + + /** What Pod Security "restricted" asks of a container, and the uid and gid it runs as. */ + private static void assertRestricted(EphemeralContainer stress, long uid, long gid) { + var sc = stress.getSecurityContext(); + assertNotNull(sc); + assertEquals(Boolean.TRUE, sc.getRunAsNonRoot()); + assertEquals(uid, sc.getRunAsUser()); + assertEquals(gid, sc.getRunAsGroup()); + assertEquals(Boolean.FALSE, sc.getAllowPrivilegeEscalation()); + assertEquals(List.of("ALL"), sc.getCapabilities().getDrop()); + assertEquals("RuntimeDefault", sc.getSeccompProfile().getType()); } @Test @@ -131,6 +146,11 @@ void testIoStress() throws ExecutionException, InterruptedException, TimeoutExce .addToLabels("app", "kafka2") .endMetadata() .withNewSpec() + // As kafka-cluster's pools set it; OpenShift admits only the namespace's uid range. + .withNewSecurityContext() + .withRunAsUser(1001L) + .withRunAsGroup(1001L) + .endSecurityContext() .addNewContainer() .withName("kafka") .withImage("kafka") @@ -141,6 +161,8 @@ void testIoStress() throws ExecutionException, InterruptedException, TimeoutExce String subresource = "/api/v1/namespaces/default/pods/broker-1/ephemeralcontainers"; server.expect().put().withPath(subresource).andReturn(200, pod).once(); + // kates.chaos.stress-image, as a cluster without Docker Hub sets it. + provider.stressImage = "registry.example.com/mirror/stress:1.0.4"; FaultSpec spec = FaultSpec.builder("test-io-stress") .targetNamespace("default") @@ -160,9 +182,94 @@ void testIoStress() throws ExecutionException, InterruptedException, TimeoutExce assertEquals(subresource, write.getPath()); Pod sent = client.getKubernetesSerialization().unmarshal(write.getUtf8Body(), Pod.class); assertEquals(1, sent.getSpec().getEphemeralContainers().size()); - assertEquals( - "chaos-io-stress", - sent.getSpec().getEphemeralContainers().get(0).getName()); + EphemeralContainer stress = sent.getSpec().getEphemeralContainers().get(0); + assertEquals("chaos-io-stress", stress.getName()); + assertEquals("registry.example.com/mirror/stress:1.0.4", stress.getImage()); + // The pod's own user and group, which its namespace already admits. + assertRestricted(stress, 1001L, 1001L); + } + + @Test + void stressNeverRunsAsRootEvenWhenThePodDoes() throws Exception { + Pod pod = new PodBuilder() + .withNewMetadata() + .withName("broker-2") + .withNamespace("default") + .addToLabels("app", "kafka3") + .endMetadata() + .withNewSpec() + .withNewSecurityContext() + .withRunAsUser(0L) + .withRunAsGroup(0L) + .endSecurityContext() + .addNewContainer() + .withName("kafka") + .withImage("kafka") + .endContainer() + .endSpec() + .build(); + client.pods().inNamespace("default").resource(pod).create(); + server.expect() + .put() + .withPath("/api/v1/namespaces/default/pods/broker-2/ephemeralcontainers") + .andReturn(200, pod) + .once(); + + FaultSpec spec = FaultSpec.builder("test-cpu-stress-root") + .targetNamespace("default") + .targetLabel("app=kafka3") + .disruptionType(DisruptionType.CPU_STRESS) + .cpuCores(1) + .chaosDurationSec(10) + .build(); + + assertTrue(provider.triggerFault(spec).get(5, TimeUnit.SECONDS).isPass()); + + Pod sent = client.getKubernetesSerialization() + .unmarshal(server.getLastRequest().getUtf8Body(), Pod.class); + // runAsNonRoot refuses uid 0, so stress falls back to nobody and keeps the pod's group. + assertRestricted(sent.getSpec().getEphemeralContainers().get(0), 65534L, 0L); + } + + @Test + void stressTakesTheContainersUserWhereOnlyTheContainerSetsIt() throws Exception { + // OpenShift's restricted-v2 writes the namespace's uid on each container, not the pod. + Pod pod = new PodBuilder() + .withNewMetadata() + .withName("broker-3") + .withNamespace("default") + .addToLabels("app", "kafka4") + .endMetadata() + .withNewSpec() + .addNewContainer() + .withName("kafka") + .withImage("kafka") + .withNewSecurityContext() + .withRunAsUser(1000650000L) + .endSecurityContext() + .endContainer() + .endSpec() + .build(); + client.pods().inNamespace("default").resource(pod).create(); + server.expect() + .put() + .withPath("/api/v1/namespaces/default/pods/broker-3/ephemeralcontainers") + .andReturn(200, pod) + .once(); + + FaultSpec spec = FaultSpec.builder("test-cpu-stress-container-uid") + .targetNamespace("default") + .targetLabel("app=kafka4") + .disruptionType(DisruptionType.CPU_STRESS) + .cpuCores(1) + .chaosDurationSec(10) + .build(); + + assertTrue(provider.triggerFault(spec).get(5, TimeUnit.SECONDS).isPass()); + + Pod sent = client.getKubernetesSerialization() + .unmarshal(server.getLastRequest().getUtf8Body(), Pod.class); + assertRestricted(sent.getSpec().getEphemeralContainers().get(0), 1000650000L, 65534L); } @Test diff --git a/scripts/check-chart-policies.sh b/scripts/check-chart-policies.sh new file mode 100755 index 00000000..78b5d9cc --- /dev/null +++ b/scripts/check-chart-policies.sh @@ -0,0 +1,187 @@ +#!/usr/bin/env bash +# Hold every chart under charts/ to the cluster policies in .github/kyverno/: +# Pod Security Standards "restricted", CPU and memory requests and a memory +# limit, an image tag that is not latest, and a sample rule that every Pod +# carries a team label. +# +# Each chart is rendered with its defaults and once per value set in +# optional_sets below (the values that switch optional workloads on: jobs, +# exporters, extra components), every render with the label team=ci set +# through the chart's label values (label_args below), and the Kyverno CLI +# applies all four policies to each render together with every +# PolicyException in .github/kyverno/exceptions/: the label policy proves +# those values reach every pod each render has. One line per chart, then every +# failure; exit 1 on any failure or error. +# +# Usage, from the repository root: +# scripts/check-chart-policies.sh [chart ...] +# Needs helm, kyverno and the Helm repositories the charts depend on (the CI +# job adds them with .github/actions/helm-deps). +set -uo pipefail + +policies=.github/kyverno/policies +exceptions=.github/kyverno/exceptions + +# Value sets that switch a chart's optional workloads on, one render per line. +# A chart that gains an optional workload adds its switch here. +optional_sets() { + case "$1" in + apicurio-registry) echo "--set ui.enabled=true" ;; + connect-cluster) + echo "--set preflight.enabled=true --set secretSync.watch=true --set secretSync.secrets[0].name=ci-user --set secretSync.secrets[0].fromNamespace=kafka" ;; + # Not seaweedfs.enabled: its hostPath volumes cannot pass, and it stays false under restricted. + kafka-cluster) + echo "--set tieredStorage.enabled=true --set tieredStorage.credentials.existingSecret=ci-s3 --set tieredStorage.image=registry.example/kafka-rsm:1.0.0 --set tieredStorage.remoteStorageManager.className=io.aiven.kafka.tieredstorage.RemoteStorageManager --set tieredStorage.remoteStorageManager.classPath=/opt/kafka/plugins/tiered-storage/*" ;; + kates) echo "--set backup.enabled=true --set migration.enabled=true --set cleanup.enabled=true" ;; + kates-chaos) echo "--set gameday.enabled=true --set litmus-core.exporter.enabled=true" ;; + kates-platform) + echo "--set apicurio-registry.enabled=true --set mirror-maker2.enabled=true --set connect-cluster.enabled=true" ;; + legacy-kafka) echo "--set mode=zookeeper" ;; + mirror-maker2) + echo "--set preflight.enabled=true --set secretSync.enabled=true --set secretSync.watch=true --set secretSync.secrets[0].name=ci-user --set secretSync.secrets[0].fromNamespace=kafka --set tests.replication.enabled=true --set tests.offsetTranslation.enabled=true --set tests.failover.enabled=true" ;; + strimzi-operator) echo "--set drainCleaner.enabled=true --set drainCleaner.tls.certManager.enabled=true" ;; + velero) echo "--set deployNodeAgent=true --set cleanUpCRDs=true" ;; + esac +} + +# The values that label every pod a chart renders team=ci, in every render: the +# chart's podLabels, and the key of each wrapped subchart whose pods take their +# labels from one of its own. +label_args() { + local kps=kube-prometheus-stack + case "$1" in + kates-chaos) echo "--set-string podLabels.team=ci,litmus-core.customLabels.team=ci" ;; + kates-platform) + echo "--set-string podLabels.team=ci,kafka-cluster.podLabels.team=ci,kates.podLabels.team=ci,apicurio-registry.podLabels.team=ci,mirror-maker2.podLabels.team=ci,connect-cluster.podLabels.team=ci" ;; + monitoring) + echo "--set-string podLabels.team=ci,$kps.commonLabels.team=ci,$kps.grafana.podLabels.team=ci,$kps.prometheusOperator.podLabels.team=ci,$kps.kube-state-metrics.podLabels.team=ci,$kps.prometheus-node-exporter.podLabels.team=ci" ;; + strimzi-operator) echo "--set-string podLabels.team=ci,strimzi-kafka-operator.labels.team=ci" ;; + velero) echo "--set-string podLabels.team=ci,nodeAgent.podLabels.team=ci,kubectl.labels.team=ci" ;; + *) echo "--set-string podLabels.team=ci" ;; + esac +} + +for tool in helm kyverno; do + command -v "$tool" > /dev/null || { echo "check-chart-policies: $tool is not on PATH" >&2; exit 2; } +done +[ -d "$policies" ] || { echo "check-chart-policies: run from the repository root" >&2; exit 2; } + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT + +pod_policies=("$policies/pss-restricted.yaml" "$policies/require-requests-limits.yaml" "$policies/disallow-latest-tag.yaml" "$policies/require-pod-labels.yaml") +exception_args=() +for f in "$exceptions"/*.yaml; do + [ -f "$f" ] && exception_args+=(--exception "$f") +done + +if [ $# -gt 0 ]; then + charts=("$@") +else + # kates-platform last: it packages its file:// subcharts as they are on disk, + # their own dependencies included. + charts=() + for dir in charts/*/; do + c=$(basename "$dir") + [ "$c" = kates-platform ] || charts+=("$c") + done + [ -d charts/kates-platform ] && charts+=(kates-platform) +fi + +# deps_current : helm lists every dependency of the chart ok, and no +# file:// one changed on disk after its archive was built (helm compares only +# the version, which an edit leaves as it is). Then the archives the CI job's +# helm-deps step just fetched are not fetched again. +deps_current() { + local rows name repo status archive + rows=$(helm dependency list "charts/$1" 2> /dev/null | awk -F'\t' 'NR > 1 && NF > 1 { for (i = 1; i <= NF; i++) gsub(/^ +| +$/, "", $i); print $1 "\t" $3 "\t" $4 }') || return 1 + [ -n "$rows" ] || return 1 + while IFS=$'\t' read -r name repo status; do + [ "$status" = ok ] || return 1 + case "$repo" in + file://*) + for archive in "charts/$1/charts/$name"-[0-9]*.tgz; do + [ -f "$archive" ] || return 1 + [ -z "$(find "charts/$1/${repo#file://}" -type f -newer "$archive")" ] || return 1 + done ;; + esac + done <<< "$rows" +} + +# render [helm args...]: render the chart with those args and its label args, +# and apply the pod policies to it. +render() { + local rc label=defaults + [ $# -gt 0 ] && label="$*" + renders=$((renders + 1)) + if ! helm template "$chart" "$root/$chart" -n "$chart" ${labels[@]+"${labels[@]}"} "$@" > "$work/render.yaml" 2> "$work/helm.log"; then + errors=$((errors + 1)) + { echo " $label: helm template failed"; sed 's/^/ /' "$work/helm.log"; } >> "$work/report" + return + fi + kyverno apply "${pod_policies[@]}" ${exception_args[@]+"${exception_args[@]}"} --resource "$work/render.yaml" > "$work/kyverno.log" 2>&1 + rc=$? + # pass: 20, fail: 8, warn: 0, error: 0, skip: 0 + local counts + counts=$(sed -n 's/^pass: \([0-9]*\), fail: \([0-9]*\), warn: [0-9]*, error: \([0-9]*\), skip: \([0-9]*\).*/\1 \2 \3 \4/p' "$work/kyverno.log" | tail -1) + if [ -z "$counts" ]; then + errors=$((errors + 1)) + { echo " $label: kyverno apply exited $rc without a summary"; sed 's/^/ /' "$work/kyverno.log"; } >> "$work/report" + return + fi + local p f e s + read -r p f e s <<< "$counts" + passes=$((passes + p)) fails=$((fails + f)) errors=$((errors + e)) skips=$((skips + s)) + # A policy that matches nothing passes nothing: the old CI job did exactly that. + if [ $((p + f + s)) -eq 0 ]; then + errors=$((errors + 1)) + echo " $label: the policies matched no workload" >> "$work/report" + fi + if [ "$f" -gt 0 ] || [ "$e" -gt 0 ] || [ "$rc" -ne 0 ]; then + [ "$f" -gt 0 ] || [ "$e" -gt 0 ] || errors=$((errors + 1)) + { echo " $label:"; grep -vE '^(Warning:|Applying |pass: [0-9]|[[:space:]]*$)' "$work/kyverno.log" | sed 's/^/ /'; } >> "$work/report" + fi +} + +failed=0 +for chart in "${charts[@]}"; do + dir=charts/$chart + if [ ! -f "$dir/Chart.yaml" ]; then + echo "$chart: ERROR no chart at $dir"; failed=1; continue + fi + grep -q '^type: library' "$dir/Chart.yaml" && continue + renders=0 passes=0 fails=0 errors=0 skips=0 + : > "$work/report" + + root=charts + if grep -q '^dependencies:' "$dir/Chart.yaml"; then + cmd=build + # kates-platform's Chart.lock pins its file:// subcharts' versions and goes + # stale with every bump of one, so it is updated, in a copy of charts/ to + # leave the checkout's Chart.lock as it is. + if [ "$chart" = kates-platform ]; then + cmd=update root=$work/charts + rm -rf "$root" + cp -R charts "$work/" || { echo "$chart: ERROR could not copy charts/"; failed=1; continue; } + elif deps_current "$chart"; then + cmd= + fi + if [ -n "$cmd" ] && ! scripts/retry.sh 3 helm dependency "$cmd" --skip-refresh "$root/$chart" > "$work/deps.log" 2>&1; then + echo "$chart: ERROR helm dependency $cmd failed" + sed 's/^/ /' "$work/deps.log" + failed=1; continue + fi + fi + + read -r -a labels <<< "$(label_args "$chart")" + render + while read -r -a set_args <&3; do + [ ${#set_args[@]} -gt 0 ] && render "${set_args[@]}" + done 3<<< "$(optional_sets "$chart")" + + status=PASS + if [ "$fails" -gt 0 ] || [ "$errors" -gt 0 ]; then status=FAIL; failed=1; fi + echo "$chart: $status ($renders renders; pass $passes, fail $fails, error $errors, skip $skips)" + cat "$work/report" +done +exit "$failed" From a5947de3b12b73b6e683c88ad23a3e2401500a01 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Wed, 21 Oct 2026 23:53:50 +0200 Subject: [PATCH 2/3] fix(kates-chaos): GameDay and the Helm tests run kates-tester, which has the shell they need The GameDay Job and both helm test pods ran their checks with sh in registry.k8s.io/kubectl, which has no shell, so they ended in StartError and helm test and gameday.enabled always failed. They now run images.tester, ghcr.io/bmscomp/kates-tester:1.28.0, as its own uid and gid 1000, and scripts/check-versions.sh holds that pin to the kates appVersion like the Strimzi charts' test images. images.kubectl is the installer Job's image only. --- charts/kates-chaos/Chart.yaml | 13 ++++++---- charts/kates-chaos/README.md | 3 ++- charts/kates-chaos/templates/_helpers.tpl | 9 +++++++ charts/kates-chaos/templates/gameday.yaml | 2 +- .../templates/tests/test-connectivity.yaml | 6 ++--- charts/kates-chaos/values.schema.json | 1 + charts/kates-chaos/values.yaml | 25 +++++++++++-------- scripts/check-versions.sh | 4 ++- 8 files changed, 41 insertions(+), 22 deletions(-) diff --git a/charts/kates-chaos/Chart.yaml b/charts/kates-chaos/Chart.yaml index 19d3ba60..3ff1c0ac 100644 --- a/charts/kates-chaos/Chart.yaml +++ b/charts/kates-chaos/Chart.yaml @@ -1,8 +1,9 @@ apiVersion: v2 name: kates-chaos -# Minor bump: the installer, GameDay and test pods now run as uid 65534 with -# a PSS restricted securityContext and resources by default, so an installer -# or GameDay image that needs root stops starting. +# Minor bump: the installer, GameDay and test pods now run non-root with a +# PSS restricted securityContext and resources by default, so an installer or +# GameDay image that needs root stops starting; GameDay and the tests move to +# kates-tester, which has the shell they need. version: 2.3.0 # Tracks the LitmusChaos execution-plane version this chart deploys. appVersion: "3.28.0" @@ -27,10 +28,12 @@ annotations: artifacthub.io/category: skip-prediction artifacthub.io/prerelease: "false" artifacthub.io/changes: | + - kind: fixed + description: "The GameDay Job and both helm test pods never started: they run their checks with sh, and registry.k8s.io/kubectl has no shell (StartError: exec: \"sh\": executable file not found in $PATH), so helm test and gameday.enabled always failed. They now run images.tester, ghcr.io/bmscomp/kates-tester:1.28.0 (a shell and kubectl, pinned to the kates appVersion like every chart's test image), as its own uid and gid 1000. images.kubectl is now the installer Job's image only; a gameday.image override still wins" - kind: changed - description: "The experiment installer Job, the GameDay Job and both helm test pods pass Pod Security Standards restricted by default: seccompProfile RuntimeDefault, runAsNonRoot as uid/gid 65534 (registry.k8s.io/kubectl runs as root otherwise), allowPrivilegeEscalation false, every capability dropped, and CPU and memory requests with a memory limit (10m/64Mi, limit 128Mi). Set through experiments.installer.podSecurityContext/securityContext/resources (resources now reach the installer's two init containers too), the new gameday.podSecurityContext/securityContext/resources and the new tests block. An installer or GameDay image that must run as root needs its podSecurityContext set back" + description: "The experiment installer Job, the GameDay Job and both helm test pods pass Pod Security Standards restricted by default: seccompProfile RuntimeDefault, runAsNonRoot as uid/gid 65534 for the installer (registry.k8s.io/kubectl runs as root otherwise) and 1000 for GameDay and the tests, allowPrivilegeEscalation false, every capability dropped, and CPU and memory requests with a memory limit (10m/64Mi, limit 128Mi). Set through experiments.installer.podSecurityContext/securityContext/resources (resources now reach the installer's two init containers too), the new gameday.podSecurityContext/securityContext/resources and the new tests block. An installer or GameDay image that must run as root needs its podSecurityContext set back" - kind: changed - description: "With values-generic.yaml, whose kyvernoPolicy ClusterPolicy (still on, mutate true) patches a pod with no securityContext to uid 1001 at admission, the installer, GameDay and test pods now run as the uid 65534 they set themselves. litmus-core's operator and exporter take no securityContext from values: they are restricted only through that mutate policy or a kyvernoPolicy.exceptions PolicyException, so a release that turns kyvernoPolicy off on a cluster enforcing restricted needs the exception" + description: "With values-generic.yaml, whose kyvernoPolicy ClusterPolicy (still on, mutate true) patches a pod with no securityContext to uid 1001 at admission, the installer, GameDay and test pods now run as the uid they set themselves (65534 for the installer, 1000 for GameDay and the tests). litmus-core's operator and exporter take no securityContext from values: they are restricted only through that mutate policy or a kyvernoPolicy.exceptions PolicyException, so a release that turns kyvernoPolicy off on a cluster enforcing restricted needs the exception" - kind: added description: "kyvernoPolicy.exceptions, a list of {policyName, ruleNames}: one kyverno.io/v2 PolicyException (-exceptions, rendered where that API exists) in the release namespace exempting litmus-core's operator and exporter (app.kubernetes.io/instance= and app.kubernetes.io/name=litmus-core's nameOverride, litmus; the chart's own pods pass on their own, and chaos runner and experiment pods are not matched) from those rules of any Kyverno policy, the cluster's own included, each rule's autogen- and autogen-cronjob- variants too. litmus-core takes no securityContext from values, so its operator (and exporter) Deployment fails PSS restricted unless kyvernoPolicy's mutate rules patch it; this is how a cluster that enforces restricted without them exempts them" - kind: added diff --git a/charts/kates-chaos/README.md b/charts/kates-chaos/README.md index 823e0fa8..46be3aff 100644 --- a/charts/kates-chaos/README.md +++ b/charts/kates-chaos/README.md @@ -71,9 +71,10 @@ helm upgrade --install chaos charts/kates-chaos \ | `global.imagePullPolicy` | Pull policy for chart-managed pods | `IfNotPresent` | | `global.imagePullSecrets` | Pull secrets for chart-managed pods | `[]` | | `global.clusterDomain` | Cluster DNS domain | `cluster.local` | -| `images.kubectl.repository` | Image for installer Job / GameDay / tests | `registry.k8s.io/kubectl` | +| `images.kubectl.repository` | Image for the installer Job | `registry.k8s.io/kubectl` | | `images.kubectl.tag` | kubectl image tag | `v1.33.0` | | `images.kubectl.digest` | Pin by digest (wins over tag) | `""` | +| `images.tester` | Image for GameDay and the Helm tests, whose scripts need a shell; runs as uid 1000 | `ghcr.io/bmscomp/kates-tester:1.28.0` | | `images.goRunner.repository` | Default runtime for experiments | `litmuschaos/go-runner` | | `images.goRunner.tag` | go-runner image tag | `3.28.0` | diff --git a/charts/kates-chaos/templates/_helpers.tpl b/charts/kates-chaos/templates/_helpers.tpl index 33448056..17c591aa 100644 --- a/charts/kates-chaos/templates/_helpers.tpl +++ b/charts/kates-chaos/templates/_helpers.tpl @@ -94,6 +94,15 @@ experiments.image (string), then the images.kubectl dict. {{- end -}} {{- end }} +{{/* +Image GameDay and the Helm tests run: kates-tester, which has the shell their +scripts need, with global.imageRegistry prepended. +*/}} +{{- define "kates-chaos.testerImage" -}} +{{- $registry := (.Values.global | default dict).imageRegistry | default "" -}} +{{- if $registry -}}{{- printf "%s/%s" $registry .Values.images.tester -}}{{- else -}}{{- .Values.images.tester -}}{{- end -}} +{{- end }} + {{/* go-runner image used as the default for ChaosExperiment definitions. */}} diff --git a/charts/kates-chaos/templates/gameday.yaml b/charts/kates-chaos/templates/gameday.yaml index 5b2070da..5b522621 100644 --- a/charts/kates-chaos/templates/gameday.yaml +++ b/charts/kates-chaos/templates/gameday.yaml @@ -1,7 +1,7 @@ {{- if .Values.gameday.enabled }} {{- $ns := .Release.Namespace }} {{- $kafkaNamespace := include "kates-chaos.kafkaNamespace" . }} -{{- $image := .Values.gameday.image | default (include "kates-chaos.kubectlImage" .) }} +{{- $image := .Values.gameday.image | default (include "kates-chaos.testerImage" .) }} {{- $pullPolicy := .Values.global.imagePullPolicy | default "IfNotPresent" }} {{- $checks := .Values.gameday.checks }} {{- if not $checks }} diff --git a/charts/kates-chaos/templates/tests/test-connectivity.yaml b/charts/kates-chaos/templates/tests/test-connectivity.yaml index c851d5de..b9e50c5e 100644 --- a/charts/kates-chaos/templates/tests/test-connectivity.yaml +++ b/charts/kates-chaos/templates/tests/test-connectivity.yaml @@ -1,4 +1,4 @@ -{{- $kubectl := include "kates-chaos.kubectlImage" . }} +{{- $image := include "kates-chaos.testerImage" . }} {{- $pullPolicy := .Values.global.imagePullPolicy | default "IfNotPresent" }} {{- $kafkaNamespace := include "kates-chaos.kafkaNamespace" . }} {{- $testSA := printf "%s-test" (include "kates-chaos.fullname" .) }} @@ -80,7 +80,7 @@ spec: {{- end }} containers: - name: test - image: {{ $kubectl }} + image: {{ $image }} imagePullPolicy: {{ $pullPolicy }} {{- with .Values.tests.securityContext }} securityContext: @@ -126,7 +126,7 @@ spec: {{- end }} containers: - name: test - image: {{ $kubectl }} + image: {{ $image }} imagePullPolicy: {{ $pullPolicy }} {{- with .Values.tests.securityContext }} securityContext: diff --git a/charts/kates-chaos/values.schema.json b/charts/kates-chaos/values.schema.json index 297ce586..818a9245 100644 --- a/charts/kates-chaos/values.schema.json +++ b/charts/kates-chaos/values.schema.json @@ -40,6 +40,7 @@ "type": "object", "properties": { "kubectl": { "$ref": "#/definitions/image" }, + "tester": { "type": "string", "minLength": 1 }, "goRunner": { "$ref": "#/definitions/image" } } }, diff --git a/charts/kates-chaos/values.yaml b/charts/kates-chaos/values.yaml index e26d8d74..acbcb267 100644 --- a/charts/kates-chaos/values.yaml +++ b/charts/kates-chaos/values.yaml @@ -40,11 +40,15 @@ global: # Single source of truth for the images this chart manages directly. Set # `digest` to pin by content (takes precedence over `tag`). images: - # kubectl — used by the experiment installer Job, GameDay, and Helm tests + # kubectl — used by the experiment installer Job kubectl: repository: registry.k8s.io/kubectl tag: "v1.33.0" digest: "" + # tester — kates-tester, for GameDay and the Helm tests, whose scripts need + # the shell the kubectl image lacks. Runs as uid 1000. Pinned to the kates + # appVersion, like every chart's test image (scripts/check-versions.sh). + tester: "ghcr.io/bmscomp/kates-tester:1.28.0" # go-runner — default runtime for ChaosExperiment definitions goRunner: repository: litmuschaos/go-runner @@ -296,7 +300,7 @@ kyvernoPolicy: # template). Override to add your own. gameday: enabled: false - image: "" # empty = images.kubectl + image: "" # empty = images.tester backoffLimit: 1 ttlSecondsAfterFinished: 600 checks: [] @@ -307,12 +311,12 @@ gameday: memory: 64Mi limits: memory: 128Mi - # -- GameDay pod securityContext (PSS restricted; the kubectl image runs as - # root unless told otherwise) + # -- GameDay pod securityContext (PSS restricted; uid 1000 is kates-tester's + # own user) podSecurityContext: runAsNonRoot: true - runAsUser: 65534 - runAsGroup: 65534 + runAsUser: 1000 + runAsGroup: 1000 seccompProfile: type: RuntimeDefault # -- GameDay container securityContext (PSS restricted) @@ -322,7 +326,7 @@ gameday: drop: [ALL] # ── Helm tests (`helm test`) ───────────────────────────────────────────────── -# The two test pods run images.kubectl, PSS restricted by default. +# The two test pods run images.tester, PSS restricted by default. tests: # -- Test container resources resources: @@ -331,12 +335,11 @@ tests: memory: 64Mi limits: memory: 128Mi - # -- Test pod securityContext (the kubectl image runs as root unless told - # otherwise) + # -- Test pod securityContext (uid 1000 is kates-tester's own user) podSecurityContext: runAsNonRoot: true - runAsUser: 65534 - runAsGroup: 65534 + runAsUser: 1000 + runAsGroup: 1000 seccompProfile: type: RuntimeDefault # -- Test container securityContext diff --git a/scripts/check-versions.sh b/scripts/check-versions.sh index 8ae2a55e..6983a8ca 100755 --- a/scripts/check-versions.sh +++ b/scripts/check-versions.sh @@ -618,7 +618,8 @@ fi # --------------------------------------------------------------------------- # The tester image the Strimzi charts' Helm tests, hooks and secret-sync jobs -# run, and the Litmus experiments' probes and jobs. ghcr.io/bmscomp/kates-tester +# run, kates-chaos's GameDay and Helm tests, and the Litmus experiments' +# probes and jobs. ghcr.io/bmscomp/kates-tester # is published with each Kates release, so every pin carries the same tag, and # it is the kates chart's appVersion (docs/kafka-charts-refactor-plan.md, M4). # A file left behind runs tooling a release older than the rest. @@ -640,6 +641,7 @@ while IFS= read -r hit; do done < <(grep -HE '^[[:space:]]*[A-Za-z]+:[[:space:]]*"?ghcr.io/bmscomp/kates-tester:' \ charts/strimzi-operator/values*.yaml charts/kafka-cluster/values*.yaml \ charts/connect-cluster/values*.yaml charts/mirror-maker2/values*.yaml \ + charts/kates-chaos/values*.yaml \ config/litmus/experiments/*.yaml 2>/dev/null) if [[ -z "$kates_app_version" ]]; then echo "ERROR: could not read charts/kates/Chart.yaml appVersion" >&2 From 258950ddc33d09b73ae765b62cb555791e11e24c Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Wed, 21 Oct 2026 23:57:31 +0200 Subject: [PATCH 3/3] fix(ci): the Kyverno CLI moves to v1.16.4, the Kyverno kates deploy installs Kyverno Policy Validation failed on the three workloads with a named PolicyException (the Litmus operator, node-exporter, Velero's node agent): CLI v1.13.0 ignores the podSecurity controls those exceptions lift, so they failed pss-restricted anyway. v1.16.4, the version of the kyverno chart 3.6.4 that kates deploy installs, applies them: every chart passes. On it, kyverno apply -o writes one file per mutated resource into a directory, so the own-policies step gives it one. --- .github/workflows/ci.yml | 4 +++- docs/book/appendix-c-cicd.md | 2 +- versions.env | 5 ++++- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e3aa29f2..8108ad25 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -540,8 +540,10 @@ jobs: yq 'select(.kind == "ClusterPolicy" or .kind == "Policy")' "${out}.yaml" > "${out}-policies.yaml" NS="${chart}" yq 'select(.kind == "Pod" or .kind == "Deployment" or .kind == "StatefulSet" or .kind == "DaemonSet" or .kind == "Job" or .kind == "CronJob") | .metadata.namespace |= (. // strenv(NS))' "${out}.yaml" > "${out}-workloads.yaml" + # A directory: with more than one mutated resource the CLI writes one file each. + mkdir -p "${out}-mutated" kyverno apply "${out}-policies.yaml" --resource "${out}-workloads.yaml" \ - -o "${out}-mutated.yaml" > "${out}.log" 2>&1 || rc=1 + -o "${out}-mutated" > "${out}.log" 2>&1 || rc=1 cat "${out}.log" if grep -q '^pass: 0, fail: 0,' "${out}.log"; then echo "::error::${chart}: its own policies matched none of its workloads"; rc=1 diff --git a/docs/book/appendix-c-cicd.md b/docs/book/appendix-c-cicd.md index 9f6a3ef9..e50b908c 100644 --- a/docs/book/appendix-c-cicd.md +++ b/docs/book/appendix-c-cicd.md @@ -107,7 +107,7 @@ Every workflow also shares three conventions: a `concurrency` group per branch ( | **Dependency CVE scan** | — | Trivy over the Maven tree and `govulncheck` over the Go module, as its own job so a new CVE and a failing test are two different red checks | | **CLI Tests** | Go 1.25 | Runs `go test -race` across all CLI packages, then the terminal-compatibility and CLI style harnesses | | **Helm Lint** | Helm v3.17.0 | Lints, renders and schema-validates every chart under `charts/`, and runs the repository's cross-file consistency checks | -| **Kyverno Policy Validation** | Kyverno CLI v1.13.0 | Holds every chart to the cluster policies in `.github/kyverno/`, then applies the `kates`, `kafka-cluster` and `kates-chaos` charts' own policies to their workloads | +| **Kyverno Policy Validation** | Kyverno CLI v1.16.4 | Holds every chart to the cluster policies in `.github/kyverno/`, then applies the `kates`, `kafka-cluster` and `kates-chaos` charts' own policies to their workloads | | **YAML Validation** | Python + PyYAML | Parses every YAML file under `config/` to catch syntax errors | ### Key Details diff --git a/versions.env b/versions.env index 64abd79b..ce20e6e2 100755 --- a/versions.env +++ b/versions.env @@ -43,7 +43,10 @@ HELM_UNITTEST_VERSION="v0.8.2" # not here: cli/go.mod is its pin and setup-go reads it (go-version-file). HELM_VERSION="v3.17.0" KUBECONFORM_VERSION="v0.6.7" -KYVERNO_CLI_VERSION="v1.13.0" +# The Kyverno that kates deploy installs (chart 3.6.4, cli/cmd/deploy_components.go). +# 1.13 ignores the podSecurity controls of the PolicyExceptions in +# .github/kyverno/exceptions/, so scripts/check-chart-policies.sh fails on it. +KYVERNO_CLI_VERSION="v1.16.4" JAVA_VERSION="21" NODE_VERSION="22" ACTIONLINT_VERSION="1.7.12"