ci: Hardening and Improving chart/Graylog CI - #178
Draft
alix-graylog wants to merge 8 commits into
Draft
Conversation
alix-graylog
changed the title
The operator owns the pod template of the StatefulSet it manages and injects its own resource defaults of 500m CPU and 500M memory per container and per init container. mongo-rs.yaml overrode serviceAccountName and affinity but never containers or initContainers, so nothing in the chart - no value, no --set - could change what a MongoDB pod requested. The init containers matter as much as the containers: a pod reserves max(max(initContainer), sum(containers)), so each merely copying a binary into a shared volume at 500m sets the pod's effective floor whenever the containers ask for less. Overriding only the containers is not enough. Adds mongodb.resources, mongodb.agentResources and mongodb.initResources, patched onto the operator-owned template by container name, with defaults sized for a production replica set rather than inherited from the operator. mongod gets 2Gi: WiredTiger sizes its cache from the container limit and an idle 3-member replica set already sits at ~340Mi, so the operator's 500M limit left almost nothing before an OOMKill. CPU goes the other way, since the same replica set idles under 50m per member. A MongoDB pod's reservation therefore moves from 1000m/800M to 300m/1152Mi, and ci/ci-values.yaml shrinks all three blocks to 150m/640Mi for CI. Existing installs get a rolling restart of the replica set on upgrade.
helm-ct-install had no timeout, so a wedged install could burn the default 360-minute ceiling on every leg of the matrix. helm-extra-args --timeout=900s bounds the install itself, not namespace termination hanging on a finalizer. The step was also silent: helm --wait prints nothing until it succeeds or times out, so a stuck install looked exactly like a slow one for fifteen minutes. A watcher now streams not-yet-Running pods and recent warning events every 20s, and node capacity plus reservations are printed after the cluster comes up, which turns "Insufficient cpu" into a one-line diagnosis. Sizing moves entirely into charts/graylog/ci/ci-values.yaml, which ct discovers on its own. It was duplicated as --set flags, and those take precedence over the values file: the graylog memory request in ci-values.yaml was silently overridden and the two disagreed. Only the generated rootPassword is passed as a flag now, because it cannot be committed. The MongoDB operator is shrunk to 100m: its chart asks for 500m to reconcile a single one-member replica set. ci-values.yaml also sets graylog.terminationGracePeriodSeconds to 30. The chart default of 300s exists to let the preStop drain flush the journal, and ct pays it on every leg when it deletes the namespace, but nothing in CI holds data worth draining. The chart default is unchanged.
Answering "what happened in CI" meant opening a job and reading log output. Every job now writes what it learned to the run summary instead. - changes: which jobs run, which are skipped and why, plus the changed file list. A skipped run is now self-explanatory rather than suspicious. - lint: the ct lint verdict and a per-example table - rendered object count for each values overlay, and for a failure the offending key rather than Helm's generic schema error. Runs even when ct lint fails, so the table is not lost with it. - unittest: suite and test counts lifted from helm-unittest's own output, the two behaviour-script verdicts, and failing suites when something breaks. - install: per matrix leg, the result, how long the install took, node allocatable CPU and memory, and the pod requests as last observed. On failure it adds the matching error lines from the ct log, the pods that never reached Running, and recent warning events. The pod-request snapshot is taken by the watcher loop rather than at summary time: ct deletes its namespace when it finishes, so by then the workloads under test are gone and cannot be queried.
ct discovers charts/graylog/ci/ci-values.yaml on its own and the workflow passes no sizing flags, so nothing in a run said what the chart was linted or installed with. Both ct jobs now report it. The report comes from rendering the chart with that overlay, not from printing the overlay file. The overlay mentions only a handful of keys, so only the rendered result accounts for chart defaults - MongoDB most of all, where the container names being patched belong to the operator and the pod's reservation is max(max(initContainer), sum(containers)) rather than a number written down anywhere. Per workload it reports replicas, the pod's request, per-container limits and the grace period, then the whole stack's total request, which is the figure that decides whether the install fits on the runner at all. It lives in .github/scripts/ci_values_summary.py because two jobs need it and the quantity arithmetic is worth having in one testable place. In the install job it runs before ct install, so the summary says what was attempted even when the install times out. Both calls are continue-on-error: a step that only describes the run must never fail it.
Reviewing a chart change meant rendering it locally to see what actually changed. The lint job now renders the chart six ways - chart defaults, the CI overlay, and each values-overlay example - and uploads them as rendered-manifests-helm-<version>, kept for 14 days. Reviewing a change becomes downloading that and the same artifact from main, and diffing the two. It lives in the lint job because rendering needs no cluster, so the artifact exists even for a change that skips the install matrix. The bundle is 232K. Secret values are redacted before upload. helm template generates a root password, the password_secret pepper and both MongoDB passwords for every value left empty, which is the default, and artifacts on a public repository are world-downloadable: those would read as leaked credentials to anyone who found them, and to a secret scanner. Keys are kept so a diff still shows one appearing or disappearing, and the placeholder is valid base64 so the manifests stay loadable. The step then proves the redaction worked and discards the bundle rather than uploading it if any value survived. A failed redaction must not become a published credential.
alix-graylog
force-pushed
the
wave-4-ci-hardening
branch
from
August 21, 2026 21:39
7b2c8a6 to
54b4351
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Short description of the change.
Details
Linked issues
This fixes #??
PR Checklist
Please check the items that apply to your change.
Testing Checklist
Static Validation
helm lint ./charts/grayloghelm template graylog ./charts/graylog --validateInstallation
helm install graylog ./charts/graylogkubectl rollout status statefulset/grayloghelm test graylogFunctional (if applicable)
Upgrade (if applicable)
Specific to this PR
Notes for reviewers