|
| 1 | +// SPDX-License-Identifier: CC-BY-SA-4.0 |
| 2 | +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk> |
| 3 | += CI/CD for panic-attack |
| 4 | +:toc: left |
| 5 | +:toclevels: 3 |
| 6 | +:sectnums: |
| 7 | + |
| 8 | +== The failure this replaces |
| 9 | + |
| 10 | +On 2026-10-25 this repository's CI went dark for four hours. Twelve |
| 11 | +`pull_request` workflows concluded `startup_failure`: **zero jobs, no log, no |
| 12 | +annotation, no check run**. Nothing in the GitHub UI said why, and the only |
| 13 | +diagnosis available was to read the estate's settings canon and compare it with |
| 14 | +our own workflow files. |
| 15 | + |
| 16 | +Two causes, both policy-level, neither visible from the code: |
| 17 | + |
| 18 | +`uses:` a reusable workflow from `metadatastician/standards`:: |
| 19 | +The canonical allowlist - |
| 20 | +`rhodium-standard-repositories/actions-allowlist/allowed-actions.json` in |
| 21 | +`hyperpolymath/standards` - permits `hyperpolymath/*`. It does not permit |
| 22 | +`metadatastician/*`. `hyperpolymath/standards` and `metadatastician/standards` |
| 23 | +are the *same repository* (a rename; the lockfile records one `repo_id`), so the |
| 24 | +correct spelling is the one the allowlist covers. Six thin wrappers had been |
| 25 | +re-pointed at the new owner, and every one of them died before its first job. |
| 26 | + |
| 27 | +tag-pinned `uses:`, e.g. `actions/checkout@v7.0.1`:: |
| 28 | +The estate posture is `sha_pinning_required = true`. Under it, a tag-pinned |
| 29 | +`uses:` is refused **at startup** - the same opaque death. The pin is a security |
| 30 | +control, so the fix is to pin properly, not to turn the control off. |
| 31 | + |
| 32 | +The lesson generalises and is the reason this document exists: *a run that never |
| 33 | +starts is invisible to every gate that watches runs*. It produces no check run, |
| 34 | +so a required context silently stops arriving and the branch looks |
| 35 | +**healthier** than a broken one. Both causes are now gated before they can bite |
| 36 | +(see <<integrity>>). |
| 37 | + |
| 38 | +== The shape of the pipeline |
| 39 | + |
| 40 | +[cols="2,3,2",options="header"] |
| 41 | +|=== |
| 42 | +| Workflow | What it answers | When |
| 43 | + |
| 44 | +| `verify.yml` |
| 45 | +| Does it build, lint, test, document and self-scan honestly? The floor that |
| 46 | + cannot be taken out by another repository. |
| 47 | +| every PR, every push |
| 48 | + |
| 49 | +| `ci-integrity.yml` |
| 50 | +| Can the CI itself start? SHA pins, lockfile coverage, allowlist coverage, |
| 51 | + YAML parseability. |
| 52 | +| every PR, every push, Mondays |
| 53 | + |
| 54 | +| `proven-tests.yml` |
| 55 | +| Is the suite's claimed coverage derived from tests that ran, and are its tiers |
| 56 | + honest? Plus the framework's own Idris2 suite. |
| 57 | +| ledger: every PR. framework: Mondays/dispatch |
| 58 | + |
| 59 | +| `squabble-gate.yml` |
| 60 | +| Is any required context missing, and is any CI upstream dead? Runs |
| 61 | + `cicd-squabbler`. |
| 62 | +| Wednesdays, dispatch, PRs labelled `ci-squabble` |
| 63 | + |
| 64 | +| `recon.yml` |
| 65 | +| What do two independent analysers say about this repository, and has any |
| 66 | + category regressed? |
| 67 | +| Tuesdays, dispatch |
| 68 | + |
| 69 | +| `rust-ci.yml`, `governance.yml`, `hypatia-scan.yml`, `secret-scanner.yml`, |
| 70 | + `scorecard.yml`, `mirror.yml` |
| 71 | +| The estate's richer reusable gates (audit, coverage upload, Hypatia SARIF, |
| 72 | + governance, secret scanning). |
| 73 | +| per estate canon |
| 74 | + |
| 75 | +| `e2e.yml`, `codecov`, `cargo-audit`, `chapel-ci`, `dogfood-gate`, |
| 76 | + `bridge-gate`, `pages`, `release` |
| 77 | +| Behavioural, supply-chain, contract and release checks. |
| 78 | +| per file header |
| 79 | +|=== |
| 80 | + |
| 81 | +=== Why `verify.yml` exists at all |
| 82 | + |
| 83 | +Every estate gate except `verify` depends on `hyperpolymath/standards` being |
| 84 | +reachable *and* permitted at that instant. That is the right long-term shape - |
| 85 | +one place to configure, propagate everywhere - but it means a settings change in |
| 86 | +another repository can remove this repository's entire build signal. `verify` |
| 87 | +depends on nothing but actions this repository pins itself, so the "does it |
| 88 | +compile and pass" answer survives any policy change elsewhere. |
| 89 | + |
| 90 | +<<integrity>> === CI integrity |
| 91 | + |
| 92 | +`ci-integrity.yml` runs four gates, each of which corresponds to a way a |
| 93 | +workflow dies before its first job: |
| 94 | + |
| 95 | +SHA pins:: `.githooks/validate-sha-pins.sh` (canon). Every third-party `uses:` |
| 96 | +names a 40-hex commit plus a `# <version>` comment. |
| 97 | +Lockfile coverage:: `.githooks/validate-actions-lock.sh` (canon), plus |
| 98 | +`scripts/ci/pin-workflow-uses.py --check`, which derives the lockfile from the |
| 99 | +workflows. A caller pinned to a commit whose lockfile has drifted dies with |
| 100 | +`failure`, 0 jobs and *no reason in the API* - which is why this is a gate and |
| 101 | +not a courtesy. |
| 102 | +Allowlist coverage:: `scripts/ci/check-allowed-actions.sh` (canon) against the |
| 103 | +vendored `.github/actions-allowlist.json`. No API, no admin: it lists exactly |
| 104 | +which `uses:` the estate policy does not cover. |
| 105 | +YAML parseability:: every workflow parses *and* declares a `jobs:` map. A file |
| 106 | +whose quoting swallowed the document parses to a scalar and never runs. |
| 107 | + |
| 108 | +Maintenance: after changing any `uses:`, run |
| 109 | + |
| 110 | +[source,sh] |
| 111 | +---- |
| 112 | +bash scripts/ci/resolve-action-shas.sh # network: resolves tag -> commit |
| 113 | +python3 scripts/ci/pin-workflow-uses.py --map .github/workflows/action-pins.json |
| 114 | +---- |
| 115 | + |
| 116 | +`action-pins.json` is committed on purpose: it records which tag a pin stood for |
| 117 | +on the day it was taken, so a reviewer can tell an intentional bump from a |
| 118 | +silent re-point. |
| 119 | + |
| 120 | +== Testing: evidence-graded, not declared |
| 121 | + |
| 122 | +`proven/ledger.json` grades this repository's suite on |
| 123 | +`proven-tests-and-benches`' 17-category x 14-aspect lattice. The rule the |
| 124 | +framework exists to enforce is that **coverage is derived from executed tests, |
| 125 | +never declared**, so: |
| 126 | + |
| 127 | +* `scripts/proven/inventory.sh` asks the test runner what tests exist, per |
| 128 | + binary; |
| 129 | +* `scripts/proven/verify-ledger.py` fails when a cell selects no executed test, |
| 130 | + when an executed test is claimed by nothing, or when a stronger tier is |
| 131 | + claimed than the evidence supports; |
| 132 | +* the resulting report is fed to `panic-attack aggregate --proven-tests |
| 133 | + --fail-on-defect`, which re-checks the report's internal consistency and fails |
| 134 | + on an unsupported claim (an Actually-Proven entry with no ladder, a cited proof |
| 135 | + file that is not there, a summary that disagrees with the entries). |
| 136 | + |
| 137 | +Tiers are used honestly: this repository has no proof-carrying Rust tests, so |
| 138 | +almost everything is `Provisionally-Proven` and says so. Five cells are declared |
| 139 | +**absent** with the command that would test them. A declared absence is a to-do |
| 140 | +list, not a silencer - `MutationTest` is absent because no mutation campaign |
| 141 | +runs here, which is the same `PA025 MutationGap` this tool reports about other |
| 142 | +repositories. |
| 143 | + |
| 144 | +== Recon: two analysers, one baseline |
| 145 | + |
| 146 | +`recon.yml` scans this repository with `panic-attack assail` and with |
| 147 | +`pons-asinorum` (pinned). They answer different questions - "what here is weak?" |
| 148 | +versus "what here is wasted, self-contradictory, or missing an escape hatch?" - |
| 149 | +and a finding that appears in both is corroborated evidence while a finding in |
| 150 | +exactly one is a prompt to look. |
| 151 | + |
| 152 | +Findings do **not** fail the job. This is the scanner: a scan of its own source |
| 153 | +will report what it is built to report, and failing on that creates pressure to |
| 154 | +suppress. Instead `recon/baseline.json` freezes the standing counts; a *new* |
| 155 | +category or a count *above* baseline fails; a count *below* is reported as debt |
| 156 | +paid and should be tightened in the same PR. See |
| 157 | +`docs/RECON-AND-EVIDENCE.adoc`. |
| 158 | + |
| 159 | +== Squabbling: fixing stuck gates without bypassing them |
| 160 | + |
| 161 | +`cicd-squabbler` (pinned to a commit) fetches this PR's live gate and reports |
| 162 | +which required contexts are missing or stuck, and maps the cross-repo CI chains |
| 163 | +this repository depends on. Its invariant is machine-checked: *green if and only |
| 164 | +if a required check actually ran and passed*, so an admin override or a dropped |
| 165 | +required context are provably not paths to green. It cannot fix code, and does |
| 166 | +not try. |
| 167 | + |
| 168 | +Exit codes are part of the contract and are never swallowed: |
| 169 | +`0` ok, `2` tool failure (**fails the job**), `3` no required-status-check rule |
| 170 | +on the base branch (reported), `4` a blocking chain finding - a cycle or a dead |
| 171 | +upstream (reported; fails when `strict` is requested), `5` agent items remain. |
| 172 | + |
| 173 | +== What is deliberately not here |
| 174 | + |
| 175 | +* **No `paths:` filter on a check intended to be required.** A required check |
| 176 | + whose workflow filters itself off does not report as skipped - it does not |
| 177 | + report at all, and the PR waits forever on a context that cannot arrive. |
| 178 | +* **No `continue-on-error` on a gate that can disagree with nobody.** An |
| 179 | + advisory job that is always green carries no information; where a check is |
| 180 | + genuinely advisory it says so in its name and its summary. |
| 181 | +* **No suppression of a finding to make a gate green.** Where debt exists it is |
| 182 | + frozen in a baseline, in a reviewable file, with the reason. |
0 commit comments