fix(ci): report a staging failure and its fix consistently (ENG-1549)

[lucas-koontz]

User story

As an engineer who merges a fix to staging on a Thursday afternoon
I want to be told when the pipeline that publishes it fails, and when it is fixed
So that I find out from the channel rather than by reading the Actions run list three hours later

Why this matters

On 13 August the staging desktop build spent about three and a half hours pointing at a release candidate that failed every chat turn, with the fix already merged. The fix's own publish run failed, so rc8 never published and the broken rc7 stayed newest. Nobody was told: staging alerting is deliberately silent outside the release-freeze window, which is correct. What was not correct is that the re-run then posted a green Recovered to the channel. The only thing anyone ever heard about a stalled rc stream was the good news.

The same shape, in the other direction, hit cowork the following day: the main → staging sync failed and alerted, the re-run succeeded, and the notify job was skipped, leaving a red alert standing for a sync that had already recovered.

Both are one problem. An alert channel that reports half a story teaches people to ignore it.

What happens today

flowchart TD
    F["staging pipeline fails<br/>outside the freeze window"] --> S["notify job: freeze-scoped<br/>SILENT, correct"]
    R["re-run succeeds"] --> G["notify job: status recovered<br/>freeze check SKIPPED"]
    G --> P["posts green Recovered"]
    W["release-train wrapper<br/>if: failure()"] --> WF["fails: posts red"]
    W --> WR["re-run succeeds: job SKIPPED"]
    WR --> N["nothing posted, red alert stands"]
    Note["Recovery is announced for a failure<br/>the channel was never told about"]
    P -.-> Note

Loading

What should happen

flowchart TD
    F["staging pipeline fails<br/>outside the freeze window"] --> S["SILENT"]
    R["re-run succeeds"] --> S2["SILENT, same veto"]
    W["release-train reusable<br/>ends with the notify step"] --> WF["fails: posts red"]
    W --> WR["re-run succeeds: posts Recovered"]
    O["freeze window OPENS on a red staging"] --> SW["red-branch sweep posts"]
    Note["Both halves share one decision,<br/>and the freeze itself is the trigger"]
    S2 -.-> Note

Loading

Acceptance criteria

• A freeze-scoped staging caller posts nothing outside the freeze window, for failed and recovered alike
• A freeze-scoped staging caller posts both a failure and its recovery while the window is open
• A failure on main posts regardless of freeze state, and so does its recovery
• A release-train workflow that fails, and is then re-run successfully, posts both messages
• The freeze window opening on an already-red staging produces an alert naming the workflow and commit
• A green run whose previous run was also green still posts nothing
• A startup_failure produces exactly one alert, not one per sweep
• Renaming a repo's freeze ruleset changes one input, not four files
• A PR-close cleanup workflow beside a PR pipeline is not reported as two run trees

How to test

1. From this repo's Actions tab, dispatch Notify main-branch failure with status: failed. A red message posts. Dispatch it with status: recovered. A green one posts. Both skip the freeze check on a dispatch, which is the smoke test.
2. uv run --with pyyaml --with pytest --python 3.12 -m pytest tests/ -q — 134 pass.
3. ./actionlint -color — clean.
4. uv run --with pyyaml --python 3.12 scripts/workflow_graph.py --workflow-dir .github/workflows --allow-external-reusables — one run tree per event.
5. After merge, dispatch Pipeline watchdog on cowork-server with min-age-minutes: 0 and lookback-minutes: 4320. The summary table lists any deploy branch sitting red; staging rows appear only while frozen.
6. Re-run a single failed job (the per-job button, not "Re-run failed jobs") on any pipeline and confirm no notify fires, then confirm the watchdog reports it 30 minutes later. This is the gap the sweep exists for.

Notes for the reviewer

The recovery bug is verified against two real runs, not inferred. cowork-server run 31749965274: attempt 1 failed and Notify Slack was skipped; attempt 2 recovered and posted. mindshub_inference run 31793683185 did the same at 11:38 UTC on 14 August, before that day's 13:47 freeze.

The outside-freeze: notice mode is deleted rather than defaulted off. No caller set it, and a grey middle setting is the thing that would reintroduce half-a-story alerts.

The four release-train reusables declare no permissions: block at all, so they inherit the caller's grant. Corrected in 569b6e4 after self-review: they previously declared contents: read, and a called workflow's own block is the ceiling for its token as well as the caller's, so that capped it and the recovery lookup was refused however much the caller granted. The actions: read grant all seven consumer PRs add was therefore inert, and the headline fix of this ticket could not fire. Declaring actions: read instead was not available either: sync-main-to-staging.yml floats @main and its live caller grants no permissions: block, so that would startup_failure every consumer's push: main the moment this merged. Inheriting gives the grant where a caller has it and a quiet refused lookup where it does not. The cost is that workflow_graph.py check 2 compares what a callee declares against its callers' grants, so these four are invisible to it; each file says so.

github.job_workflow_sha is not real, and the scoped ignore was muting a true positive. Corrected in 569b6e4 after self-review. It is an OIDC token claim, confirmed a documentation bug in actions/runner#2417 ("only supported as an OIDC claim"), and the sentence I quoted as documentation is from the OIDC page rather than the contexts reference. It evaluated to empty, so all six checkouts silently took the default branch: a consumer pinning release-freeze.yml@<sha> still ran main's freeze_state.py, the exact failure the comment claimed to prevent. The tell was that actionlint already carried the newer github.workflow_ref and github.workflow_sha; it was not behind, the property was not real. Now job.workflow_sha, which GitHub added for this in April 2026, with the ignore retargeted (verified load-bearing: 12 findings without it).

The composite mints no App token. continue-on-error is not available to composite steps, and a repo without the release-train App must degrade to "assume frozen" rather than fail the notify job, so the caller mints it and passes it in.

The red sweep excludes startup_failure on purpose, leaving it to the existing sweep, so one incident never produces two differently-worded alerts.

The rolling release PR opens as a draft. An always-open staging → main PR is otherwise a merge button next to production all week. It only ever moves draft to ready, never back, so it cannot fight a human who marked it ready.

The lint fix is a real false positive, not a convenience. pull_request: types: [closed] and a pipeline on opened/synchronize never co-fire. That was mindshub_services' only finding, and the sole way to silence it was a run-tree-ok comment, which is how a genuine finding on the same file later gets waved through.

Verified locally

Check	Result
pytest tests/ -q	134 passed (was 61; 73 new across five files)
actionlint -color	clean
workflow_graph.py self-check	one run tree per event, every local call composes
Consumer lint, 5 repos	clean
Consumer lint, mindshub_services	1 finding before, 0 after
Consumer lint, anton	3 findings before, 2 after; both pre-existing and untouched here
Slack post path	not exercised locally; needs the dispatch in step 1 after merge

Ships with

Seven consumer PRs that drop their notify jobs and grant actions: read: cowork#658, cowork-server#318, anton#351, auth#278, mindshub_inference#417, mindshub_services#188, mindshub_frontend#1422.

Merge this one first. mindshub_services#188 requires it: its newly wired workflow-lint job pulls scripts/workflow_graph.py from this repo's main, so the disjoint-types fix has to be there before that gate can pass. And the seven consumer PRs drop their notify jobs, so landing them first would leave freeze, unfreeze and release-PR with no alerting at all, plus an actions: read grant pointing at reusables that still capped it away.

Note the real rollout window, which the first version of this section got wrong. All four consumer wrappers execute only from each repo's DEFAULT branch (schedule, workflow_run, pull_request into main, push: main), and the seven consumer PRs target staging. So they change nothing when they merge; they take effect at the next staging → main release, not "when each consumer PR lands". Two consequences worth knowing before merging:

• sync-main-to-staging.yml already floats @main, so a failed sync double-posts (the reusable's new notify step plus the wrapper's surviving if: failure() job) for that whole window, not "briefly".
• pipeline-watchdog.yml is a second @main overlap this section originally missed. The new red-branch sweep defaults on, and none of the seven watchdog wrappers pass the input, so it goes live in all seven repos on the next 30-minute cron with no consumer PR opting in. It now excludes its own workflow and takes an optional workflows: allowlist, but the default is still on.

The two Definition-of-Done items that observe the rolling release PR ("one real staging merge", "one freeze flipping it to ready") are therefore post-release checks, not tickable at merge.

---

Self-review pass (2026-08-14)

Reviewed my own diff before asking anyone else to. Six commits on top of a1b8e58; the two that matter are blockers I introduced and had not noticed:

• 569b6e4 — the four reusables capped their own token to contents: read, so the recovery lookup was refused always and this ticket's headline fix could not fire. And github.job_workflow_sha does not exist, so every "pin" resolved to main. Both explained in the reviewer notes above.
• 26e1917 — a manual run of any of the four wrappers posted a green Recovered for a failure that never happened, because the composite inferred "this is the smoke test" from github.event_name == 'workflow_dispatch' and a called workflow inherits the caller's event. Now an explicit force-post input. The decision moved into scripts/notify_decision.py with 27 tests, since none of it was reachable by a test while it lived in YAML, which is how this got past me.

The rest: ca162b2 the sweep read created_at (pinned to attempt 1, so a failing re-run was invisible) and reported every workflow on the branch including itself; 4438eb7 and 6e0cd07 are the two Copilot findings; 7c525ca corrects the docs the above invalidated.

Verified: 134 tests, actionlint clean, workflow_graph.py clean here and on all seven consumers, and every one of the six commits independently green so a bisect never lands on a broken tree.

[Copilot]

Pull request overview

This PR makes CI alerting report staging/main failures and recoveries consistently by centralizing notification logic, formalizing the release-freeze contract, and enhancing workflow-lint run-tree detection to avoid common false positives.

Changes:

• Introduces a shared notify-pipeline-status composite action and updates reusable workflows to post both failure and recovery (with freeze-scoped silence applied symmetrically).
• Adds scripts/freeze_state.py (single source of truth for ruleset-based freeze state) plus scripts/branch_health.py (red-branch sweep) with unit tests.
• Refines scripts/workflow_graph.py event-keying and collision reporting to account for PR types and deduplicate overlap findings.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_workflow_graph.py	Updates/expands tests to validate new (event, branch, type) keying and collision deduping.
tests/test_freeze_state.py	Adds unit tests for the freeze ruleset read/write contract and failure-mode asymmetry.
tests/test_branch_health.py	Adds unit tests for selecting “still red” branches and freeze-opening behavior.
scripts/workflow_graph.py	Keys triggers by branch and PR type; deduplicates collision findings and improves error messaging.
scripts/freeze_state.py	Centralizes freeze ruleset lookup and enforcement flipping for reuse across workflows.
scripts/branch_health.py	Implements red-branch sweep selection + API access helpers with bounded-repeat behavior.
README.md	Documents the new alerting model, rolling release PR behavior, and the freeze-state contract.
notify-pipeline-status/action.yml	Adds the shared composite action for Slack notifications with freeze scoping + recovery detection.
.github/workflows/sync-main-to-staging.yml	Switches to using the shared notify composite at workflow end and checks out shared logic at the reusable SHA.
.github/workflows/release-unfreeze.yml	Uses freeze_state.py for ruleset updates and adds shared notify composite at workflow end.
.github/workflows/release-pr.yml	Keeps a rolling release PR updated; reads freeze state to decide draft vs ready; adds notify composite.
.github/workflows/release-freeze.yml	Uses freeze_state.py to enforce the freeze ruleset and posts via the shared notify composite.
.github/workflows/notify-startup-failure.yml	Expands into a deploy-branch health watchdog with separate startup-failure + red-branch sweeps.
.github/workflows/notify-main-failure.yml	Becomes a thin wrapper around the shared composite (still supports dispatch smoke tests).
.github/actionlint.yaml	Scopes an actionlint ignore for github.job_workflow_sha to prevent false positives.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lucas-koontz]

Both Copilot threads are fixed and resolved. Repeating it here because pushing the fixes moved the lines they were anchored to, so GitHub marked them outdated and folded them away, and a reply inside a folded thread is invisible on this page.

• scripts/freeze_state.py unguarded json.loads → 4438eb7. A non-JSON body now raises LookupError_ and prints the "provisioning has drifted" annotation instead of a traceback.
• tests/test_workflow_graph.py name/docstring mismatch → 6e0cd07. It pinned nothing, because its input declared no branches: at all.

The two blockers I found in my own diff are in the reviewer notes above, and 569b6e4 is the one to read first: the recovery half of this ticket could not fire at all.