Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 20 additions & 11 deletions .github/workflows/notify-main-failure.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,24 @@
# every merge. This is derived from the GitHub API rather than cross-run
# state, so it is self-healing (no cache to go stale).
#
# `freeze-scoped: true` adds a third outcome for STAGING callers. A staging
# failure is only release-blocking once the freeze window is open; mid-week it is
# an ordinary integration-branch red, and paging the channel for it is what
# teaches people to ignore the channel. So a freeze-scoped failure posts the red
# alert while the `staging-freeze` ruleset is active, and outside the freeze
# posts a muted grey notice instead (`outside-freeze: silent` drops it entirely,
# at the cost of a broken staging branch being able to sit red unnoticed until
# the next freeze opens). Prod and freeze/unfreeze callers leave this alone — a
# failure there is always worth interrupting for.
# `freeze-scoped: true` scopes a STAGING caller to the release window. The policy
# this implements:
#
# failure on main -> alert
# failure on staging, freeze OPEN -> alert (staging is the release candidate)
# failure on staging, freeze NOT open -> nothing (ordinary integration-branch red)
#
# Mid-week staging is the integration branch, and paging the channel for it is
# what teaches people to ignore the channel. `outside-freeze: notice` posts a
# muted grey message instead of nothing, for a caller that wants the trail; the
# default is silence. Either way the run is still red in the Actions tab, and
# pipeline-watchdog.yml still covers a staging branch that stops building at all.
#
# Prod and freeze/unfreeze callers leave this alone: a failure there is always
# worth interrupting for. A workflow triggered on BOTH main and staging from one
# notify job cannot hardcode this — pass an expression instead, or a main failure
# gets silenced too:
# freeze-scoped: ${{ github.ref_name == 'staging' }}
#
# It posts its own message rather than the deploy-notification composite, whose
# copy is deploy-specific ("has failed deploying to ...") and reads wrong for
Expand Down Expand Up @@ -79,9 +88,9 @@ on:
type: boolean
default: false
outside-freeze:
description: "What a freeze-scoped failure does while staging is NOT frozen: 'notice' (muted, no panic styling) or 'silent' (post nothing)."
description: "What a freeze-scoped failure does while staging is NOT frozen: 'silent' (post nothing) or 'notice' (muted, no panic styling)."
type: string
default: notice
default: silent
runs-on:
description: "Runner label for the notify job"
type: string
Expand Down
16 changes: 14 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,13 @@ missing recovery message is the symptom to look for.

A staging failure is not the same event all week. Once the freeze window is open, `staging` is the release candidate and a red pipeline blocks the release. Before it, `staging` is the integration branch and the same red is routine, so paging the channel for it is what teaches people to scroll past the channel.

`freeze-scoped: true` on a **staging** caller keeps the red `:rotating_light:` alert while the `staging-freeze` ruleset is `active`, and drops it to a muted grey `:warning:` notice outside the window:
`freeze-scoped: true` on a **staging** caller implements that policy:

| Where it failed | Freeze window | Result |
|---|---|---|
| `main` | n/a | red alert |
| `staging` | open | red alert |
| `staging` | not open | nothing posted |

```yaml
with:
Expand All @@ -97,7 +103,13 @@ A staging failure is not the same event all week. Once the freeze window is open
freeze-scoped: true
```

Add `outside-freeze: silent` to post nothing at all outside the window instead. That trades the noise for a blind spot: a staging pipeline can then sit red unnoticed until the next freeze opens on a branch that no longer builds, so `notice` is the default.
Add `outside-freeze: notice` to post a muted grey `:warning:` message instead of nothing, for a caller that wants the trail. Silence is the default; the run is still red in the Actions tab either way, and `pipeline-watchdog.yml` still covers a staging branch that stops building at all.

A workflow triggered on **both** main and staging from one notify job must not hardcode `true`, or main failures get silenced too. Pass an expression:

```yaml
freeze-scoped: ${{ github.ref_name == 'staging' }}
```

Freeze state is read from the `staging-freeze` ruleset itself, not inferred from workflow history — a freeze that skipped itself because staging had nothing unreleased still concludes `success`, and history cannot tell that apart from a real freeze. Reading rulesets needs admin, so this reuses the same App that toggles them (`vars.RELEASE_APP_CLIENT_ID` + `secrets.RELEASE_APP_PRIVATE_KEY`); no extra `permissions:` on the caller. If the App token or the ruleset lookup fails, it escalates to the red alert rather than downgrading, so a lookup problem can never silence a real release-blocking failure.

Expand Down
Loading