Skip to content

fix(ci): keep GitHub's synthesized runs out of the deploy-branch sweeps (ENG-1549) - #54

Merged
lucas-koontz merged 1 commit into
mainfrom
fix/eng-1549-synthesized-runs-out-of-red-branch-sweep
Aug 24, 2026
Merged

fix(ci): keep GitHub's synthesized runs out of the deploy-branch sweeps (ENG-1549)#54
lucas-koontz merged 1 commit into
mainfrom
fix/eng-1549-synthesized-runs-out-of-red-branch-sweep

Conversation

@lucas-koontz

Copy link
Copy Markdown
Contributor

User story

As an engineer watching the engineering channel, I want a :rotating_light: from the pipeline watchdog to mean a deploy branch is actually not deployed, so that I keep reacting to it instead of learning to scroll past it.

Today it does not. On 2026-08-24 at 13:36 UTC four Dependabot version updates failed on mindsdb/anton, and the red-branch sweep reported one of them as a pipeline failure on main:

🚨 anton — 1 pipeline(s) still failing. The newest run is red and nobody has fixed it.
Workflow dynamic/dependabot/dependabot-updates · Branch main · Commit c2e4de11 · Last touched by dependabot[bot]

Every real workflow on anton/main was green at the time.

What was wrong

The red-branch sweep groups a branch's run history by workflow path and reports the newest red group. GitHub gives its own products a synthesized path under dynamic/ (dynamic/dependabot/dependabot-updates, dynamic/github-code-scanning/codeql, dynamic/pages/pages-build-deployment), attributes those runs to the default branch, and returns them from the same /actions/runs?branch=main listing. With the workflows: allowlist empty, which is the default all seven callers use, they were in scope and looked exactly like a pipeline.

A failed Dependabot version update is not a statement about whether main got deployed, and that is the only question either sweep asks.

flowchart LR
  A["GET /actions/runs?branch=main"] --> B["group by workflow path"]
  B --> C[".github/workflows/release.yml (green)"]
  B --> D[".github/workflows/docs.yml (green)"]
  B --> E["dynamic/dependabot/dependabot-updates (RED)"]
  E --> F["in_scope: only is empty, so yes"]
  F --> G["Slack: 'anton — 1 pipeline(s) still failing'"]
Loading

What changes

The wide default now means every workflow file, so the dynamic/ namespace is out of both sweeps. It stays a scope decision rather than a ban: an allowlist that names a synthesized path still reports it, so a repo that wants a red default-setup CodeQL scan surfaced can ask for it explicitly.

flowchart LR
  A["GET /actions/runs?branch=main"] --> B["drop the 'dynamic/' prefix"]
  B --> C["group by workflow path"]
  C --> D[".github/workflows/release.yml (green)"]
  C --> E[".github/workflows/docs.yml (green)"]
  B -. "named in workflows: input" .-> F["reported anyway"]
  D --> G["no finding, nothing posted"]
Loading
  • scripts/branch_health.py: SYNTHESIZED_PREFIX = "dynamic/", and in_scope() applies it only when only is empty, so an explicit allowlist still wins.
  • .github/workflows/notify-startup-failure.yml: the same filter on the startup sweep's jq, for consistency. A startup_failure on a Dependabot run is not "the branch is not deployed" either.
  • README.md: the scope section says what the wide default leaves out.

All seven watchdog callers (anton, auth, cowork, cowork-server, mindshub_frontend, mindshub_inference, mindshub_services) take the defaults, so no per-repo change is needed.

Acceptance criteria

  • A red dynamic/dependabot/dependabot-updates on a swept branch produces no finding.
  • The other two synthesized namespaces (github-code-scanning, pages) are out too.
  • Naming a synthesized path in workflows: still reports it.
  • A red .github/workflows/* pipeline is still reported, on main and on a frozen staging.
  • A real startup_failure is still reported by the first sweep.
  • No caller signature change; existing inputs and defaults keep their meaning.

How to test

Replay it against the live incident, which is what the numbers below are:

# BEFORE (this branch's parent): 1 finding, the Dependabot run
python3 scripts/branch_health.py --repo mindsdb/anton --branches main \
  --lookback-minutes 2000 --min-age-minutes 0 --frozen false \
  --self-path .github/workflows/pipeline-watchdog.yml --out out.json

# AFTER (this branch): 0 findings
# and a real red pipeline is still found:
python3 scripts/branch_health.py --repo mindsdb/mindshub_inference \
  --branches "main staging" --frozen true --lookback-minutes 2000 \
  --min-age-minutes 30 --self-path .github/workflows/pipeline-watchdog.yml --out inf.json

Or dispatch Pipeline watchdog on anton with dry-run: true once this is on main and read the job summary.

Reviewer notes

  • Why the dynamic/ prefix and not an allowlist of .github/workflows/. Requiring the .github/workflows/ prefix is the stronger invariant, and it would also catch a future synthesized namespace. It was not worth it here: the test fixtures use bare paths like publish-staging.yml, so that predicate meant rewriting the whole fixture set of a safety-critical watchdog to buy coverage of a namespace GitHub has not shipped yet. If GitHub adds one, the constant is one line.
  • The exclusion is checked before only, not after. exclude (the watchdog's own path) still beats an allowlist, which the existing test_exclusion_wins_over_an_allowlist pins. The new rule sits on the other side, so workflows: overrides it.
  • The startup-sweep filter is absolute. That sweep has no workflows: input, so there is no override there by design.

Verified locally

Check Result
uv run --with pyyaml --with pytest --python 3.12 -m pytest tests/ -q 137 passed (3 new)
New tests fail without the fix 2 of 3 fail on the parent commit, as intended
actionlint .github/workflows/notify-startup-failure.yml clean (v1.7.12)
Live sweep, mindsdb/anton main 1 finding before, 0 after
Live sweep, mindsdb/mindshub_inference frozen staging still reports .github/workflows/staging-build-deploy.yml
Patched jq, real startup_failure on a repo workflow still reported
Patched jq, startup_failure on a dynamic/ path dropped

Refs: ENG-1549

…ps (ENG-1549)

The red-branch sweep reports the newest red run per workflow PATH on a deploy
branch, and the wide `workflows:` default meant every path with a run. GitHub
gives its own products a synthesized path under `dynamic/` and attributes those
runs to the default branch, so `dynamic/dependabot/dependabot-updates` read as a
pipeline on `main`: anton's channel got ":rotating_light: anton — 1 pipeline(s)
still failing" for a failed Dependabot version update, which says nothing about
whether `main` is deployed.

The wide default now means every workflow FILE, dropping the `dynamic/`
namespace (Dependabot updates, default-setup code scanning, Pages builds) from
both sweeps. It is a scope decision, not a ban: an allowlist that names one
still reports it, so a repo that wants a red default-setup CodeQL scan can ask.

All seven watchdog callers take the defaults, so this covers them with no
per-repo change.

Lucas Koontz. ENG-1549: Alert when a staging release candidate fails to publish

Refs: ENG-1549
@lucas-koontz
lucas-koontz requested a review from a team as a code owner August 24, 2026 16:14
@lucas-koontz
lucas-koontz merged commit 44f0a6b into main Aug 24, 2026
5 checks passed
@lucas-koontz
lucas-koontz deleted the fix/eng-1549-synthesized-runs-out-of-red-branch-sweep branch August 24, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants