fix(ci): keep GitHub's synthesized runs out of the deploy-branch sweeps (ENG-1549) - #54
Merged
lucas-koontz merged 1 commit intoAug 24, 2026
Conversation
…ps (ENG-1549) The red-branch sweep reports the newest red run per workflow PATH on a deploy branch, and the wide `workflows:` default meant every path with a run. GitHub gives its own products a synthesized path under `dynamic/` and attributes those runs to the default branch, so `dynamic/dependabot/dependabot-updates` read as a pipeline on `main`: anton's channel got ":rotating_light: anton — 1 pipeline(s) still failing" for a failed Dependabot version update, which says nothing about whether `main` is deployed. The wide default now means every workflow FILE, dropping the `dynamic/` namespace (Dependabot updates, default-setup code scanning, Pages builds) from both sweeps. It is a scope decision, not a ban: an allowlist that names one still reports it, so a repo that wants a red default-setup CodeQL scan can ask. All seven watchdog callers take the defaults, so this covers them with no per-repo change. Lucas Koontz. ENG-1549: Alert when a staging release candidate fails to publish Refs: ENG-1549
mindsdb-devops
approved these changes
Aug 24, 2026
lucas-koontz
deleted the
fix/eng-1549-synthesized-runs-out-of-red-branch-sweep
branch
August 24, 2026 17:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User story
As an engineer watching the engineering channel, I want a
:rotating_light:from the pipeline watchdog to mean a deploy branch is actually not deployed, so that I keep reacting to it instead of learning to scroll past it.Today it does not. On 2026-08-24 at 13:36 UTC four Dependabot version updates failed on
mindsdb/anton, and the red-branch sweep reported one of them as a pipeline failure onmain:Every real workflow on
anton/mainwas green at the time.What was wrong
The red-branch sweep groups a branch's run history by workflow
pathand reports the newest red group. GitHub gives its own products a synthesized path underdynamic/(dynamic/dependabot/dependabot-updates,dynamic/github-code-scanning/codeql,dynamic/pages/pages-build-deployment), attributes those runs to the default branch, and returns them from the same/actions/runs?branch=mainlisting. With theworkflows:allowlist empty, which is the default all seven callers use, they were in scope and looked exactly like a pipeline.A failed Dependabot version update is not a statement about whether
maingot deployed, and that is the only question either sweep asks.What changes
The wide default now means every workflow file, so the
dynamic/namespace is out of both sweeps. It stays a scope decision rather than a ban: an allowlist that names a synthesized path still reports it, so a repo that wants a red default-setup CodeQL scan surfaced can ask for it explicitly.scripts/branch_health.py:SYNTHESIZED_PREFIX = "dynamic/", andin_scope()applies it only whenonlyis empty, so an explicit allowlist still wins..github/workflows/notify-startup-failure.yml: the same filter on the startup sweep'sjq, for consistency. Astartup_failureon a Dependabot run is not "the branch is not deployed" either.README.md: the scope section says what the wide default leaves out.All seven watchdog callers (
anton,auth,cowork,cowork-server,mindshub_frontend,mindshub_inference,mindshub_services) take the defaults, so no per-repo change is needed.Acceptance criteria
dynamic/dependabot/dependabot-updateson a swept branch produces no finding.github-code-scanning,pages) are out too.workflows:still reports it..github/workflows/*pipeline is still reported, onmainand on a frozenstaging.startup_failureis still reported by the first sweep.How to test
Replay it against the live incident, which is what the numbers below are:
Or dispatch
Pipeline watchdogonantonwithdry-run: trueonce this is onmainand read the job summary.Reviewer notes
dynamic/prefix and not an allowlist of.github/workflows/. Requiring the.github/workflows/prefix is the stronger invariant, and it would also catch a future synthesized namespace. It was not worth it here: the test fixtures use bare paths likepublish-staging.yml, so that predicate meant rewriting the whole fixture set of a safety-critical watchdog to buy coverage of a namespace GitHub has not shipped yet. If GitHub adds one, the constant is one line.only, not after.exclude(the watchdog's own path) still beats an allowlist, which the existingtest_exclusion_wins_over_an_allowlistpins. The new rule sits on the other side, soworkflows:overrides it.workflows:input, so there is no override there by design.Verified locally
uv run --with pyyaml --with pytest --python 3.12 -m pytest tests/ -qactionlint .github/workflows/notify-startup-failure.ymlmindsdb/antonmainmindsdb/mindshub_inferencefrozenstaging.github/workflows/staging-build-deploy.ymljq, realstartup_failureon a repo workflowjq,startup_failureon adynamic/pathRefs: ENG-1549