ci: unstarve the post-merge gates and alert when one goes dark (#7856) - #7860
Merged
Conversation
added 2 commits
August 11, 2026 17:22
Ten heavy gates ran on `push: branches: [main]`. Fourteen workflows fired on every merge, ~29 jobs each time, against a repo that runs ~9 jobs concurrently. At 58 merges/day demand outran drain, the queue grew without bound, and the `main` runs -- the ones that gate nothing and that nobody watches -- aged out. `gc-ratchet` had 22 of its last 25 `main` runs queued (oldest 9h18m) and had not succeeded on `main` since 2026-08-09, across five collector-touching merges. The constraint is NOT macOS capacity, which was the natural reading. At the moment of measurement the queue held 45 `ubuntu-latest` jobs against 14 `macos-14`, and `zizmor` (ubuntu) was queued in the same second as `gc-ratchet` (macOS). Rebalancing pools cannot help; only cutting total demand can. Nor is it the `concurrency:` blocks, which are already correct and twice-repaired (#7205) -- they are left untouched. Post-merge arm becomes a staggered six-hourly sweep plus release tags. That removes 19 jobs from every merge: ~1,100 job-starts/day replaced by ~76, a 93% cut on this slice. Pull-request arms are unchanged, so every PR is still measured before it can merge, and no probe, threshold or baseline moved. `gate-freshness.yml` + `scripts/check_gate_freshness.py` close the hole that let this last two days: an empty result set is indistinguishable from a healthy one nobody checked. It fails when a gate has no successful post-merge `main` run inside its budget, and maintains one self-closing sticky issue. The checker is sabotage-tested, not merely exercised -- `--self-test` plants a stale gate, a gate with no successful run, and a gate whose only recent successes are `pull_request` runs, which is the exact shape that made `gc-root-dominance` look healthy while its `main` arm was dark. Its first live run already found two things outside the issue's scope: `llvm-inprocess` has been dark for 171.8h, and `security-audit` -- a required context, left on every merge -- is itself at 17.4h.
📝 WalkthroughWalkthroughPost-merge CI gates now run on staggered six-hour schedules instead of every ChangesCI gate reliability
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant GateWorkflow
participant check_gate_freshness.py
participant ActionsAPI
participant StickyIssue
Scheduler->>GateWorkflow: Start scheduled post-merge gate run
GateWorkflow-->>Scheduler: Complete workflow run
Scheduler->>check_gate_freshness.py: Start freshness check
check_gate_freshness.py->>ActionsAPI: Query successful post-merge runs
ActionsAPI-->>check_gate_freshness.py: Return latest run and commit
check_gate_freshness.py->>StickyIssue: Create, update, or close stale-gate issue
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/check_gate_freshness.py`:
- Around line 143-151: Update the check_gate_freshness flow around
newest_post_merge_success and Verdict creation to distinguish API or workflow
lookup failures from a genuine stale result. Represent collection errors as an
unknown/error outcome, make the monitor fail when collection is incomplete, and
skip sticky-issue synchronization in that case; only publish “NO successful
post-merge run” and close an existing stale issue after a complete successful
collection.
- Around line 131-135: Update the workflow-run selection logic in the fetch path
to return each completed successful run’s updated_at value instead of
created_at, while preserving the existing head_sha and no-match behavior. Add a
self-test covering a run with an old created_at and recent updated_at, asserting
freshness is based on updated_at.
In `@scripts/gate_freshness.json`:
- Around line 3-4: Correct the freshness-check cadence wording from hourly to
every two hours in scripts/gate_freshness.json lines 3-4 and
docs/src/testing/ci-gate-scheduling.md lines 155-160; update both references
without changing the workflow configuration.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 34bd80e0-c15a-450f-bbe7-914b84f67ab9
📒 Files selected for processing (16)
.github/workflows/auto-opt-app-patterns.yml.github/workflows/eh-transport.yml.github/workflows/gate-freshness.yml.github/workflows/gc-moving-witnesses.yml.github/workflows/gc-native-roots.yml.github/workflows/gc-parse-churn-gate.yml.github/workflows/gc-ptr-shape-off-witness.yml.github/workflows/gc-ratchet.yml.github/workflows/gc-root-dominance.yml.github/workflows/llvm-inprocess.yml.github/workflows/tls-budget.ymlchangelog.d/7860-ci-gate-starvation.mddocs/src/SUMMARY.mddocs/src/testing/ci-gate-scheduling.mdscripts/check_gate_freshness.pyscripts/gate_freshness.json
Comment on lines
+131
to
+135
| payload = fetch(path) | ||
| for run in payload.get("workflow_runs", []): | ||
| if run.get("event") in POST_MERGE_EVENTS: | ||
| return run["created_at"], run.get("head_sha", "") | ||
| return None |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
: "${GITHUB_REPOSITORY:?Set GITHUB_REPOSITORY to owner/repo}"
gh api \
"repos/${GITHUB_REPOSITORY}/actions/workflows/gc-ratchet.yml/runs?branch=main&status=success&per_page=20" \
--jq '.workflow_runs[]
| select(.event == "push" or .event == "schedule" or .event == "workflow_dispatch")
| {event, status, conclusion, created_at, run_started_at, updated_at}'Repository: PerryTS/perry
Length of output: 220
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- candidate files ---'
fd -i 'check_gate_freshness.py|test.*fresh|freshness' . || true
printf '%s\n' '--- relevant symbols and fields ---'
rg -n -C 4 'newest_post_merge_success|created_at|updated_at|evaluate|sync_issue|POST_MERGE_EVENTS|workflow_runs|stale' . \
-g '*.py' -g '*.yml' -g '*.yaml' -g 'README*' || true
printf '%s\n' '--- tracked file status ---'
git status --short
git ls-files | rg '(^|/)(scripts/check_gate_freshness\.py|.*test.*fresh.*|.*fresh.*test.*)$' || trueRepository: PerryTS/perry
Length of output: 50370
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file='scripts/check_gate_freshness.py'
printf '%s\n' '--- file outline ---'
ast-grep outline "$file" 2>/dev/null || true
printf '%s\n' '--- target implementation ---'
cat -n "$file" | sed -n '1,230p'
printf '%s\n' '--- focused tests and callers ---'
rg -n -C 6 'check_gate_freshness|newest_post_merge_success|POST_MERGE_EVENTS|workflow_runs|created_at|updated_at|evaluate\(|sync_issue' \
"$file" tests .github scripts -g '*.py' -g '*.yml' -g '*.yaml' 2>/dev/null | head -n 500Repository: PerryTS/perry
Length of output: 48797
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- self-test and verdict construction ---'
cat -n scripts/check_gate_freshness.py | sed -n '266,345p'
printf '%s\n' '--- workflow-run field documentation references in repository ---'
rg -n -C 3 'run_started_at|updated_at|created_at' \
.github scripts docs tests \
-g '*.py' -g '*.yml' -g '*.yaml' -g '*.md' 2>/dev/null | head -n 250
printf '%s\n' '--- standalone behavioral probe ---'
python3 - <<'PY'
from datetime import datetime, timezone
from pathlib import Path
# Model the exact current selection and age calculation without importing or
# executing repository code.
source = Path("scripts/check_gate_freshness.py").read_text()
assert 'return run["created_at"], run.get("head_sha", "")' in source
assert '(now - _parse_ts(created)).total_seconds() / 3600.0' in source
now = datetime(2026, 8, 1, 12, 0, tzinfo=timezone.utc)
run = {
"event": "schedule",
"created_at": "2026-08-01T00:00:00Z",
"run_started_at": "2026-08-01T11:55:00Z",
"updated_at": "2026-08-01T11:59:00Z",
"status": "completed",
"conclusion": "success",
}
created_age = (now - datetime.fromisoformat(run["created_at"].replace("Z", "+00:00"))).total_seconds() / 3600
updated_age = (now - datetime.fromisoformat(run["updated_at"].replace("Z", "+00:00"))).total_seconds() / 3600
print({"created_age_hours": created_age, "updated_age_hours": updated_age,
"created_based_stale_for_12h_budget": created_age > 12,
"updated_based_stale_for_12h_budget": updated_age > 12})
assert created_age == 12
assert updated_age == 1
assert created_age > 12 or created_age == 12
assert updated_age <= 12
PYRepository: PerryTS/perry
Length of output: 8101
🌐 Web query:
GitHub REST API list workflow runs created_at updated_at workflow run response fields official documentation
💡 Result:
The GitHub REST API workflow run object, returned by endpoints such as "List workflow runs for a repository" and "Get a workflow run," includes both created_at and updated_at fields in its response [1]. These fields are formatted as strings representing date-times in ISO 8601 format (e.g., 2020-01-22T19:33:08Z) [1]. Response details: - created_at: The timestamp indicating when the workflow run was created [1]. - updated_at: The timestamp indicating the last time the workflow run was updated [1]. These fields are standard components of the workflow run object, which also contains other relevant metadata such as run_id, run_number, status, conclusion, run_started_at, and html_url [1]. You can access these endpoints using paths like GET /repos/{owner}/{repo}/actions/runs or GET /repos/{owner}/{repo}/actions/runs/{run_id} [2].
Citations:
- 1: https://help.github.com/en/enterprise-server@3.10/rest/actions/workflow-runs
- 2: https://www.getorchestra.io/guides/github-actions-api-get-a-workflow-run
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from datetime import datetime, timezone
now = datetime(2026, 8, 1, 12, 0, tzinfo=timezone.utc)
run = {
"event": "schedule",
"created_at": "2026-07-31T22:59:00Z", # 13h 1m old
"run_started_at": "2026-08-01T11:55:00Z",
"updated_at": "2026-08-01T11:59:00Z", # completed shortly before the check
"status": "completed",
"conclusion": "success",
}
parse = lambda key: datetime.fromisoformat(run[key].replace("Z", "+00:00"))
ages = {key: (now - parse(key)).total_seconds() / 3600 for key in ("created_at", "updated_at")}
print(ages)
print({
"current_created_at_logic_stale_for_12h_budget": ages["created_at"] > 12,
"updated_at_logic_stale_for_12h_budget": ages["updated_at"] > 12,
})
assert ages["created_at"] > 12
assert ages["updated_at"] < 12
PYRepository: PerryTS/perry
Length of output: 326
Measure freshness from updated_at.
For a completed successful run, use updated_at instead of created_at. A run can spend most of its budget queued and finish shortly before the check; created_at then makes the gate stale despite a fresh result. Add a self-test with an old created_at and recent updated_at.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/check_gate_freshness.py` around lines 131 - 135, Update the
workflow-run selection logic in the fetch path to return each completed
successful run’s updated_at value instead of created_at, while preserving the
existing head_sha and no-match behavior. Add a self-test covering a run with an
old created_at and recent updated_at, asserting freshness is based on
updated_at.
Comment on lines
+143
to
+151
| try: | ||
| found = newest_post_merge_success(repo, gate.workflow, branch, fetch=fetch) | ||
| except RuntimeError as exc: | ||
| # A workflow file that no longer exists, or an API failure, must not be | ||
| # silently treated as "fresh". Report it as stale with no timestamp. | ||
| print(f"::warning::{gate.workflow}: {exc}", file=sys.stderr) | ||
| found = None | ||
| if found is None: | ||
| verdicts.append(Verdict(gate, None, None, None)) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Keep API failures separate from stale verdicts.
Lines 145-151 convert an Actions API failure into Verdict(..., None, ...). Lines 187-188 then publish that as “NO successful post-merge run,” which is false when the query failed. Model this as an unknown collection error. Fail the monitor run and skip sticky-issue synchronization when collection is incomplete. Preserve an existing stale issue until a complete successful collection can close it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/check_gate_freshness.py` around lines 143 - 151, Update the
check_gate_freshness flow around newest_post_merge_success and Verdict creation
to distinguish API or workflow lookup failures from a genuine stale result.
Represent collection errors as an unknown/error outcome, make the monitor fail
when collection is incomplete, and skip sticky-issue synchronization in that
case; only publish “NO successful post-merge run” and close an existing stale
issue after a complete successful collection.
Comment on lines
+3
to
+4
| "Freshness budgets for the post-merge gate sweeps. Checked by", | ||
| "scripts/check_gate_freshness.py, run hourly by .github/workflows/gate-freshness.yml.", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Correct the freshness-check cadence.
The workflow runs every two hours at minute five. Both locations state that it runs hourly.
scripts/gate_freshness.json#L3-L4: Change “run hourly” to “run every two hours.”docs/src/testing/ci-gate-scheduling.md#L155-L160: Change “runs hourly” to “runs every two hours.”
📍 Affects 2 files
scripts/gate_freshness.json#L3-L4(this comment)docs/src/testing/ci-gate-scheduling.md#L155-L160
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/gate_freshness.json` around lines 3 - 4, Correct the freshness-check
cadence wording from hourly to every two hours in scripts/gate_freshness.json
lines 3-4 and docs/src/testing/ci-gate-scheduling.md lines 155-160; update both
references without changing the workflow configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7856.
The constraint I measured (it is not the one in the issue)
Repo-wide total Actions concurrency, not macOS runner availability. At 2026-08-11 15:07 UTC:
ubuntu-latest, 3macos-14)pull_request+ 208push)main/daypush: maingc-ratchetmainruns queued, of last 25Demand outran drain and the queue grew without bound. The entries that aged out are the
mainruns — precisely the ones that gate nothing, so nobody watches them.Two claims in #7856 do not survive checking
gc-root-dominancerunsmacos-14, notubuntu-latest(two jobs). Its healthy-looking run count was pull-request runs, which drain because they supersede each other; itsmainarm was queued like everything else. The ubuntu-vs-macOS contrast that localised this to macOS was comparing a PR arm against amainarm, not Linux against Darwin.zizmor(ubuntu) was queued in the same second asgc-ratchet(macOS).Consequence: rebalancing pools cannot help. Only cutting total job demand can. That also disposes of the "move it to Linux" direction — and independently,
gc-ratchet's baseline is keyeddarwin-arm64and its checker refuses a platform mismatch, so moving it would turn the gate red rather than relieve it.What I changed
1. Post-merge arm → staggered six-hourly sweep + release tags, for ten gates:
gc-ratchet,gc-root-dominance,tls-budget,gc-native-roots,gc-ptr-shape-off-witness,gc-parse-churn-gate,gc-moving-witnesses,auto-opt-app-patterns,eh-transport,llvm-inprocess.Removes 19 jobs from every merge: ~1,100 job-starts/day → ~76, a 93% cut on this slice. Cron minutes are staggered and none sits at
:00, so the ten do not re-create the herd they were meant to relieve.eh-transportandllvm-inprocessalso needed their relevance guard changed from= "push"to!= "pull_request"— under ascheduleevent the old test fell through to the PR branch and would dereference an empty PR number.2.
gate-freshness.yml+scripts/check_gate_freshness.py— a cheap ubuntu job that fails when a gate has no successful post-mergemainrun inside its budget (scripts/gate_freshness.json), and maintains a single self-closing sticky issue.pull_requestruns are deliberately not counted as evidence of health, because counting them is what madegc-root-dominancelook fine while it was dark.3.
docs/src/testing/ci-gate-scheduling.md— the measurement, the two refuted claims, and aDO NOT RESTORE push: branches: [main]note in each workflow pointing at it.Verification that the change can still fail
actionlintexits 0; shellcheck warning count is identical toorigin/main(6 vs 6), so I introduced none. All ten workflows were re-parsed to assert thepull_requestarm survived, nopush.branchesremains, and no two crons collide.The freshness checker is sabotage-tested, not merely exercised. Three independent sabotages, each caught with a specific message; pristine exits 0:
pull_requestruns count as post-merge evidenceLive
--dry-runagainst the real API exits 1 and quantifies the outage, which also surfaced two things outside the issue's scope:What I deliberately did NOT change — for maintainer review
concurrency:blocks. Already correct and twice-repaired (CI: gc-ratchet's main runs are cancelled while queued, so the gate has executed zero times in three merges #7205). "Fixing" them again is the trap; All macos-14 GC gates have been starved of runners for 2+ days: gc-ratchet and tls-budget have 0 successful main runs, 27 queued #7856 says so and I agree.gate-freshnessis explicitly not one — it reports on other gates and must never block a PR.security-auditstays on every merge (required context) — but it is itself stale at 17.4h. This needs a decision I did not make for you.container-tests(6 jobs/merge, ~350 job-starts/day) is the next-largest lever, but it is not a GC gate; left alone.on.pull_request.pathswould save PR-side slots — the larger half of demand, 392 of 600 runs — but a path-filtered workflow reports no status instead of a passing one, which can wedge a required context. Not worth the risk here.The cost, stated plainly
Attribution latency. A regression that slips past the PR arm used to be pinned to one commit; now the next sweep names it against a window of commits (bisect
previous sweep SHA .. this sweep SHA). In exchange the gate produces an answer at all, which for the two days before this it did not. Four completed runs a day beat 58 that never start.Note the schedules only take effect once this is on
main— GitHub readscronfrom the default branch.Summary by CodeRabbit
CI Improvements
Documentation