CI: PRs opened with a label get stuck on 'Required Checks — Expected'

[andygrove]

Describe the bug

A PR that is opened with a label already applied never gets a usable Required Checks status. The merge box shows Required Checks — Expected — Waiting for status to be reported. "Merge when ready" then waits forever with "This pull request will be added to the merge queue when all requirements are met".

Opening a PR with a label sends both an opened event and a labeled event for the same commit, so ci.yml runs twice on that SHA:

• the opened run publishes Required Checks (success)
• the labeled run, which starts a few seconds later, publishes only Required Checks (label run)

The check-runs API and GraphQL (isRequired: true, rollup SUCCESS) both show Required Checks as present and passing. The PR's merge requirements seem to be evaluated against the newest check suite of the workflow, though, and that suite is the label run, which has no Required Checks. The label-run renaming added for #5007 assumed GitHub uses the most recent check run per name. For merge requirements it appears to use the most recent run of the workflow.

Steps to reproduce

Seen on #6156, which was opened with run-all-spark-profiles already applied:

• run 35874311100 (opened): Required Checks success at 15:13:45Z
• run 35874318859 (labeled): Required Checks (label run) success at 15:13:23Z
• mergeStateStatus: BLOCKED, auto-merge enabled, the PR never enters the queue

Every recent PR that did enter the queue (#6094, #6073, #4816, #6041, #5677, #6072) had exactly one Comet CI run at its head SHA.

Expected behavior

Labels applied when the PR is opened should not stop it from merging.

Additional context

Possible directions:

• skip or cancel the labeled run when the label event arrives together with opened (same SHA, within seconds), so only one check suite exists
• have the label run also publish Required Checks, but only as a pass-through of the commit run's verdict (this needs to preserve the CI: non-gating label (e.g. auto-added performance) spawns a phantom all-skipped CI run that shadows real checks #5007 guarantee)
• document the workaround: re-run the Required Checks job in the opened run, or push a new commit

[andygrove]

Re-running only the Required Checks job in the opened run (35874311100) does not unblock the PR. The rerun succeeded at 17:26:33Z, and two minutes later #6156 was still BLOCKED and not in the queue. That fits the explanation above: a job rerun reuses the existing check suite, so the labeled run's suite is still the newest. The only workaround is a new check suite at a new commit, either by pushing a commit or by closing and reopening the PR.

[comphead]

Agree