Skip to content

fix(ci): fast-path verified clean reverts without review claims - #5109

Open
benjaminshafii wants to merge 1 commit into
devfrom
fix/verified-revert-check-fastpath
Open

benjaminshafii wants to merge 1 commit into
devfrom
fix/verified-revert-check-fastpath

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Summary

  • Add one trusted-base, fail-closed exact-revert preflight shared by CI, Warden, and the revert approval workflow. Candidates must be open, ready, same-repository dev PRs with full immutable SHAs; current head/base and the dev tip are checked before and after Git tree verification. Titles/bodies are hints only.
  • Preserve the existing openwork-tests-required publisher. Its verified-revert lane succeeds only with validated exemption output and all expected heavy jobs skipped. Normal lanes and non-PR routing remain unchanged.
  • Keep the existing Warden job, explicitly report not reviewed, and skip analysis/reporting without inventing a review receipt. Clearance binds the actual successful producer attempt/marker and independently re-verifies the exact inverse before bypassing; it never grants Warden approval for an exemption. Ordinary missing receipts still fail closed.
  • Protect review machinery (including the verifier) from self-exemption; retain existing clearance guards and ruleset checks. Re-check approval eligibility against both verified head and base. This policy-changing PR takes normal checks and human review.
  • Extend the existing unit tests and Warden lifecycle journey with positive/negative boundaries; no new journey file. Wire those tests into the normal CI lane. Document the exact exemption and rollout caveat.

Verification on 0416daa642ac932e543a25098a9028b021a305ff

Exact command Result
node --test scripts/ci/verify-clean-revert.test.mjs exit 0; 15 passed, 0 skipped
node --test .github/scripts/warden-review.test.mjs exit 0; 23 passed, 0 skipped
node --test scripts/ci/workflow-authoring-gate.test.mjs exit 0; 13 passed, 0 skipped
node --test scripts/ci/ubuntu-apt-https.test.mjs exit 0; 25 passed, 0 skipped
pnpm evals:pr specs/warden-review-lifecycle.test.ts exit 0; 2 passed, 0 skipped; app-less local PR lane (Vitest printed no placement line)
git diff --check exit 0
actionlint -ignore 'label "blacksmith-' -ignore 'SC2015:' -ignore 'SC2024:' .github/workflows/ci-tests.yml .github/workflows/revert-fastlane.yml .github/workflows/warden.yml .github/workflows/warden-clearance.yml exit 0

Raw actionlint on those same four files exited 1: custom Blacksmith runner labels plus SC2015/SC2024 in the unchanged APT bootstrap. A pristine detached control at 854695eb1677ead48c314bbefd4ff3f78eafa029 reproduces the same six diagnostics (exit 1); only those diagnostics were filtered above. No repo-wide build was run.

Earlier development reds: one new unit assertion expected an exception for an empty diff, but the preflight correctly returned ineligible before verification; fixed the assertion and reran. The first journey invocation exited 1 before tests because the separate evals workspace was not installed; pnpm --dir evals install --frozen-lockfile fixed the environment. Root and evals frozen installs both exited 0.

Full final local Warden: Incomplete — credentials required

Sole owner ran the exact unscoped pnpm warden:check after committed-head journey verification. Exit 1, auth_failed; zero model tokens and no reviewed skill coverage. Do not interpret the CLI's zero findings as clearance.

  • Expected/matched skills: diff-security-review, confidentiality-review, spec-provenance-review.
  • Actual reviewed coverage: none; authentication failed and the remaining skills/files skipped. No native finding IDs were produced.
  • Scope: 13 changed files, 30 chunks, origin/dev...HEAD.
  • Before and after HEAD: 0416daa642ac932e543a25098a9028b021a305ff.
  • Before and after origin/dev: 854695eb1677ead48c314bbefd4ff3f78eafa029.
  • Local run reference: 8e7b06dc-2026-09-17T14-26-55-694Z (private analysis log not published).
  • Needs an approved WARDEN_OPENAI_API_KEY in the process environment or Infisical dev, then the same bare command rerun with refs recorded. A name-only Infisical lookup found no Warden/OpenAI credentials in dev.

Rollout / limitations

This PR is ready for review, not asserted merge-ready. No merge, branch-rule change, admin bypass, global disabling, or duplicate success status was performed. Local tests cover routing and Git/API boundaries, not live GitHub execution. Trusted-base/default-branch helpers must land before an eligible revert can exercise the exemption. Validate the real fast lane after rollout; runner queues prevent any seconds-of-walltime guarantee.

@vercel

vercel Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
openwork-app Ready Ready Preview Sep 17, 2026 2:29pm UTC
openwork-den Ready Ready Preview Sep 17, 2026 2:29pm UTC
openwork-diagnostics Ready Ready Preview Sep 17, 2026 2:29pm UTC
openwork-landing Ready Ready Preview, v0 Sep 17, 2026 2:29pm UTC

@benjaminshafii

benjaminshafii commented Sep 17, 2026 •

Copy link
Copy Markdown
Member Author

Selected evidence: Incomplete · 12/12 tests · 18/18 assertions · 8 images

Commit 0416daa642ac932e543a25098a9028b021a305ff · selected evidence
Required verification is reported separately by the current-head Required verification check.

Open review report

Coverage gaps: Required verification: incomplete. Selected evidence does not satisfy all required specs. No authenticated current-head required plan is available.

@github-actions

Copy link
Copy Markdown
Contributor

Warden review summary — Clear

Native findings: 0. Blocking policy matches before consolidation: 0. Full finding text remains in Warden's native checks and review threads. This comment retains only native IDs, severity, skill attribution, and safe locations. Analysis run

Reported trigger coverage (not proof of complete repository or context coverage): diff-security-review (152510 ms), confidentiality-review (192617 ms), spec-provenance-review (239125 ms).

Blockers

  • None.

Advisories

  • None.

Needs recheck

  • None.

Observed review-thread metrics

  • Unique observed run attempts in retained window: 1
  • Distinct heads in retained window: 1
  • Current unresolved Warden threads: 0 of 0 observed
  • Observed resolved/unresolved transitions in retained snapshots: 0 / 0
  • First observed resolved/unresolved (state at first collection, not event time): 0 / 0
  • Precision: unavailable (no adjudications; no TP/FP values are inferred)
  • Observation window: 1/20 run attempts retained; truncated: false
  • Transition snapshot: 0/0 current attributed threads retained (limit 500); truncated: false

Thread states are read-only collection snapshots. These metrics are neither lifetime totals nor exact resolution times, and this summary is not clearance authority.

This branch was successfully deployed

4 active deployments
Preview – openwork-landing — 0416daa6 Deployed Sep 17, 2026 by vercel[bot]
Preview – openwork-app — 0416daa6 Deployed Sep 17, 2026 by vercel[bot]
Preview – openwork-den — 0416daa6 Deployed Sep 17, 2026 by vercel[bot]
Preview – openwork-diagnostics — 0416daa6 Deployed Sep 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant