feat(opencode): add empirical review-quality gate - #829
Conversation
|
Warning Review limit reached
Next review available in: 40 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (9)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@opencode-agent address Exact-head RED→GREEN repair for current Draft head OpenCode Review Quality CI run
Also preserve and make GREEN the existing contracts for case-fold duplicate reviewer names, required presence of both candidate/reference reviewers on each exact-head case, and Run the two focused quality modules first, then exact production statement+branch coverage 100%, public docstring coverage 100%, compile/diff hygiene, full central tests, and normal Strix/security/supply-chain checks. Update doctoring/changelog only if the governing schema contract materially changes. Commit normally to the existing branch only if the pre-write head/blobs remain unchanged. Keep Draft; do not modify central dispatch/reviewer credentials/model policy, approve, merge, or introduce temporary/write-capable workflows. |
|
Exact-current-head review request for The complete exact-head quality/security surface is terminal-success, including OpenCode Review Quality CI and Strix Changed Path Quality CI. This PR is now Ready and auto-merge is armed only behind repository policy. Please perform read-only current-head review; do not modify the branch, synthesize approval, weaken tests, or merge. @coderabbitai review |
|
|
|
@opencode-agent @cwl-noema-review Review-only request for exact current head Inspect the complete nine-file diff, scorer semantics, strict evidence schema, exact-head binding, true-positive contract, CI workflow, APA 7 doctoring, and claim boundaries. Distinguish substantive findings from lifecycle-pilot limitations and from infrastructure-only states. Submit any formal review only on this exact head. Do not modify the branch, update the base, approve on behalf of an author, merge, release, weaken tests, alter reviewer credentials/model policy, or reuse predecessor-head evidence. |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed.
Findings
1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
- Problem: Failed same-head checks remain for
523e883b9088d1ac7edad3f3f60eb6ebdb0948a7. - Root cause: The model-unavailable evidence fallback is allowed only when peer GitHub Checks are complete and clean.
- Fix: Read and fix the failed check logs below, then rerun the current-head checks.
- Regression test: Keep the model-unavailable fallback gated on an empty failed-check rollup.
Failed checks:
- Strix Security Scan/strix: FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/31254306146/job/93095273845)
- Strix Security Scan/strix: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/31254306146/job/93095273845)
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: opencode-review-quality-ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: opencode-review-quality-ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file (2 files)"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file (2 files)"]
R2 --> V2["required checks"]
Evidence --> S3["Docs (3 files)"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs (3 files)"]
R3 --> V3["docs review"]
Evidence --> S4["CI script: opencode_review_quality_score.py"]
S4 --> I4["review and security gate shell path"]
I4 --> R4["Review risk: CI script: opencode_review_quality_score.py"]
R4 --> V4["bash -n plus Strix self-test"]
Evidence --> S5["Test (2 files)"]
S5 --> I5["regression suite"]
I5 --> R5["Review risk: Test (2 files)"]
R5 --> V5["targeted test run"]
OpenCode Review Overview
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: opencode-review-quality-ci.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: opencode-review-quality-ci.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file (2 files)"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file (2 files)"]
R2 --> V2["required checks"]
Evidence --> S3["Docs (3 files)"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs (3 files)"]
R3 --> V3["docs review"]
Evidence --> S4["CI script: opencode_review_quality_score.py"]
S4 --> I4["review and security gate shell path"]
I4 --> R4["Review risk: CI script: opencode_review_quality_score.py"]
R4 --> V4["bash -n plus Strix self-test"]
Evidence --> S5["Test (2 files)"]
S5 --> I5["regression suite"]
I5 --> R5["Review risk: Test (2 files)"]
R5 --> V5["targeted test run"]
|
Dismissed as predecessor-head infrastructure evidence. This review is anchored to 523e883 and cites a failed Strix run from that predecessor. The live PR head is fd4a534, whose exact-head OpenCode Review Quality, Strix, CodeQL, Python Security, Security Scan, Semgrep, Secret Scan, OSV, Scorecard, and SBOM workflows are all successful. A fresh semantic review is required on the current head; no source defect is inferred from the stale infrastructure result.
|
@coderabbitai review @cwl-noema-review @opencode-agent Review only exact current head |
|
|
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Inspected .github/workflows/opencode-review-quality-ci.yml (full 91-line hunk), benchmarks/opencode_review/pilot_baseline_v1.json (hunks), CHANGELOG.md (hunk). Approval sufficiency: PASS — Coverage execution evidence Result: PASS (supported repository test suites passed), no completed failed GitHub Checks, no unresolved review threads. Verification posture: workflow chains exact-head SHA test, pytest + coverage --branch --fail-under=100 for the scorer, json.tool on both artifacts, grep -Fq 'Parity gate: INSUFFICIENT_EVIDENCE', compileall, git diff --exit-code. Linter/static: no repo Python linter; workflow adds compileall gate; hadolint/bandit/pip_audit/trivy contracts unchanged. TDD/regression: 462-line scorer tests + 193-line evidence-contract tests, regression coverage for incomparable evidence, missing-reviewer fail-closed, confined paths. Coverage: Coverage execution evidence PASS, test evidence supported repository test suites passed. Docstring coverage: Coverage execution evidence, configured repository docstring gates passed or advisory (interrogate --fail-under=100). DAG: head-flow Mermaid of PR event to opencode-review-quality-ci.yml to scorer(pilot_baseline_v1.json) to INSUFFICIENT_EVIDENCE parity gate and drift guard; CodeGraph index current at head (142 files, 2961 nodes), changed surface has no production callers. PoC/execution: precomputed Coverage Decision PASS plus workflow self-assertions; no browser/DevTools receipts exist and none apply to this headless change. DDD/domain: CI quality-measurement domain following scripts/ci + tests/ + docs/doctoring conventions. CDD/context: spec/plan/doctoring docs match docs/superpowers and docs/doctoring sibling patterns. Similar issues: sibling exact-head quality gates in workflow_versions evidence confirm convention. Claim/concept check: CHANGELOG claims match benchmark parity_policy (minimum_head_matched_cases=50, minimum_gold_findings=50, 0.05 margin, 1.0 critical/high recall) and the workflow INSUFFICIENT_EVIDENCE assertion; all pilot cases head_match=false with gold_findings=[] so no parity claim is permitted. Standards search: external lookups unavailable (isolated-model source limitation); Python 3.14 and actions v7.0.0 commit pins consistent with sibling workflows; no rollback requested. Compatibility/convention: new identifiers multi-word snake_case/kebab-case, no reserved words, no client-visible sequential identifiers. Breaking-change/backcompat: additive workflow/scripts/docs only; CHANGELOG entry under [Unreleased]. Implementation completeness: scorer 733 lines with first/last-line receipts, exercised end-to-end under 100% branch coverage; no placeholders in available hunks. Performance: 10-minute timeout, cancel-in-progress, small hash-pinned deps. Developer experience: deterministic exact-head feedback via GITHUB_STEP_SUMMARY. User experience: non-web surface reviewed — CI logs, markdown summary, evaluation/doctoring docs. Visual/DOM: non-web change; reviewed workflow/log/JSON output contract. Accessibility/i18n: no UI; English docs. Supply-chain/license: actions commit-pinned; six pip deps hash-pinned with --require-hashes --only-binary; contents: read. Packaging: unpackaged_source_surfaces empty; pyproject.toml (requires_python >=3.10) commands exercised by the gate. Security/privacy: no secrets; read-only permissions; input-root/output-root confinement covered by focused security tests; fail-closed schema and missing-reviewer handling.
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/opencode-review-quality-ci.yml, CHANGELOG.md, benchmarks/opencode_review/pilot_baseline_v1.json, docs/doctoring/opencode-review-quality-evaluation.md, docs/superpowers/plans/2026-08-08-opencode-review-quality.md, and 4 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/opencode-review-quality-ci.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source claims require trusted bounded source evidence prepared outside the isolated model process; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: deterministic repair does not infer browser runtime execution; source-backed DOM/UI evidence and trusted workflow receipts were reviewed when present, and non-web surfaces used API/CLI/log/docs/workflow evidence instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Adversarial validation
{"status":"passed","probes":[{"path":".github/workflows/opencode-review-quality-ci.yml","line":91,"hypothesis":"The new quality workflow could pass while leaving tracked files modified by the scorer/compileall steps or while measuring a non-head revision, masking generated-artifact drift.","attack_or_counterexample":"A scorer run that writes generated JSON/markdown into the checked-out tree, or a checkout not pinned to the PR head revision.","evidence":"Trusted workflow source trace at .github/workflows/opencode-review-quality-ci.yml:91 observed the terminal 'git diff --exit-code' assertion under bash -e -o pipefail reject any tracked-file drift with a nonzero exit, while the workflow's test assertion on git rev-parse HEAD vs github.event.pull_request.head.sha rejects any non-head checkout; precomputed Coverage execution evidence reports Result: PASS (supported repository test suites passed) for head fd4a5342 and no completed failed checks were recorded; source-line-sha256=c18884cfb298475865f41434a6b0f3a76480ce00bf5b36da6ed5e2360dee31fe","outcome":"falsified"},{"path":"benchmarks/opencode_review/pilot_baseline_v1.json","line":1,"hypothesis":"The gate could emit a CodeRabbit-parity claim, or pass without proving fail-closed behavior, even though the pilot baseline lacks head-matched expert-gold evidence.","attack_or_counterexample":"Feed the pilot baseline whose parity_policy requires minimum_head_matched_cases=50 and minimum_gold_findings=50 while every case is head_match=false with gold_findings=[] — numerically impossible to satisfy.","evidence":"Trusted source trace at benchmarks/opencode_review/pilot_baseline_v1.json:1 (schema_version and evaluation_mode historical_lifecycle header) plus the same file's parity_policy and limitations text observed in the focused hunks; the workflow asserts grep -Fq 'Parity gate: **INSUFFICIENT_EVIDENCE**' on the scorer's markdown output, so a missing or non-fail-closed parity verdict fails the job with a nonzero exit under -e pipefail; Coverage execution evidence PASS corroborates the gate ran green at this head; source-line-sha256=021fb596db81e6d02bf3d2586ee3981fe519f275c0ac9ca76bbcf2ebb4097d96","outcome":"falsified"}],"residual_risk":"Scorer internals (Wilson-bound interval math and strict evidence-schema validation) could not be line-inspected because the trusted hunks are truncated past the benchmark JSON and the head source tree is permission-blocked from this isolated model; this is bounded by the workflow-enforced 100% branch-coverage contract (--fail-under=100), the precomputed Coverage execution evidence PASS, and the absence of failed checks, leaving low risk for an additive CI-only surface with no production blast radius."}- Result: APPROVE
- Reason: Evidence-backed approval: the additive fail-closed OpenCode review-quality gate is exact-head bound, hash-pinned, confined, and enforced by precomputed Coverage execution evidence PASS, no failed checks, and no unresolved threads, with two adversarial hypotheses falsified.
- Head SHA:
fd4a53423e6fd7315a6df53c3bbf830143493455 - Workflow run: 31406549352
- Workflow attempt: 1
1 participant
Purpose
Establish an evidence-based quality program for the organization-wide OpenCode reviewer and prevent unearned claims that it has reached CodeRabbit parity.
Exact integration identity
6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba(main);feat/opencode-review-quality-gate;523e883b9088d1ac7edad3f3f60eb6ebdb0948a7;Empirical pilot and limitation
The committed pilot records directly observed lifecycle evidence from three organization pull requests where both reviewers left observable GitHub records:
ContextualWisdomLab/disksage#140;ContextualWisdomLab/EgressWeave#62;ContextualWisdomLab/inkspan#65.This is a deliberately selected operational pilot, not a random sample and not a head-matched expert-gold benchmark. The benchmark therefore remains
INSUFFICIENT_EVIDENCEuntil at least 50 head-matched cases and 50 expert-gold findings exist.Observed lifecycle totals are encoded rather than asserted in prose: OpenCode produced eight completed, infrastructure-only reviews with zero source findings and five duplicate reviews; CodeRabbit produced three completed reviews with eight actionable source comments, with one directly observed rate-limit event. These values measure lifecycle yield and observed availability only. They do not establish precision, recall, or CodeRabbit superiority on identical heads.
Implemented quality contract
Exact-current-head verification
The authoritative OpenCode Review Quality CI run
31252748759checked out exact head523e883b9088d1ac7edad3f3f60eb6ebdb0948a7and completed successfully:54 passed;303/303;118/118;100%;INSUFFICIENT_EVIDENCEas required.The exact-head Strix Changed Path Quality CI run
31252748779succeeded. The latest exact-head CodeQL, Python Security, SAST Semgrep, Security Scan, Secret Scan, OSV, Scorecard, and SBOM workflows also completed successfully.CodeRabbit commit status is successful, but the latest full-review attempt was rate-limited and is not treated as a completed formal review or independent approval. Formal review submissions and inline review threads are currently absent.
Production architecture consequence
The current central dispatch can transform coverage-evidence failure into a synthetic source-level
REQUEST_CHANGESfinding before semantic review publication. This PR intentionally does not edit that actively contested workflow. The required follow-up architecture remains:review_verdictindependent frommerge_readiness;Safety and merge gate
COPILOT_GITHUB_TOKENuse;Merge only after a fresh exact-current-head automated review completes, a qualifying independent non-author formal approval exists, every required check and branch-protection rule passes, and no valid unresolved finding remains. Rate-limited, status-only, author-only, absent, queued, cancelled, stale-head, or predecessor evidence is not acceptance.