fix(opencode): bind coverage artifacts to workflow attempts #812
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| name: OpenCode Coverage Artifact Rerun Quality CI | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: [main] | ||
| paths: | ||
| - ".github/workflows/opencode-review-dispatch.yml" | ||
| - ".github/workflows/opencode-coverage-artifact-rerun-quality-ci.yml" | ||
| - "tests/test_opencode_coverage_artifact_rerun_contract.py" | ||
| - "docs/doctoring/opencode-coverage-artifact-reruns.md" | ||
| - "CHANGELOG.md" | ||
| - "requirements-opencode-review-ci-hashes.txt" | ||
| - "pyproject.toml" | ||
| push: | ||
| branches: [main] | ||
| paths: | ||
| - ".github/workflows/opencode-review-dispatch.yml" | ||
| - ".github/workflows/opencode-coverage-artifact-rerun-quality-ci.yml" | ||
| - "tests/test_opencode_coverage_artifact_rerun_contract.py" | ||
| - "docs/doctoring/opencode-coverage-artifact-reruns.md" | ||
| - "CHANGELOG.md" | ||
| - "requirements-opencode-review-ci-hashes.txt" | ||
| - "pyproject.toml" | ||
|
|
||
| concurrency: | ||
| group: opencode-coverage-artifact-rerun-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| env: | ||
| FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true | ||
|
|
||
| jobs: | ||
| exact-head-contract: | ||
| name: Python 3.14 attempt-scoped artifact contract | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 25 | ||
| steps: | ||
| - name: Harden runner | ||
| uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 | ||
| with: | ||
| egress-policy: audit | ||
|
|
||
| - name: Checkout exact source | ||
| uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 | ||
| with: | ||
| persist-credentials: false | ||
| ref: ${{ github.event.pull_request.head.sha || github.sha }} | ||
|
|
||
| - name: Set up current stable Python | ||
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | ||
| with: | ||
| python-version: "3.14" | ||
| cache: pip | ||
| cache-dependency-path: requirements-opencode-review-ci-hashes.txt | ||
|
|
||
| - name: Install hash-locked quality tooling | ||
| run: python -m pip install --disable-pip-version-check --require-hashes -r requirements-opencode-review-ci-hashes.txt | ||
|
|
||
| - name: Run attempt-scoped artifact regression | ||
| run: python -m pytest tests/test_opencode_coverage_artifact_rerun_contract.py -q | ||
|
|
||
| - name: Enforce complete central test and branch coverage | ||
| run: | | ||
| set -euo pipefail | ||
| python -m coverage erase | ||
| python -m coverage run -m pytest tests -q | ||
| python -m coverage report --show-missing | ||
|
|
||
| - name: Enforce production docstring coverage | ||
| run: python -m interrogate scripts/ci | ||
|
|
||
| - name: Compile permanent contracts | ||
| run: python -m compileall -q scripts tests | ||
|
|
||
| - name: Reject uncommitted generated state | ||
| run: git diff --exit-code --check && test -z "$(git status --porcelain)" | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -222,6 +222,9 @@ jobs: | |||||||||||||||||
| permissions: | ||||||||||||||||||
| contents: read | ||||||||||||||||||
| id-token: write | ||||||||||||||||||
| outputs: | ||||||||||||||||||
| coverage_source_artifact_id: ${{ steps.coverage_source_upload.outputs.artifact-id }} | ||||||||||||||||||
| coverage_source_run_attempt: ${{ steps.coverage_source_attempt.outputs.run_attempt }} | ||||||||||||||||||
| env: | ||||||||||||||||||
| FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true | ||||||||||||||||||
| steps: | ||||||||||||||||||
|
|
@@ -349,10 +352,23 @@ jobs: | |||||||||||||||||
| git -C "$COVERAGE_SOURCE_WORKDIR" status --short | ||||||||||||||||||
| tar -cf "$COVERAGE_SOURCE_ARCHIVE" -C "$COVERAGE_SOURCE_WORKDIR" . | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Record coverage source workflow attempt | ||||||||||||||||||
| id: coverage_source_attempt | ||||||||||||||||||
| env: | ||||||||||||||||||
| GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} | ||||||||||||||||||
| shell: bash --noprofile --norc -e -o pipefail {0} | ||||||||||||||||||
| run: | | ||||||||||||||||||
| if ! [[ "$GITHUB_RUN_ATTEMPT" =~ ^[1-9][0-9]*$ ]]; then | ||||||||||||||||||
| echo "::error::Coverage producer workflow attempt is not a positive integer." | ||||||||||||||||||
| exit 1 | ||||||||||||||||||
| fi | ||||||||||||||||||
| printf 'run_attempt=%s\n' "$GITHUB_RUN_ATTEMPT" >>"$GITHUB_OUTPUT" | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Upload materialized pull request merge tree | ||||||||||||||||||
| id: coverage_source_upload | ||||||||||||||||||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||||||||||||||||||
| with: | ||||||||||||||||||
| name: opencode-coverage-source-${{ github.run_attempt }} | ||||||||||||||||||
| path: ${{ runner.temp }}/opencode-coverage-source.tar | ||||||||||||||||||
| if-no-files-found: error | ||||||||||||||||||
| retention-days: 1 | ||||||||||||||||||
|
|
@@ -433,12 +449,48 @@ jobs: | |||||||||||||||||
| echo "::error::Coverage source tree could not be materialized; see the coverage-source-tree job log for the exact target repository, base SHA, head SHA, and fetch or merge failure." | ||||||||||||||||||
| exit 1 | ||||||||||||||||||
|
Contributor
Author
There was a problem hiding this comment. This early termination prevents the later
Suggested change
|
||||||||||||||||||
|
|
||||||||||||||||||
| - name: Verify coverage source identity for current workflow attempt | ||||||||||||||||||
| id: coverage_source_identity | ||||||||||||||||||
|
Contributor
Author
There was a problem hiding this comment. Identity validation must still run after a producer failure so it can produce a bounded, inspectable outcome for the unified recovery gate.
Suggested change
|
||||||||||||||||||
| continue-on-error: true | ||||||||||||||||||
| env: | ||||||||||||||||||
| COVERAGE_SOURCE_ARTIFACT_ID: ${{ needs.coverage-source-tree.outputs.coverage_source_artifact_id }} | ||||||||||||||||||
| COVERAGE_SOURCE_RUN_ATTEMPT: ${{ needs.coverage-source-tree.outputs.coverage_source_run_attempt }} | ||||||||||||||||||
| CURRENT_RUN_ATTEMPT: ${{ github.run_attempt }} | ||||||||||||||||||
| shell: bash --noprofile --norc -e -o pipefail {0} | ||||||||||||||||||
| run: | | ||||||||||||||||||
| if ! [[ "$CURRENT_RUN_ATTEMPT" =~ ^[1-9][0-9]*$ ]] || \ | ||||||||||||||||||
| [ "$COVERAGE_SOURCE_RUN_ATTEMPT" != "$CURRENT_RUN_ATTEMPT" ]; then | ||||||||||||||||||
| echo "::error::Coverage source was not produced in current workflow attempt ${CURRENT_RUN_ATTEMPT:-missing}; producer attempt=${COVERAGE_SOURCE_RUN_ATTEMPT:-missing}." | ||||||||||||||||||
| echo "::error::Use a full rerun or a fresh repository dispatch; failed-jobs-only reruns cannot reuse prior-attempt source evidence." | ||||||||||||||||||
| exit 1 | ||||||||||||||||||
| fi | ||||||||||||||||||
| if ! [[ "$COVERAGE_SOURCE_ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]; then | ||||||||||||||||||
| echo "::error::Coverage source artifact ID is missing or malformed for current workflow attempt." | ||||||||||||||||||
| echo "::error::Use a full rerun or a fresh repository dispatch so the producer publishes current-attempt evidence." | ||||||||||||||||||
| exit 1 | ||||||||||||||||||
| fi | ||||||||||||||||||
| artifact_id=$COVERAGE_SOURCE_ARTIFACT_ID | ||||||||||||||||||
| printf 'artifact_id=%s\n' "$artifact_id" >>"$GITHUB_OUTPUT" | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Download current-attempt materialized pull request merge tree | ||||||||||||||||||
| if: steps.coverage_source_identity.outcome == 'success' | ||||||||||||||||||
|
Contributor
Author
There was a problem hiding this comment. Download must run only for a successful producer and successful current-attempt identity, while remaining evaluable under failed-job reruns.
Suggested change
|
||||||||||||||||||
| id: coverage_source_download | ||||||||||||||||||
| continue-on-error: true | ||||||||||||||||||
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | ||||||||||||||||||
| with: | ||||||||||||||||||
| artifact-ids: ${{ steps.coverage_source_identity.outputs.artifact_id }} | ||||||||||||||||||
| path: ${{ runner.temp }}/opencode-coverage-artifact | ||||||||||||||||||
|
seonghobae marked this conversation as resolved.
|
||||||||||||||||||
|
|
||||||||||||||||||
| - name: Report missing current-attempt coverage source | ||||||||||||||||||
| if: steps.coverage_source_identity.outcome != 'success' || steps.coverage_source_download.outcome != 'success' | ||||||||||||||||||
|
Contributor
Author
There was a problem hiding this comment. Make the final recovery step authoritative for every producer, identity, or download failure.
Suggested change
|
||||||||||||||||||
| env: | ||||||||||||||||||
| GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }} | ||||||||||||||||||
| run: | | ||||||||||||||||||
| set -euo pipefail | ||||||||||||||||||
| echo "::error::Coverage source evidence is unavailable for workflow run attempt ${GITHUB_RUN_ATTEMPT}; a failed-jobs-only rerun cannot safely reconstruct or reuse source evidence from another attempt." | ||||||||||||||||||
| echo "::error::Use a full rerun or a fresh repository dispatch so coverage-source-tree uploads exact current-attempt evidence." | ||||||||||||||||||
| exit 1 | ||||||||||||||||||
|
Comment on lines
+484
to
+492
There was a problem hiding this comment. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
file=".github/workflows/opencode-review-dispatch.yml"
printf '%s\n' '--- target workflow section ---'
sed -n '400,470p' "$file"
printf '%s\n' '--- related job/step references ---'
rg -n -C 3 'coverage-source-tree|coverage_source_download|Report coverage source|Report missing current' "$file"Repository: ContextualWisdomLab/.github Length of output: 6826 🏁 Script executed: #!/bin/bash
set -euo pipefail
file=".github/workflows/opencode-review-dispatch.yml"
printf '%s\n' '--- workflow structure around the consumer job ---'
sed -n '1,120p' "$file"
printf '%s\n' '--- all needs and status-check expressions in this file ---'
rg -n '^\s*(needs:|if:|uses:|name:)|always\(\)|success\(\)|failure\(\)|cancelled\(\)' "$file" | sed -n '1,240p'Repository: ContextualWisdomLab/.github Length of output: 6439 🏁 Script executed: #!/bin/bash
set -euo pipefail
file=".github/workflows/opencode-review-dispatch.yml"
printf '%s\n' '--- complete coverage-evidence job control flow ---'
sed -n '363,490p' "$file"
printf '%s\n' '--- producer output and terminal behavior ---'
sed -n '215,365p' "$file"
printf '%s\n' '--- deterministic step-gating probe ---'
python3 - "$file" <<'PY'
from pathlib import Path
import re
import sys
text = Path(sys.argv[1]).read_text()
job = text[text.index(" coverage-evidence:"):text.index("\n opencode-review:", text.index(" coverage-evidence:"))]
steps = re.findall(r"(?m)^ - name: (.+)$", job)
for i, name in enumerate(steps):
print(f"{i}: {name}")
def implicit_success(previous_failed):
return not previous_failed
# Model the relevant path: the producer is not successful, so the explicit
# materialization-report condition is true and that step exits 1.
previous_failed = False
report_condition = "needs.coverage-source-tree.result != 'success'"
report_runs = True
if report_runs:
previous_failed = True
download_runs = implicit_success(previous_failed)
final_report_runs = implicit_success(previous_failed)
print(f"producer_result=failure")
print(f"materialization_report_runs={report_runs}")
print(f"materialization_report_exits=1")
print(f"download_runs={download_runs}")
print(f"missing_source_report_runs={final_report_runs}")
assert report_runs and not download_runs and not final_report_runs
PYRepository: ContextualWisdomLab/.github Length of output: 13228 🌐 Web query:
💡 Result: In GitHub Actions, every step has an implicit success status check applied to its if conditional [1][2][3]. This means that by default, a step will only execute if all previous steps have succeeded [4][3]. Key points regarding this behavior: 1. Loss of Implicit Guard: When you provide a custom if condition for a step, the implicit success check is removed [4][3]. Your custom condition becomes the sole determinant of whether the step runs [3]. If your custom condition does not explicitly include a status check function (like success, failure, or always), the step will execute even if a previous step failed [4][3]. 2. Overriding the Default: If you want to maintain the requirement that a step only runs if previous steps succeeded while also adding your own custom logic, you must explicitly include success in your condition [1][3]. For example: if: ${{ success && github.ref == 'refs/heads/main' }} [3] 3. Handling Failures: If you intend for a step to run specifically after a failure, you must use the failure function in your if condition [1][5]. Simply adding an if condition (without failure or always) will not cause the step to run after a failure; it will result in the step being skipped because the implicit success guard is gone and the custom condition is likely false (or, if the custom condition is true, it may run unintendedly) [4][3]. 4. Running Regardless: If you want a step to run regardless of the outcome of previous steps (e.g., for cleanup tasks), you must explicitly use the always status check function [5][6][3]. In summary, the implicit success check acts as a safeguard that is automatically dropped the moment any custom if condition is defined [4][3]. Developers must manually re-implement status checks if they wish to preserve or modify this default execution behavior [4][3]. Citations:
🌐 Web query:
💡 Result: In the context of GitHub Actions workflows, the success function acts as the default status check for if conditional statements [1]. When you define an if conditional in a workflow step or job, GitHub Actions automatically applies success if no other status check function (such as failure, always, or cancelled) is explicitly included [1]. This ensures that, by default, a step or job will only execute if all previous steps or the job itself have succeeded [1]. To execute a step regardless of previous failures, or specifically upon a failure, you must explicitly use a different status check function to override this default behavior [1]. Citations: 🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
from urllib.request import urlopen
url = "https://docs.github.com/en/actions/reference/workflows-and-actions/expressions"
try:
text = urlopen(url, timeout=20).read().decode("utf-8", "replace")
except Exception as exc:
print(f"FETCH_ERROR: {exc}")
else:
needles = [
"A default status check of success() is applied",
"unless you include one of these functions",
"If you have a conditional that does not contain a status check function",
]
for needle in needles:
index = text.find(needle)
print(f"{needle}: index={index}")
if index >= 0:
print(text[max(0,index-500):index+700])
PYRepository: ContextualWisdomLab/.github Length of output: 313
🤖 Prompt for AI Agents
Contributor
Author
There was a problem hiding this comment. @coderabbitai autofix This finding is still valid on exact head
Preserve the producer-attested run-attempt output, immutable artifact ID, one-day retention, There was a problem hiding this comment. Rate Limit Exceeded
Contributor
Author
There was a problem hiding this comment. @coderabbitai autofix Exact head
Do not create or restore any patch file, helper workflow, branch, or PR. Preserve artifact ID/run-attempt binding, one-day retention, action pins, archive validation, and |
||||||||||||||||||
|
|
||||||||||||||||||
| - name: Prepare pull request merge tree for coverage measurement | ||||||||||||||||||
| env: | ||||||||||||||||||
| COVERAGE_SOURCE_ARCHIVE: ${{ runner.temp }}/opencode-coverage-artifact/opencode-coverage-source.tar | ||||||||||||||||||
|
|
||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # OpenCode coverage artifact rerun contract | ||
|
|
||
| ## Decision | ||
|
|
||
| The central OpenCode review workflow binds every materialized pull-request merge tree to one workflow-run attempt and one immutable GitHub Actions artifact identifier. The credential-free `coverage-evidence` job may consume only that exact artifact identifier. It never searches by a mutable artifact name and never falls back to an artifact produced by another run or attempt. | ||
|
|
||
| The producer also exports a step-recorded literal workflow attempt. Before download, the consumer verifies that this attempt equals its current `github.run_attempt` and that the immutable artifact ID is a positive decimal identifier. Artifact immutability selects one upload; attempt attestation proves that the producer executed in the current attempt. | ||
|
|
||
| The source artifact retains the existing one-day retention period. A failed-jobs-only rerun that does not rerun the successful producer is therefore expected to fail closed once that producer artifact expires. The operator response is a **full rerun or a fresh repository dispatch**, both of which rerun `coverage-source-tree` and create current-attempt evidence. Increasing retention or reusing prior-attempt source evidence is not an accepted repair. | ||
|
|
||
| ## Incident | ||
|
|
||
| On August 7, 2026, failed-jobs-only rerun attempt 2 of OpenCode workflow run `31022108085` retried `coverage-evidence` for `ContextualWisdomLab/pg-llm-batch#53` without retrying the successful `coverage-source-tree` producer. The attempt-1 artifact `opencode-coverage-source` had a one-day retention period and was already expired. `actions/download-artifact` therefore returned `Artifact not found` before any current-head tests or docstring checks could run. | ||
|
|
||
| The product pull request was not the source of this failure. The failing boundary was the central producer/consumer lifecycle: a static name did not prove that the consumer received evidence uploaded by the current attempt. | ||
|
|
||
| ## Contract | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| participant D as Repository dispatch | ||
| participant V as validate-pr-metadata | ||
| participant P as coverage-source-tree | ||
| participant A as Immutable Actions artifact | ||
| participant C as coverage-evidence | ||
|
|
||
| D->>V: Exact repository, PR, base SHA, head SHA | ||
| V->>P: Validated current-head metadata | ||
| P->>P: Materialize exact merge tree | ||
| P->>A: Upload attempt-scoped name | ||
| A-->>P: artifact-id | ||
| P-->>C: Immutable artifact-id job output | ||
| C->>A: Download exact artifact-id | ||
| alt Artifact belongs to current producer attempt | ||
| A-->>C: Merge-tree archive | ||
| C->>C: Validate archive, sandbox tests, coverage, docstrings | ||
| else Producer was omitted or evidence expired | ||
| A-->>C: Download failure | ||
| C-->>D: Fail closed; require full rerun or fresh dispatch | ||
| end | ||
| ``` | ||
|
|
||
| The implementation must preserve all of the following properties: | ||
|
|
||
| - `coverage-source-tree` remains the only job with repository-read and OIDC credentials for target-repository materialization. | ||
| - `coverage-evidence` remains limited to `actions: read`; it receives no repository-content token, OIDC credential, model secret, or review-write credential. | ||
| - The upload name includes `github.run_attempt` for operator diagnostics and collision resistance. | ||
| - The upload step exports the immutable `artifact-id`; the consumer validates that it is a positive decimal identifier and passes only the validated step output to `download-artifact`. | ||
| - The producer exports its step-recorded run attempt; the consumer rejects empty or prior-attempt provenance before download. | ||
| - Retention remains one day to minimize retention of private source evidence. | ||
| - Missing current-attempt evidence produces a bounded diagnostic containing the run attempt and the required recovery action. | ||
| - Exact-head metadata validation, same-repository validation, merge-tree construction, archive-member validation, isolated execution, coverage, docstring, security, and approval gates remain unchanged. | ||
|
|
||
| ## Rerun operations | ||
|
|
||
| | Operator action | Producer behavior | Consumer behavior | Accepted outcome | | ||
| |---|---|---|---| | ||
| | Fresh repository dispatch | Producer runs and uploads a new attempt-scoped artifact | Downloads the producer's immutable artifact ID | Accepted | | ||
| | Full workflow rerun | Producer reruns and uploads a new attempt-scoped artifact | Downloads the new immutable artifact ID | Accepted | | ||
| | Failed-jobs-only rerun while producer is omitted | Producer attempt marker or artifact ID is missing or belongs to an earlier attempt | Rejects identity before download | Expected failure | | ||
| | Attempt to reuse an earlier artifact by name | Current-attempt identity is not proven | Rejected by contract | Rejected | | ||
| | Increase retention to hide missing producer execution | Stale source remains available longer | Does not repair attempt identity | Rejected | | ||
|
|
||
| ## Security and privacy rationale | ||
|
|
||
| Artifact immutability prevents later jobs from mutating a successfully uploaded archive, but immutability alone does not identify which workflow attempt produced the archive. The producer's exact `artifact-id` closes upload-selection ambiguity, while its step-recorded attempt closes execution-attempt ambiguity. The consumer validates both before download; attempt-qualified names remain diagnostic only. | ||
|
|
||
| The one-day retention period is intentionally short because the archive can contain proprietary or otherwise sensitive source code. Recovery must create fresh, exact-head evidence rather than preserve source archives for a longer period. No product test executes in the credentialed producer. No trusted follow-up consumes command files after untrusted coverage execution begins. | ||
|
|
||
| ## Rollback | ||
|
|
||
| Rollback consists of reverting the attempt-scoped producer output and exact-ID consumer selection together. Reverting only one side leaves the workflow unable to exchange evidence. A rollback must preserve one-day retention, credential separation, and fail-closed behavior; it must not restore mutable-name fallback across attempts. | ||
|
|
||
| ## Verification | ||
|
|
||
| The permanent regression suite must verify: | ||
|
|
||
| 1. attempt-scoped artifact naming and immutable `artifact-id` producer output; | ||
| 2. producer-attested attempt output and pre-download current-attempt equality; | ||
| 3. positive-decimal artifact-ID validation and exact-ID download; | ||
| 4. actionable failure for missing, malformed, or prior-attempt evidence; | ||
| 5. one-day retention; and | ||
| 6. absence of repository, OIDC, secret, and review-write credentials from `coverage-evidence`. | ||
|
|
||
| The complete repository test suite, Python compilation, production statement and branch coverage, public docstring gate, security and supply-chain checks, current-head review, independent approval, and protected merge remain required. | ||
|
|
||
| ## References | ||
|
|
||
| GitHub. (2026a). *Downloading workflow artifacts*. GitHub Actions documentation. https://docs.github.com/en/actions/how-tos/manage-workflow-runs/download-workflow-artifacts | ||
|
|
||
| GitHub. (2026b). *Re-running workflows and jobs*. GitHub Actions documentation. https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs | ||
|
|
||
| GitHub. (2026c). *actions/download-artifact* [Computer software]. GitHub. https://github.com/actions/download-artifact | ||
|
|
||
| GitHub. (2026d). *actions/upload-artifact* [Computer software]. GitHub. https://github.com/actions/upload-artifact |
Uh oh!
There was an error while loading. Please reload this page.