diff --git a/.claude/skills/pr-review-process/SKILL.md b/.claude/skills/pr-review-process/SKILL.md new file mode 100644 index 0000000..bfb9254 --- /dev/null +++ b/.claude/skills/pr-review-process/SKILL.md @@ -0,0 +1,117 @@ +--- +name: pr-review-process +description: Procedure for triaging merged findings from parallel PR reviewers (currently GPT, Claude, and Claude Code), bundling fixes into at most 3 task branches (<10 files each), opening a PR per bundle, and summarizing everything back on the original PR. Used by the multi-model PR review GitHub Action. +--- + +# PR Review → Plan → Implement Process + +You are running non-interactively in CI, on a checkout of the reviewed PR's head branch. Several +models already reviewed this PR's diff independently, each following the +[[review-response-format]] contract. Their raw outputs are on disk as JSON files. Your job is to +triage those findings, decide what's worth auto-fixing, implement it, and report back. + +You have full `Bash`/`Read`/`Write`/`Edit` access and an authenticated `gh` CLI. Work directly in +the checked-out repo — do not ask for confirmation, this run is unattended. + +## Inputs you'll be given in the prompt + +- Paths to one findings file per reviewer, each shaped per [[review-response-format]] (a + `parse_error: true` file means that reviewer's output wasn't valid JSON — treat its findings as + empty but mention the failure in your final summary). Each file may also carry a top-level + `usage` field — `{ input_tokens, output_tokens, total_tokens, cost_usd }` — added by the calling + script, not by the reviewer model itself. `cost_usd` is an *estimate* from a hardcoded + $/token pricing table (not returned by either provider's API), so treat it as approximate, + round it to a sensible precision, and never invent a figure when the field is absent. +- The original PR number, its head branch, and its base branch. +- Who initiated this review run (as `Initiated by: @`). +- The implement model you yourself are running as (as `Implement model: `). Each + reviewer's own model id is already on its findings file as a top-level `model` field — don't + ask for it separately. + +## Step 1 — Merge and rank findings + +- Load every findings file you were given. If a file has `parse_error: true` or is missing, skip + it and note the gap. +- Findings from different reviewers describing the same underlying issue (same file, overlapping + lines, same root cause) are duplicates — merge them into one entry and record which reviewers + flagged it. A finding flagged by 2+ reviewers is higher-confidence than a single-reviewer + finding of the same severity; treat multi-reviewer agreement as a tiebreaker above severity + order. +- Rank the merged list: `critical` > `high` > `medium` > `low`, with multi-reviewer agreement + breaking ties within a severity tier. +- Drop findings that are pure style/taste with no concrete suggested fix, or that would require + a design decision only a human should make (e.g. "consider a different architecture here"). + You are implementing surgical fixes, not redesigning the PR. + +## Step 2 — Bundle into task groups + +- Group the remaining ranked findings into coherent bundles — findings that touch the same + subsystem/feature area belong together so each resulting PR tells one story. +- Hard constraints: + - **At most 3 bundles total.** Take the highest-priority findings first; anything left over + once you have 3 bundles (or once remaining findings are too risky/ambiguous to auto-fix) + stays unaddressed — list it in the final PR comment instead of forcing it in. + - **Each bundle must touch fewer than 10 files.** If a coherent group would exceed that, either + split it into two bundles or drop its lowest-priority members until it fits — never exceed + the limit. +- It's fine to ship fewer than 3 bundles, or a single bundle, if that's all the findings support. + Don't manufacture busywork to hit 3. + +## Step 3 — Implement each bundle + +For each bundle, in priority order: + +1. `git fetch origin ` and branch from its tip: + `git checkout -b task/- origin/` where `` is the bundle's 1-based + index and `` is a short kebab-case description (e.g. `task/1-fix-auth-null-checks`). +2. Implement the fix for every finding in the bundle. Keep changes minimal and scoped to what the + finding describes — this is a targeted fix, not a refactor. Match the surrounding code's style. +3. If a fast build/lint/test command is obviously available for the touched area (e.g. a + `package.json` script, existing CI config you can read for the command), run it and fix + anything it flags. Don't go hunting for a test suite that isn't obviously there, and don't let + this block you if nothing fast is available. +4. Commit with a message summarizing the bundle and referencing each finding's `id`. Stage only + the specific files you intentionally edited for this bundle (`git add ...`) — never + `git add -A` or `git add .`. The working tree may contain files unrelated to any bundle + (including this skill's own files, if they were staged into the checkout for this run) that + must never end up in a commit. +5. `git push -u origin task/-`. +6. `gh pr create --base --head task/- --title "[PR #] " --body "..."` — + the `[PR #]` prefix makes it immediately obvious, from the PR list alone, which original PR + each task PR feeds back into (it also means multiple concurrent review runs on different PRs + don't produce ambiguous-looking task PRs). The base is the **original PR's head branch**, not + its base branch, so merging this task PR feeds the fix back into the PR under review. The body + must: + - Summarize what the bundle fixes, in prose. + - List each finding addressed (id, file, severity, one-line description). + - Say which reviewer(s) flagged each one. + - Link back to the original PR (`#`). + +## Step 4 — Report back on the original PR + +Once all bundles are handled (or you've determined none are worth auto-fixing), post **one** +comment on the original PR via `gh pr comment --body "..."` containing: + +- A short overview: how many findings each reviewer produced, how many were unique after merging. +- The findings grouped by severity, each with a one-line description and which reviewer(s) raised + it. +- For findings that became a task PR: a link to that PR and its bundle number. +- For findings left unaddressed (over the 3-bundle cap, too risky, or design-level): a short note + on why, so a human knows to look at them manually. +- A closing "Run info" line (or small collapsed `
` section, so it doesn't compete with the + findings for attention): + - Who initiated the run, and the models involved — each reviewer's model (from its findings + file's `model` field) plus your own implement model, e.g. `Models: gpt (gpt-5.5) · claude + (claude-sonnet-5) · claude-code (claude-sonnet-5) · implement (claude-sonnet-5)`. Always + include this, independent of whether usage/cost data is available. + - Only for reviewers whose findings file carried a `usage` field, each one's token usage and + estimated cost, e.g. `gpt: 42,310 in / 1,204 out (~$0.25) · claude: 38,750 in / 980 out + (~$0.13) · claude-code: 40,100 in / 1,050 out (~$0.14)`. Omit a reviewer from this line + entirely if its file has no `usage` field; omit just the `(~$...)` part if `usage` is present + but has no `cost_usd`; don't report zeros or guess either figure. Note this covers only the + review calls — the separate implement/triage step + you're running right now posts its own exact cost (tracked by the Claude Code CLI itself) as + a follow-up comment after you finish, so don't try to estimate or include that cost yourself. + +Keep the comment skimmable — headings and bullet points, not a wall of prose. This comment is the +single source of truth for what happened during this review run. diff --git a/.claude/skills/review-response-format/SKILL.md b/.claude/skills/review-response-format/SKILL.md new file mode 100644 index 0000000..bc366cd --- /dev/null +++ b/.claude/skills/review-response-format/SKILL.md @@ -0,0 +1,62 @@ +--- +name: review-response-format +description: Shared response contract given to every model (currently GPT, Claude, and Claude Code) performing a parallel PR review, so their findings are directly comparable and mergeable downstream. +--- + +# Review Response Format + +You are one of several independent reviewers examining the same pull request diff. Another +process will merge your findings with the other reviewers' findings, so your response MUST be +machine-parseable and MUST follow this exact contract. Do not add commentary outside the JSON. + +## Output contract + +Respond with **only** a single JSON object, no markdown code fences, no leading/trailing prose: + +```json +{ + "reviewer": "", + "model": "", + "summary": "1-3 sentences: overall risk/quality assessment of this diff.", + "findings": [ + { + "id": "kebab-case-short-slug", + "file": "relative/path/as/shown/in/the/diff", + "line": 123, + "severity": "critical|high|medium|low", + "category": "bug|security|performance|reliability|test-coverage|maintainability|style", + "title": "one-line summary of the issue", + "description": "what is wrong and why it matters, 1-4 sentences", + "suggested_fix": "a concrete fix: what to change, 1-4 sentences or a short code sketch" + } + ] +} +``` + +If you find nothing worth reporting, return `"findings": []` with a summary saying so — do not +invent issues to have something to say. + +## Reviewing guidelines + +- Only cite `file`/`line` values that actually appear in the diff you were given. Never guess a + line number for a hunk you can't see. +- Prioritize correctness bugs, security issues, data loss/corruption risks, and reliability + problems over style. Only report `style`/`maintainability` findings that are clear-cut, not + matters of taste. +- Each finding should be independently actionable — something a downstream engineer (or agent) + could fix without needing to ask you a follow-up question. Vague findings ("this could be + cleaner") are not useful; be specific about what and why. +- `severity` reflects user/production impact, not how much you personally dislike the code: + - `critical`: data loss, security vulnerability, breaks core functionality + - `high`: a real bug in a common path, or a serious security/performance issue in an edge case + - `medium`: a bug in an uncommon path, or a moderate reliability/performance concern + - `low`: style, minor maintainability, nice-to-have +- Deduplicate within your own response — don't list the same underlying issue twice because it + recurs in several files; instead pick the clearest instance and mention in the description that + it recurs elsewhere. +- `id` should be a short, stable, kebab-case slug describing the issue (e.g. + `null-check-missing-auth-header`) so it can be matched against the same finding reported by + another reviewer. +- Keep `description` and `suggested_fix` terse. This is going to be read by another model doing + triage across three reviewers' worth of findings, not a human reading prose — density matters + more than tone. diff --git a/.github/scripts/call-review-model.mjs b/.github/scripts/call-review-model.mjs new file mode 100644 index 0000000..38d3eef --- /dev/null +++ b/.github/scripts/call-review-model.mjs @@ -0,0 +1,324 @@ +#!/usr/bin/env node +// Calls one reviewer model (GPT, Grok, Claude via the raw Messages API, or +// Claude Code via the `claude` CLI) with the shared review-response-format +// skill as its system prompt, so all reviewers return findings in the same +// shape. Used by .github/workflows/pr-multi-review.yml. + +import { readFileSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; + +function parseArgs(argv) { + const out = {}; + for (let i = 0; i < argv.length; i += 2) { + const key = argv[i]?.replace(/^--/, ""); + if (!key) continue; + out[key] = argv[i + 1]; + } + return out; +} + +const args = parseArgs(process.argv.slice(2)); +const provider = args.provider; +if (!["gpt", "grok", "claude", "claude-code"].includes(provider)) { + console.error( + `--provider must be one of gpt|grok|claude|claude-code, got: ${provider}`, + ); + process.exit(1); +} + +const MAX_DIFF_CHARS = 150_000; + +const skillRaw = readFileSync(args["skill-file"], "utf8"); +// Strip the YAML frontmatter, keep the instructions body. +const instructions = skillRaw + .replace(/^---\n[\s\S]*?\n---\n/, "") + .trim(); + +let diff = readFileSync(args["diff-file"], "utf8"); +let truncated = false; +if (diff.length > MAX_DIFF_CHARS) { + diff = diff.slice(0, MAX_DIFF_CHARS); + truncated = true; +} + +const prNumber = args["pr-number"] ?? ""; +const prTitle = args["pr-title"] ?? ""; + +const userPrompt = [ + `You are reviewing pull request #${prNumber}: "${prTitle}".`, + `Respond as reviewer "${provider}".`, + truncated + ? "NOTE: the diff below was truncated to fit a size limit; review only what is shown." + : "", + "Unified diff:", + "```diff", + diff, + "```", +] + .filter(Boolean) + .join("\n\n"); + +function extractJson(text) { + let t = text.trim(); + const fence = t.match(/```(?:json)?\s*([\s\S]*?)```/i); + if (fence) t = fence[1].trim(); + try { + return { ok: true, value: JSON.parse(t) }; + } catch (err) { + return { ok: false, raw: text, error: String(err) }; + } +} + +// $ per 1M tokens (list price, no cache/batch discounts applied). Anthropic +// figures are from https://platform.claude.com/docs/en/pricing; the GPT-5.5 +// figures are from third-party trackers (OpenAI doesn't publish a stable +// pricing page for it) and may drift — update this table when the actual +// billed rate changes. A model missing here simply gets no cost_usd field. +const PRICING_PER_MTOK = { + "gpt-5.5": { input: 5.0, output: 30.0 }, + "claude-sonnet-5": { input: 3.0, output: 15.0 }, + "claude-opus-5": { input: 5.0, output: 25.0 }, +}; + +function estimateCostUsd(model, usage) { + if (!usage) return undefined; + const pricing = PRICING_PER_MTOK[model]; + if (!pricing) return undefined; + return ( + (usage.input_tokens / 1_000_000) * pricing.input + + (usage.output_tokens / 1_000_000) * pricing.output + ); +} + +async function callWithRetry(fn, attempts = 3) { + let lastErr; + for (let i = 0; i < attempts; i++) { + try { + return await fn(); + } catch (err) { + lastErr = err; + if (i < attempts - 1) + await new Promise((r) => + setTimeout(r, 2000 * (i + 1)), + ); + } + } + throw lastErr; +} + +// Normalizes each provider's usage shape to a common { input_tokens, output_tokens, +// total_tokens } so the triage step can report token usage the same way for every reviewer. +function normalizeOpenAiUsage(usage) { + if (!usage) return undefined; + return { + input_tokens: usage.prompt_tokens ?? 0, + output_tokens: usage.completion_tokens ?? 0, + total_tokens: + usage.total_tokens ?? + (usage.prompt_tokens ?? 0) + (usage.completion_tokens ?? 0), + }; +} + +async function callGpt() { + const model = process.env.GPT_MODEL || "gpt-5.5"; + const res = await fetch( + "https://api.openai.com/v1/chat/completions", + { + method: "POST", + headers: { + "content-type": "application/json", + authorization: `Bearer ${process.env.OPENAI_API_KEY}`, + }, + body: JSON.stringify({ + model, + response_format: { type: "json_object" }, + messages: [ + { role: "system", content: instructions }, + { role: "user", content: userPrompt }, + ], + }), + }, + ); + if (!res.ok) + throw new Error( + `OpenAI API error ${res.status}: ${await res.text()}`, + ); + const body = await res.json(); + return { + model, + text: body.choices?.[0]?.message?.content ?? "", + usage: normalizeOpenAiUsage(body.usage), + }; +} + +async function callGrok() { + const model = process.env.GROK_MODEL || "grok-4"; + const res = await fetch( + "https://api.x.ai/v1/chat/completions", + { + method: "POST", + headers: { + "content-type": "application/json", + authorization: `Bearer ${process.env.XAI_API_KEY}`, + }, + body: JSON.stringify({ + model, + messages: [ + { role: "system", content: instructions }, + { + role: "user", + content: `${userPrompt}\n\nRespond with ONLY the JSON object described above — no markdown fences, no prose.`, + }, + ], + }), + }, + ); + if (!res.ok) + throw new Error( + `xAI API error ${res.status}: ${await res.text()}`, + ); + const body = await res.json(); + return { + model, + text: body.choices?.[0]?.message?.content ?? "", + usage: normalizeOpenAiUsage(body.usage), + }; +} + +async function callClaude() { + const model = + process.env.CLAUDE_REVIEW_MODEL || "claude-opus-5"; + const res = await fetch( + "https://api.anthropic.com/v1/messages", + { + method: "POST", + headers: { + "content-type": "application/json", + "x-api-key": process.env.ANTHROPIC_API_KEY, + "anthropic-version": "2023-06-01", + }, + body: JSON.stringify({ + model, + max_tokens: 8000, + system: instructions, + messages: [ + { + role: "user", + content: `${userPrompt}\n\nRespond with ONLY the JSON object described above — no markdown fences, no prose.`, + }, + ], + }), + }, + ); + if (!res.ok) + throw new Error( + `Anthropic API error ${res.status}: ${await res.text()}`, + ); + const body = await res.json(); + const text = (body.content ?? []) + .map((b) => b.text ?? "") + .join(""); + const usage = body.usage + ? { + input_tokens: body.usage.input_tokens ?? 0, + output_tokens: body.usage.output_tokens ?? 0, + total_tokens: + (body.usage.input_tokens ?? 0) + + (body.usage.output_tokens ?? 0), + } + : undefined; + return { model, text, usage }; +} + +// Reviews via the `claude` CLI itself rather than a raw API call, so it bills against +// CLAUDE_CODE_OAUTH_TOKEN's Claude Code subscription instead of a separate ANTHROPIC_API_KEY. +// --tools "" disables all tool access — this is a plain one-shot completion, same as the other +// reviewers, not an agentic run — and --system-prompt replaces (not appends to) Claude Code's own +// default system prompt so the reviewer doesn't inherit unrelated agentic-tool framing. +function callClaudeCode() { + const model = + process.env.CLAUDE_CODE_REVIEW_MODEL || "claude-sonnet-5"; + const res = spawnSync( + "claude", + [ + "-p", + userPrompt, + "--system-prompt", + instructions, + "--model", + model, + "--output-format", + "json", + "--tools", + "", + "--no-session-persistence", + ], + { encoding: "utf8", maxBuffer: 20 * 1024 * 1024 }, + ); + if (res.error) { + throw new Error(`Failed to spawn claude CLI: ${res.error.message}`); + } + if (res.status !== 0) { + throw new Error( + `claude CLI exited ${res.status}: ${res.stderr || res.stdout}`, + ); + } + const parsed = JSON.parse(res.stdout); + if (parsed.is_error) { + throw new Error( + `claude CLI reported an error: ${parsed.result ?? JSON.stringify(parsed)}`, + ); + } + // The CLI tracks its own exact spend — no pricing-table estimate needed, unlike the raw-API + // reviewers above. + const usage = parsed.usage + ? { + input_tokens: parsed.usage.input_tokens ?? 0, + output_tokens: parsed.usage.output_tokens ?? 0, + total_tokens: + (parsed.usage.input_tokens ?? 0) + + (parsed.usage.output_tokens ?? 0), + cost_usd: parsed.total_cost_usd, + } + : undefined; + return { model, text: parsed.result ?? "", usage }; +} + +const callers = { + gpt: callGpt, + grok: callGrok, + claude: callClaude, + "claude-code": callClaudeCode, +}; + +const { model, text, usage } = await callWithRetry(() => + callers[provider](), +); +// Only estimate cost when the caller didn't already supply an exact figure (callClaudeCode sets +// usage.cost_usd itself from the CLI's own tracked spend). +if (usage && usage.cost_usd === undefined) { + const costUsd = estimateCostUsd(model, usage); + if (costUsd !== undefined) usage.cost_usd = costUsd; +} +const parsed = extractJson(text); + +const result = parsed.ok + ? // `reviewer`/`model`/`usage` come after the spread so our own authoritative values (what we + // actually requested and measured) always win over parsed.value's copies — models frequently + // misreport their own id/reviewer name in the response text itself. + { ...parsed.value, reviewer: provider, model, usage } + : { + reviewer: provider, + model, + usage, + summary: + "Reviewer response was not valid JSON; see parse_error_raw.", + findings: [], + parse_error: true, + parse_error_raw: parsed.raw, + }; + +writeFileSync(args.out, JSON.stringify(result, null, 2)); +console.log( + `Wrote ${args.out} (${result.findings?.length ?? 0} findings, parse_error=${!!result.parse_error})`, +); diff --git a/.github/workflows/pr-multi-review.yml b/.github/workflows/pr-multi-review.yml new file mode 100644 index 0000000..33b400e --- /dev/null +++ b/.github/workflows/pr-multi-review.yml @@ -0,0 +1,432 @@ +name: Multi-Model PR Review +run-name: Review PR #${{ inputs.pr_number }} + +# Reviews an open PR with GPT, Claude (raw Anthropic API), and Claude Code (the `claude` CLI +# itself, billed against a Claude Code subscription instead of API credits) in parallel — all +# three follow the shared `.claude/skills/review-response-format` contract so their findings merge +# cleanly; Grok is left out for now — add a review-grok job mirroring review-gpt/review-claude-api, +# plus an XAI_API_KEY secret, to bring it back), then has +# Claude Code triage the combined findings, bundle fixes into up to 3 `task/*` branches (each +# touching < 10 files), open a PR per bundle against the reviewed PR's branch, and post a summary +# comment on the original PR. See `.claude/skills/pr-review-process` for the full +# triage/bundling/reporting procedure Claude follows. +# +# Can be run two ways: +# 1. Directly in this repo, via workflow_dispatch (Actions tab > Multi-Model PR Review > Run). +# 2. As a reusable workflow called from another repo's own workflow, to review PRs over there: +# +# name: Review PR +# on: +# workflow_dispatch: +# inputs: +# pr_number: +# required: true +# type: string +# jobs: +# review: +# uses: bcgov/aps-devops/.github/workflows/pr-multi-review.yml@main +# with: +# pr_number: ${{ inputs.pr_number }} +# source_ref: main # optional; which ref of bcgov/aps-devops to pull the shared +# # scripts/skills from — defaults to main, override to pin a +# # branch/tag/sha (e.g. while this workflow is still in review) +# review_claude_api: true # optional, defaults to false (skipped); set true to also +# # run the ANTHROPIC_API_KEY-billed review-claude-api job alongside +# # review-gpt and review-claude-code (the CLAUDE_CODE_OAUTH_TOKEN- +# # billed reviewer) — needs an actual API credit balance to spend +# secrets: inherit +# permissions: +# contents: write +# pull-requests: write +# issues: write +# +# The shared scripts and skills are always pulled from bcgov/aps-devops (this repo) at the +# `source_ref` input above, independent of which ref of *this* file the caller's `uses:` +# line is pinned to — so the calling repo only needs the one small wrapper workflow above, +# nothing else copied over. This relies on bcgov/aps-devops staying public; if it's ever +# made private, callers will additionally need to supply a token with read access to it (see +# the "Checkout shared" steps below). +# +# One-time setup required: +# - Secrets ANTHROPIC_API_KEY, OPENAI_API_KEY, available either as repo/org secrets (direct +# workflow_dispatch use) or passed via `secrets: inherit` / explicit mapping from the calling +# repo (reusable-workflow use). ANTHROPIC_API_KEY is used directly for review-claude-api's raw API +# call, and as the review-claude-code / triage fallback below either way. +# - Optional secret CLAUDE_CODE_OAUTH_TOKEN (a long-lived token from running `claude +# setup-token` under a Pro/Max subscription) — if set, both the review-claude-code job and the +# final triage/implement step use it. The `claude` CLI does NOT prefer the OAuth token when +# both credentials are present in its environment — a set ANTHROPIC_API_KEY silently +# overrides a working subscription login (confirmed directly against the CLI) — so both of +# those steps only pass ANTHROPIC_API_KEY through when CLAUDE_CODE_OAUTH_TOKEN is unset. +# - Settings > Actions > General > Workflow permissions on whichever repo is being reviewed: +# "Allow GitHub Actions to create and approve pull requests" must be enabled, since the +# final job pushes task/* branches and opens PRs using the default GITHUB_TOKEN. + +on: + workflow_dispatch: + inputs: + pr_number: + description: "Pull request number to review" + required: true + type: string + source_ref: + description: "Ref of bcgov/aps-devops to pull the shared review scripts/skills from" + required: false + type: string + default: pr-multi-review + review_claude_api: + description: "Also run review-claude-api (raw Anthropic API call, billed via ANTHROPIC_API_KEY) alongside review-gpt and review-claude-code" + required: false + type: boolean + default: false + workflow_call: + inputs: + pr_number: + description: "Pull request number to review" + required: true + type: string + source_ref: + description: "Ref of bcgov/aps-devops to pull the shared review scripts/skills from" + required: false + type: string + default: pr-multi-review + review_claude_api: + description: "Also run review-claude-api (raw Anthropic API call, billed via ANTHROPIC_API_KEY) alongside review-gpt and review-claude-code" + required: false + type: boolean + default: false + secrets: + ANTHROPIC_API_KEY: + required: true + OPENAI_API_KEY: + required: true + CLAUDE_CODE_OAUTH_TOKEN: + required: false + +permissions: + contents: read + +env: + GPT_MODEL: gpt-5.5 + CLAUDE_REVIEW_MODEL: claude-sonnet-5 + CLAUDE_CODE_REVIEW_MODEL: claude-sonnet-5 + CLAUDE_IMPLEMENT_MODEL: claude-sonnet-5 + +jobs: + prepare: + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + outputs: + head_branch: ${{ steps.meta.outputs.head_branch }} + base_branch: ${{ steps.meta.outputs.base_branch }} + pr_title: ${{ steps.meta.outputs.pr_title }} + pr_url: ${{ steps.meta.outputs.pr_url }} + # Hardcoded rather than derived from `github.repository`: in the reusable-workflow-call + # case, `github.repository` resolves to the *caller's* repo, not this one, but the shared + # scripts/skills below always live in bcgov/aps-devops regardless of who calls this workflow. + source_repo: bcgov/aps-devops + steps: + - name: Resolve PR metadata and diff + id: meta + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + PR_NUMBER: ${{ inputs.pr_number }} + run: | + set -euo pipefail + data=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json baseRefName,headRefName,title,url) + echo "head_branch=$(echo "$data" | jq -r .headRefName)" >> "$GITHUB_OUTPUT" + echo "base_branch=$(echo "$data" | jq -r .baseRefName)" >> "$GITHUB_OUTPUT" + { + echo "pr_title<> "$GITHUB_OUTPUT" + echo "pr_url=$(echo "$data" | jq -r .url)" >> "$GITHUB_OUTPUT" + gh pr diff "$PR_NUMBER" --repo "$REPO" > pr.diff + - uses: actions/upload-artifact@v7 + with: + name: pr-diff + path: pr.diff + retention-days: 1 + + review-gpt: + needs: prepare + runs-on: ubuntu-latest + steps: + - name: Checkout shared scripts and skill (workflow's own repo) + uses: actions/checkout@v7 + with: + repository: ${{ needs.prepare.outputs.source_repo }} + ref: ${{ inputs.source_ref }} + sparse-checkout: | + .github/scripts + .claude/skills/review-response-format + - uses: actions/download-artifact@v8 + with: + name: pr-diff + - name: Call GPT + env: + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + run: | + node .github/scripts/call-review-model.mjs \ + --provider gpt \ + --diff-file pr.diff \ + --skill-file .claude/skills/review-response-format/SKILL.md \ + --pr-number "${{ inputs.pr_number }}" \ + --pr-title "${{ needs.prepare.outputs.pr_title }}" \ + --out gpt.json + - uses: actions/upload-artifact@v7 + with: + name: review-gpt + path: gpt.json + retention-days: 1 + + review-claude-api: + needs: prepare + if: ${{ inputs.review_claude_api }} + runs-on: ubuntu-latest + steps: + - name: Checkout shared scripts and skill (workflow's own repo) + uses: actions/checkout@v7 + with: + repository: ${{ needs.prepare.outputs.source_repo }} + ref: ${{ inputs.source_ref }} + sparse-checkout: | + .github/scripts + .claude/skills/review-response-format + - uses: actions/download-artifact@v8 + with: + name: pr-diff + - name: Call Claude + env: + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + run: | + node .github/scripts/call-review-model.mjs \ + --provider claude \ + --diff-file pr.diff \ + --skill-file .claude/skills/review-response-format/SKILL.md \ + --pr-number "${{ inputs.pr_number }}" \ + --pr-title "${{ needs.prepare.outputs.pr_title }}" \ + --out claude.json + - uses: actions/upload-artifact@v7 + with: + name: review-claude-api + path: claude.json + retention-days: 1 + + review-claude-code: + needs: prepare + runs-on: ubuntu-latest + steps: + - name: Checkout shared scripts and skill (workflow's own repo) + uses: actions/checkout@v7 + with: + repository: ${{ needs.prepare.outputs.source_repo }} + ref: ${{ inputs.source_ref }} + sparse-checkout: | + .github/scripts + .claude/skills/review-response-format + - uses: actions/download-artifact@v8 + with: + name: pr-diff + - name: Install claude CLI + run: npm install -g @anthropic-ai/claude-code + - name: Call Claude Code + # Reviews via the `claude` CLI itself (see call-review-model.mjs's callClaudeCode) rather + # than a raw API call, so this bills against CLAUDE_CODE_OAUTH_TOKEN's subscription when + # set, instead of always drawing on ANTHROPIC_API_KEY like review-claude-api above. + # + # The CLI does NOT prefer the OAuth token when both credentials are present in its + # environment — an ANTHROPIC_API_KEY, even an invalid one, silently takes precedence over + # a working claude.ai/subscription login (confirmed against the CLI directly: it logs + # "claude.ai connectors are disabled because ANTHROPIC_API_KEY ... takes precedence", then + # bills the API key, which can fail with "Credit balance is too low" on an account that + # relies on subscription usage rather than pay-as-you-go API credits). So ANTHROPIC_API_KEY + # must only be present in this step's environment when there's no OAuth token to use — + # never pass both unconditionally. + env: + CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + ANTHROPIC_API_KEY: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN == '' && secrets.ANTHROPIC_API_KEY || '' }} + run: | + node .github/scripts/call-review-model.mjs \ + --provider claude-code \ + --diff-file pr.diff \ + --skill-file .claude/skills/review-response-format/SKILL.md \ + --pr-number "${{ inputs.pr_number }}" \ + --pr-title "${{ needs.prepare.outputs.pr_title }}" \ + --out claude-code.json + - uses: actions/upload-artifact@v7 + with: + name: review-claude-code + path: claude-code.json + retention-days: 1 + + synthesize-and-implement: + needs: + [ + prepare, + review-gpt, + review-claude-api, + review-claude-code, + ] + # review-claude-api can be skipped (review_claude_api: false) rather than succeed or fail, and a + # skipped dependency does not satisfy the implicit default `if: success()` — so this needs an + # explicit condition that tolerates a skip but still blocks on a genuine failure/cancellation + # anywhere in the dependency chain (prepare, review-gpt, review-claude-code included). + if: ${{ !failure() && !cancelled() }} + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + issues: write + steps: + - name: Checkout PR branch (repo being reviewed) + uses: actions/checkout@v7 + with: + ref: ${{ needs.prepare.outputs.head_branch }} + fetch-depth: 0 + + - name: Checkout shared skills (workflow's own repo) + uses: actions/checkout@v7 + with: + repository: ${{ needs.prepare.outputs.source_repo }} + ref: ${{ inputs.source_ref }} + sparse-checkout: | + .claude/skills/pr-review-process + .claude/skills/review-response-format + path: _shared-skills + + - name: Overlay shared skills into the PR checkout + # Claude Code's Skill tool discovers project skills under ./.claude/skills relative to + # cwd, so the two skills need to physically exist inside the PR repo's checkout for this + # run — not committed there, just present on disk. Excluded via .git/info/exclude (a + # local-only ignore, never committed) so an incautious `git add -A` while implementing a + # bundle can't accidentally sweep them into a task branch. + run: | + set -euo pipefail + mkdir -p .claude/skills + cp -r _shared-skills/.claude/skills/pr-review-process .claude/skills/ + cp -r _shared-skills/.claude/skills/review-response-format .claude/skills/ + rm -rf _shared-skills + { + echo "/.claude/skills/pr-review-process/" + echo "/.claude/skills/review-response-format/" + } >> .git/info/exclude + + - uses: actions/download-artifact@v8 + with: + name: review-gpt + path: reviews + - uses: actions/download-artifact@v8 + # review-claude-api never uploads this artifact when it was skipped via review_claude_api: + # false — leave reviews/claude.json absent rather than failing the download. The skill's + # Step 4 procedure already treats a missing findings file the same as a parse_error one. + if: ${{ inputs.review_claude_api }} + with: + name: review-claude-api + path: reviews + - uses: actions/download-artifact@v8 + with: + name: review-claude-code + path: reviews + + - name: Configure git identity + run: | + git config user.name "claude-pr-review-bot" + git config user.email "claude-pr-review-bot@users.noreply.github.com" + + - name: Triage findings, implement, and open task PRs + id: triage + uses: anthropics/claude-code-action@v1 + env: + GH_TOKEN: ${{ github.token }} + with: + # See the "Call Claude Code" step's comment in review-claude-code above: the CLI does + # NOT prefer the OAuth token when both credentials are present — a set ANTHROPIC_API_KEY + # silently overrides a working subscription login — so it must only be passed through + # when there's no OAuth token to use. + anthropic_api_key: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN == '' && secrets.ANTHROPIC_API_KEY || '' }} + claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + github_token: ${{ github.token }} + claude_args: >- + --allowedTools "Bash,Read,Write,Edit,Glob,Grep" + --max-turns 100 + --model ${{ env.CLAUDE_IMPLEMENT_MODEL }} + prompt: | + First, use the Skill tool to load the "pr-review-process" skill, then follow its + procedure exactly. It references the "review-response-format" skill for the shape of + the findings files below. + + Context for this run: + - Repo: ${{ github.repository }} + - Original PR number: ${{ inputs.pr_number }} + - Original PR title: ${{ needs.prepare.outputs.pr_title }} + - Original PR head branch (task branches must branch from here): ${{ needs.prepare.outputs.head_branch }} + - Original PR base branch: ${{ needs.prepare.outputs.base_branch }} + - Initiated by: @${{ github.triggering_actor }} + - Implement model (the model you are running as right now): ${{ env.CLAUDE_IMPLEMENT_MODEL }} + - Findings files (one per reviewer, each per the review-response-format contract): + - reviews/gpt.json + - reviews/claude.json + - reviews/claude-code.json + + You are already on a checkout of the head branch with full history. `gh` is + authenticated via GH_TOKEN. Follow the skill's steps: merge and rank findings, bundle + into at most 3 task branches (each touching fewer than 10 files), implement each + bundle, open a PR per bundle against the head branch listed above, and finish by + posting one summary comment on the original PR (#${{ inputs.pr_number }}). + + When staging changes, always `git add` the specific files you intentionally edited — + never `git add -A` or `git add .`. The working tree has files unrelated to any bundle + (including this run's own skill files) that must never be committed. + + - name: Label PR as reviewed + # Default (no `if:`) — only runs once the triage step above succeeds, so the label means + # the review/triage process actually completed rather than just "was attempted". + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + # --force makes this idempotent: creates the label on first use, or is a no-op (aside + # from re-applying description/color) if it already exists — never errors either way. + gh label create "pr-multi-reviewed" \ + --description "Reviewed by the Multi-Model PR Review workflow" \ + --color "0E8A16" \ + --force + gh pr edit "${{ inputs.pr_number }}" --add-label "pr-multi-reviewed" + + - name: Report implement-step cost + # Runs even if the triage step above failed partway through, so a partial run's cost is + # still visible. Never fails the job — this is a best-effort addendum, not core to the + # workflow's outcome. + if: always() + continue-on-error: true + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + execution_file="${{ steps.triage.outputs.execution_file }}" + if [ -z "$execution_file" ] || [ ! -f "$execution_file" ]; then + echo "No execution file produced by the triage step; skipping cost report." + exit 0 + fi + # The claude-code-action / Claude Code CLI tracks exact spend internally and emits it + # on the final SDK "result" message — no pricing table needed here, unlike the + # review-gpt/review-claude-api cost estimates. + result=$(jq -c '[.[] | select(.type == "result")] | last // empty' "$execution_file") + if [ -z "$result" ]; then + echo "No result message in execution file; skipping cost report." + exit 0 + fi + cost=$(echo "$result" | jq -r '.total_cost_usd // empty') + if [ -z "$cost" ]; then + echo "No total_cost_usd in result message; skipping cost report." + exit 0 + fi + turns=$(echo "$result" | jq -r '.num_turns // "?"') + duration_s=$(echo "$result" | jq -r '((.duration_ms // 0) / 1000 | floor)') + cost_fmt=$(printf '%.4f' "$cost") + gh pr comment "${{ inputs.pr_number }}" --body "**Implement step cost:** \$${cost_fmt} · ${turns} turns · ${duration_s}s (tracked by the Claude Code CLI)"