Skip to content

ci: run the AI reviews from actions v2.0.0 - #2312

Merged
JuArce merged 2 commits into
testnetfrom
ci/ai-review-v2
Nov 6, 2026
Merged

JuArce merged 2 commits into
testnetfrom
ci/ai-review-v2

Conversation

@MarcosNicolau

Copy link
Copy Markdown
Collaborator
  • Replace pr_review_claude.yml and pr_review_codex.yml (yetanotherco/actions v1.0.1) with one .github/workflows/ai-review.yml, the same file as yetanotherco/waas#194: six reviewers at 4a4279fdc553ad030894c98698582baaeb40327a # v2.0.0, Claude, Codex, Kimi, and DeepSeek, Grok and Gemini through OpenRouter. v2.0.0 pins the external actions it uses to commit SHAs, keeps write-capable tokens away from the review agents and runs Claude on Opus 5.5.
  • Reviews run when a same-repo PR is opened, reopened or marked ready for review, skipping drafts. /review runs all six; /claude, /codex, /kimi, /deepseek, /grok or /gemini runs one, from an OWNER, MEMBER or COLLABORATOR. Before, drafts were reviewed when opened and /claude anywhere in a comment triggered a review.
  • Each job gets only the permissions its reusable workflow needs instead of the repository's default token permissions.
  • Move the review prompt, unchanged, into .github/prompts/ai-review.md. v2.0.0 reads it from the base branch, so a PR can't change the prompt that reviews it; this PR is reviewed with the default prompt.
  • Claude reviews cost up to about twice as much on Opus; pass model: sonnet to keep Sonnet.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

google/gemini-4-argon Code Review

Here are the issues found in the PR diff:

  1. Potential Bug (Invalid Model IDs) - .github/workflows/ai-review.yml (Lines 68, 86, 102)

    • deepseek/deepseek-v4-pro-0813 (Line 68)
    • x-ai/grok-4.7 (Line 86)
    • google/gemini-4-argon (Line 102)

    These OpenRouter model slugs do not exist and will cause the corresponding workflow jobs to fail at runtime. Verify the intended model IDs on OpenRouter (e.g., deepseek/deepseek-r1 or deepseek/deepseek-chat, x-ai/grok-2-1212, google/gemini-2.5-pro or google/gemini-2.0-pro-exp-02-05).

  2. Minor/Simplicity - .github/prompts/ai-review.md (Lines 13, 17)

    • Duplicate instruction: "Be concise and to the point" (line 13) and "Be concise and actionable" (line 17). You can remove one of them.

Automated review by google/gemini-4-argon via OpenRouter

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

deepseek/deepseek-v4-pro-0813 Code Review

The PR replaces separate AI review workflows with a consolidated workflow and adds a shared prompt file. However, there are a few issues:

  1. Unused prompt file / lost custom prompt (Medium)

    • .github/prompts/ai-review.md is added but never referenced in ai-review.yml.
    • The calls to reusable workflows (e.g., lines 18, 28, 38, 48, 58, 68) no longer pass custom_prompt via with:.
    • The deleted workflows (pr_review_claude.yml, pr_review_codex.yml) explicitly passed the same prompt. Unless the reusable workflow automatically discovers this file (unlikely), the AI reviews will use default prompts, losing the intended review guidelines.
    • Action: Either pass custom_prompt to each reusable workflow or confirm that the reusable workflow automatically reads this file.
  2. Missing synchronize trigger (Low)

    • .github/workflows/ai-review.yml lines 4–5: on: pull_request: types: [opened, reopened, ready_for_review] does not include synchronize.
    • This means reviews will not run on new commits pushed to an existing PR after it has been opened. If continuous re-review is desired, add synchronize.
  3. Inconsistent permissions (Low)

    • .github/workflows/ai-review.yml line 28 (codex-review) sets issues: write, while other jobs use issues: read or omit it entirely (e.g., line 18 has issues: read).
    • If the reusable workflow does not actually need issues: write, this grants unnecessary permissions; if it does, the other jobs may fail. Verify against the reusable workflows and apply least privilege consistently.
  4. Potential resource abuse (Low)

    • The pull_request trigger automatically runs all six AI review jobs for any PR from the same repository. While restricted to branches in the same repo (not forks), any collaborator can open multiple PRs to trigger expensive reviews without a rate limit. Consider limiting automatic runs to one model or adding concurrency limits.

No critical security vulnerabilities were found. The use of pinned action commits and permissions: {} is good practice.


Automated review by deepseek/deepseek-v4-pro-0813 via OpenRouter

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Codex Code Review

No actionable issues found in the PR diff. The pinned external reusable workflows couldn’t be fetched in this environment, so their input compatibility and behavior remain unverified.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

AI review (Claude)

I didn't find any security problems or bugs in this PR.

  • Security: Nothing new. The permissions: {} default at the top level, with narrow permissions on each job, is an improvement. Pinning the reusable workflows to a commit SHA (4a4279f…) instead of the v1.0.1 tag is a good supply-chain hardening. The fork guard on pull_request and the author_association gate on issue_comment are kept. One caveat I couldn't check: the old custom_prompt input is gone. This assumes the v2.0.0 reusable workflows read .github/prompts/ai-review.md. If they read it from the PR head checkout, a same-repo PR could change the review prompt. That's low risk because only collaborators can open same-repo PRs, but it's worth confirming that the prompt is loaded from the base ref.
  • Behavior change (intended?): The trigger changed from contains(body, '/claude') to startsWith. A comment that mentions /claude in the middle of its text will no longer trigger a review. Also, every non-draft open/reopen now starts 6 paid reviews instead of 2, and max_bytes: 2400000 is set on two of them. Please check that this cost is acceptable.
  • Simplicity (optional): The same ~6-line if: expression is copied six times, with only the command name changed. GitHub Actions now supports YAML anchors, so the shared part could live in one place. As it stands, a future fix could easily be applied to some jobs and missed in others.
  • Nit: .github/prompts/ai-review.md has "Be concise" twice (lines 13 and 17). This was carried over from the old prompt.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

x-ai/grok-4.7 Code Review

The new workflow has a few real issues. The extracted prompt looks unwired, permissions are inconsistent, and /review on a fork PR still runs secret-bearing jobs.

Bugs

  • .github/prompts/ai-review.md is never referenced. The old workflows passed this text as custom_prompt. None of the jobs in .github/workflows/ai-review.yml pass it (or a prompt path). Unless yetanotherco/actions @ 4a4279f auto-loads that exact file, reviews silently drop the security/bug criteria. Wire it with with: or drop the file.
  • Permissions don’t match across jobs (ai-review.yml). Codex has issues: write (lines 36–38); Claude has issues: read (lines 21–23); Kimi/Deepseek/Grok/Gemini have neither. With top-level permissions: {}, a missing scope is a deny. If the reusable workflows post via the issues API, those jobs fail; if they don’t, Codex’s extra write is unnecessary. Align them with what the called workflows actually need.
  • Confirm the OpenRouter model IDs (deepseek/deepseek-v4-pro-0813, x-ai/grok-4.7, google/gemini-4-argon). A bad slug fails that job on every PR.

Security

  • Medium — issue_comment + secrets on fork PRs. The same-repo check only applies to pull_request (e.g. lines 14–15). A collaborator’s /review on a fork PR still runs all six jobs and passes API keys. Safe only if the pinned workflows read the diff via the API and do not checkout/execute head code or follow model tool calls against the repo. Pinning the SHA is good; don’t move these to pull_request_target.
  • Low — third-party data exposure. Every non-draft same-repo PR sends the diff to Anthropic, OpenAI, and OpenRouter (up to max_bytes: 2400000). Fine if intentional for a public repo; worth an explicit decision if the repo is private.

Cost / simplicity

  • Opening or marking a PR ready always starts six paid reviews. /review does the same. There is no concurrency: group, so a ready-for-review event plus a comment can stack another six. If that isn’t intended, gate the extra models behind their slash commands only.
  • The same ~6-line if is copied six times. Easy to drift (command prefix vs permissions). Not worth a framework, but one shared trigger job that fans out would be simpler than six copies.

No other issues stood out in this diff. The collaborator check, fork exclusion on pull_request, empty default permissions, and SHA pin are the right direction.


Automated review by x-ai/grok-4.7 via OpenRouter

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Kimi Code Review

Issues found

  • Medium – Invalid OpenRouter model identifiers will break three review jobs
    .github/workflows/ai-review.yml:68,86,102 configures:

    • deepseek/deepseek-v4-pro-0813
    • x-ai/grok-4.7
    • google/gemini-4-argon
      These do not appear to be valid OpenRouter model IDs, so the corresponding jobs will fail with a model-not-found/invalid-model API error. Replace them with exact IDs returned by OpenRouter’s model catalog and smoke-test each job.
  • Low – Provider-specific commands accept unintended prefixes
    .github/workflows/ai-review.yml:18,33,48,62,80,96 uses checks such as startsWith(comment.body, '/claude'). Consequently, comments like /claudejunk or /gemini-test trigger reviews and consume API quota. Require either an exact command or whitespace after the command name, while still allowing arguments—for example, /claude, /claude ..., and newline/CR variants.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro

@MarcosNicolau
MarcosNicolau marked this pull request as draft November 6, 2026 15:58
@MarcosNicolau
MarcosNicolau marked this pull request as ready for review November 6, 2026 15:58
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

google/gemini-4-argon Code Review

Here is the feedback on the changes:

Potential Bugs

  1. Invalid / Non-existent Model IDs (.github/workflows/ai-review.yml)
    The OpenRouter model identifiers configured in the workflow do not appear to be valid current models and will likely cause the OpenRouter API calls to fail:

    • Line 68: model: deepseek/deepseek-v4-pro-0813 (Current DeepSeek models are deepseek/deepseek-r1 or deepseek/deepseek-chat).
    • Line 86: model: x-ai/grok-4.7 (Current Grok models are v2/v3).
    • Line 102: model: google/gemini-4-argon (Current Gemini models are 2.0/2.5).
  2. Duplicate Reviews on ready_for_review (.github/workflows/ai-review.yml, line 5)
    When a non-draft PR is opened, Github triggers the opened event. If someone converts a PR from a draft to ready, ready_for_review triggers. However, if a PR is opened as ready, converted to draft, and then marked ready_for_review again (or reopened), all 6 models will run a full review again. Verify if this is the intended behavior given the cost of running 6 models simultaneously.

Simplicity & Maintainability

  1. Redundant Guidelines (.github/prompts/ai-review.md, lines 13 & 17)
    Line 13 (- Be concise and to the point) and Line 17 (- Be concise and actionable) are repetitive. Remove line 13.

Automated review by google/gemini-4-argon via OpenRouter

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

deepseek/deepseek-v4-pro-0813 Code Review

The PR consolidates multiple AI review workflows into one file and moves the custom prompt to a separate file. Overall the approach is reasonable, but there are a few issues to address.

Issues

1. High: Custom prompt file is not used by any job

ai-review.md is added but never referenced in ai-review.yml. The old workflows explicitly passed custom_prompt, but the new jobs only pass model/max_bytes where applicable. Unless the reusable workflows from yetanotherco/actions are designed to automatically read .github/prompts/ai-review.md, the AI reviewers will ignore those guidelines.

Affected: .github/workflows/ai-review.yml (all jobs), .github/prompts/ai-review.md
Action: Verify if the reusable workflow auto-loads that file; if not, add an input (e.g., custom_prompt) to each job.

2. Medium: Inconsistent and possibly insufficient permissions

The jobs have different permissions blocks:

  • claude-review: issues: read
  • codex-review: issues: write
  • kimi-review, deepseek-review, grok-review, gemini-review: no issues permission at all

If the reusable workflows post comments via the issues API (common for PR comments), those missing issues: write will cause 403 errors. Even claude-review with issues: read may fail if it attempts to write.

Affected: .github/workflows/ai-review.yml, lines 25, 36, 47, 61, 75, 88 (approximate)
Action: Check the required permissions for each reusable workflow and align them. Typically issues: write is needed to create comments on PRs.

3. Medium: Generic /review triggers all six models

Every job includes a condition that reacts to /review (or /review , /review\n, /review\r). A single comment /review will launch all six AI reviews concurrently. This may be intentional (e.g., to get multiple opinions), but it also increases noise and API costs significantly.

Affected: .github/workflows/ai-review.yml, lines 10, 23, 33, 45, 58, 72 (conditions)
Action: Confirm this is desired. If not, restrict /review to a single model or remove it and require explicit model commands.

4. Low: Complex newline detection for commands

The expression uses fromJSON('"/review\n"') and fromJSON('"/review\r"') inside startsWith to handle commands followed by a newline. It works but is hard to read and maintain. A simpler approach could use a regex or a combination of startsWith and a substring check for the next character.

Affected: .github/workflows/ai-review.yml, lines 10, 23, 33, 45, 58, 72
Action: Consider simplifying for readability.

5. Low: Duplication of long conditions

The same multi-line if condition is copy‑pasted across six jobs. Any future change (e.g., adding a new model or adjusting the trigger) must be applied six times. While a matrix may not be directly applicable due to different reusable workflows, you could extract the condition into a composite action or use YAML anchors (if supported) to reduce duplication.

Affected: .github/workflows/ai-review.yml (multiple lines)
Action: Evaluate if a reusable block or a local composite action would improve maintainability.

Security considerations

No critical security vulnerabilities found. The workflows properly restrict triggers to non‑draft PRs from the same repository and to issue comments from trusted author associations, preventing secrets from being exposed to untrusted code. Pinning reusable workflows to a commit hash is good practice.

Summary

The main actionable items are to ensure the custom prompt is actually used, correct the permissions, and decide on the /review behavior. The other points are refinements.


Automated review by deepseek/deepseek-v4-pro-0813 via OpenRouter

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Codex Code Review

No actionable issues found in the PR diff.

The pinned reusable workflows could not be fetched in this environment, so their input compatibility, required permissions, and internal security behavior remain unverified.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

AI review (Claude)

I found no security problems or bugs in this change. The new triggers are tighter than before: drafts are skipped, a command has to start the comment instead of appearing anywhere in it, and only OWNER/MEMBER/COLLABORATOR can trigger a run. Each job now gets its own scoped permissions instead of the default token, and the reusable workflows are pinned to a commit SHA.

Minor notes, none blocking:

  • Low, cost: there's no concurrency: group, so posting /review twice, or posting /review just after a reopened event, starts duplicate runs of all six paid reviewers. Something like concurrency: { group: ai-review-${{ github.event.pull_request.number || github.event.issue.number }}-${{ github.event.comment.body || github.event.action }}, cancel-in-progress: true } would deduplicate them. Optional.
  • Simplicity: the same trigger condition is copied six times with only the /<name> prefix changed. GitHub Actions now supports YAML anchors, which could cut that down. Since the file matches yetanotherco/waas#194, keeping it identical across repos is also a reasonable choice.
  • Can't check from here: whether each job's permissions covers what its reusable workflow at 4a4279f asks for, for example Codex having issues: write while Claude has issues: read. If a job grants less than the called workflow requests, that job fails at startup. The first run on this PR will show whether they match.
  • .github/prompts/ai-review.md has "Be concise" twice (lines 14 and 17). This was carried over unchanged, so it's harmless.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

x-ai/grok-4.7 Code Review

High: same-repo pull_request jobs pass live API keys to a workflow file taken from the PR branch. The same-repo check only blocks forks. Anyone who can push a branch and open a PR can replace .github/workflows/ai-review.yml and read ANTHROPIC_API_KEY, OPENAI_API_KEY, KIMI_API_KEY, and OPENROUTER_API_KEY. There is no environment: gate. issue_comment is safe because it runs the default-branch workflow; pull_request is not. Drop secrets from the pull_request path, or trigger reviews only from issue_comment / a default-branch workflow.

Medium: comment-triggered jobs (lines 16–18 and the copies below) never check that the PR head is this repo. A collaborator’s /review on a fork PR still injects those keys into a job that reads untrusted PR content. Add the same head-repo guard unless fork reviews are intentional and the reusable workflow cannot run PR code.

Bug: permissions are inconsistent and likely wrong for some callees. codex-review has issues: write (lines 35–38); claude-review has issues: read only; the OpenRouter jobs have no issues permission. Replying to an issue_comment uses the issues API, so /kimi, /deepseek, /grok, and /gemini can 403 while Codex works. Grant the same issues: write the callee actually needs, on every job.

Bug / regression: the deleted workflows passed custom_prompt; ai-review.yml never passes that input or the new .github/prompts/ai-review.md. If pr_review_*.yml@v2 does not load that path itself, the review criteria are silently dropped. Wire the prompt in explicitly or confirm the reusable workflow reads it.

Simplicity: the same long if is copied six times. A missed edit already shows up in the permission drift. A matrix (or one shared comment-prefix check) would keep the secret and association rules in one place.


Automated review by x-ai/grok-4.7 via OpenRouter

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Kimi Code Review

  • Low – .github/workflows/ai-review.yml:18,33,48,62,80,96: The reviewer-specific commands use unrestricted prefix matching, so comments such as /claude-anything or /codexfoo will trigger paid review jobs. This is inconsistent with the delimiter-aware handling of /review. Match the complete command or require a following space/newline, as already done for /review.

No material security issues found; the reusable workflows are commit-pinned, fork PRs are excluded from automatic runs, and comment triggers are restricted to trusted associations.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro

@JuArce

JuArce commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

AI review triage

I checked each bot finding against the reusable workflows pinned at yetanotherco/actions@4a4279f. Only one applied, and it's fixed in 94d1200. The others don't hold for this setup:

Finding Applies? Why
Duplicate "Be concise" lines in ai-review.md Yes, fixed Removed "Be concise and to the point" and kept "Be concise and actionable", which matches the actions default prompt.
ai-review.md is never used because custom_prompt was dropped No Every reusable workflow loads .github/prompts/ai-review.md from the PR's base commit when custom_prompt is empty, and falls back to the actions default if it's missing. Because it reads the base commit, a PR can't change its own review prompt.
The OpenRouter model IDs don't exist No deepseek/deepseek-v4-pro-0813, x-ai/grok-4.7 and google/gemini-4-argon all posted reviews on this PR. The reviewers' model lists are out of date.
Permissions differ between jobs and will cause 403s No Each job grants exactly what the workflow it calls asks for. Codex needs issues: write for its post_feedback job. Claude asks for issues: read. Kimi and OpenRouter only need pull-requests: write. Both runs succeeded and all six jobs posted.
/review on a fork PR passes secrets No Every reusable workflow starts with a "Block fork PRs" step that exits before any secret is used.
A same-repo PR can edit the workflow and read the API keys Not new Any pull_request workflow that uses secrets has this exposure, and pushing a branch already requires write access. The deleted workflows had the same exposure.
No synchronize trigger, six reviews per PR, re-runs on draft→ready, no concurrency: group By design The old workflows also ran without synchronize. Running all six models on /review is the intended behavior.
/claude also matches /claudejunk No Only OWNER, MEMBER and COLLABORATOR can trigger it, and the previous contains check was looser.
The if: is repeated six times, so use YAML anchors No An anchor can't change the command name inside each job's string, and expressions have no regex. A gate job would start an extra runner on every PR comment. The file is also kept identical to yetanotherco/waas#194.
The diff is sent to third-party providers No The repo is public, and the OpenRouter call sets data_collection: "deny".

@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown

Review: no security issues found. Actions are pinned to a SHA, top-level permissions are {}, and each job gets least-privilege permissions. The fork guard and the author_association check are kept.

Two things to check:

  1. .github/prompts/ai-review.md isn't referenced anywhere in this workflow. Confirm that the reusable workflows at v2.0.0 read it by default. Otherwise the reviews run without the custom prompt, because the old custom_prompt input was removed.
  2. I couldn't verify the OpenRouter model IDs: deepseek/deepseek-v4-pro-0813, x-ai/grok-4.7 and google/gemini-4-argon. A wrong ID would make those jobs fail on every PR. This is non-blocking, but check they exist.

Minor: comment triggers changed from contains to startsWith, so /claude now has to be at the start of the comment. That looks intentional.

@JuArce
JuArce merged commit 9bb6a3b into testnet Nov 6, 2026
2 checks passed
@JuArce
JuArce deleted the ci/ai-review-v2 branch November 6, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants