Skip to content

ci: add AI summaries to Slack notifications - #635

Closed
zhongkechen wants to merge 1 commit into
mainfrom
feat/ai-slack-notification-summary
Closed

ci: add AI summaries to Slack notifications#635
zhongkechen wants to merge 1 commit into
mainfrom
feat/ai-slack-notification-summary

Conversation

@zhongkechen

Copy link
Copy Markdown
Contributor

Summary

  • generate concise AI summaries for pull request, issue, and release Slack notifications
  • isolate model inference from webhook-bearing delivery jobs and load support code from the immutable workflow revision
  • validate, bound, and sanitize model output with a deterministic fallback
  • add unit and workflow-structure tests to the GitHub scripts CI job

Testing

  • hatch run test:all .github/scripts/tests/test_build_lambda_layer.py .github/scripts/tests/test_notify_workflow.py .github/scripts/tests/test_parse_sdk_branch.py .github/scripts/tests/test_summarize_notification.py
  • hatch fmt --check .github/scripts/summarize_notification.py .github/scripts/tests/test_summarize_notification.py .github/scripts/tests/test_notify_workflow.py

@zhongkechen
zhongkechen temporarily deployed to ai-pr-review-runtime August 13, 2026 05:07 — with GitHub Actions Inactive
@zhongkechen
zhongkechen temporarily deployed to ai-pr-review-runtime August 13, 2026 05:07 — with GitHub Actions Inactive
Comment on lines +32 to +33
repository: ${{ job.workflow_repository }}
ref: ${{ job.workflow_sha }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude AI review

job.workflow_repository and job.workflow_sha are not valid GitHub Actions contexts — the job context only exposes container, services, and status. Both expressions therefore resolve to empty strings.

Impact: the checkout will not pin to the immutable workflow revision as the comment above claims. With an empty ref, actions/checkout falls back to the event's default ref, and an empty repository will either fail the step (making needs.summarize.result != 'success', which skips all three notify jobs and silently drops every Slack notification) or clone an unintended target. Either way the stated "same immutable revision as this workflow" guarantee is not enforced.

Fix: use the documented github contexts. For a workflow defined in this repo, github.repository is the workflow's repository and github.workflow_sha is the commit SHA of the running workflow file.

Also update the corresponding assertions in test_notify_workflow.py (they currently assert the invalid literal strings, so the test passes while the workflow is broken).

Suggested change
repository: ${{ job.workflow_repository }}
ref: ${{ job.workflow_sha }}
repository: ${{ github.repository }}
ref: ${{ github.workflow_sha }}

@github-actions

Copy link
Copy Markdown
Contributor

Claude AI review

This PR is limited to CI: a new .github/scripts/summarize_notification.py, its unit tests, a workflow-structure test, and a rewrite of .github/workflows/notify.yml. No SDK package (core, testing, otel, examples) is modified, so replay determinism, serialization, and public-API compatibility are not affected.

The Python script is generally solid — untrusted GitHub content is passed as user data (not instructions), the model ID is regex-validated, the response is size-bounded, the summary is length-bounded and sanitized (control chars, </>, and @channel|everyone|here broadcasts), and there is a deterministic fallback path. Truncation stays within MAX_SUMMARY_CHARS, and normalize_summary collapses all whitespace so the GITHUB_OUTPUT write cannot inject extra keys.

One confirmed finding (see inline comment): the summarize job checks out its toolkit using ${{ job.workflow_repository }} and ${{ job.workflow_sha }}, which are not valid GitHub Actions contexts (the job context only exposes container, services, and status). Both expressions evaluate to empty strings, so the checkout does not pin to the immutable workflow revision as the adjacent comment claims. Fix at .github/workflows/notify.yml:32-33.

Residual test risk: test_notify_workflow_generates_summary_from_immutable_toolkit asserts the literal strings ${{ job.workflow_repository }} / ${{ job.workflow_sha }}, so it enshrines the invalid contexts rather than catching them — the structural tests validate YAML shape but never that the referenced contexts resolve. There is also no coverage for main() or the empty-token fallback branch, though those are lower risk.

Reviewed commit cc49f9414d11f4cf6e2f8c58ce4bfedf47268bd9. Workflow run

uses: slackapi/slack-github-action@dcb1066f776dd043e64d0e8ba94ca15cc7e1875d # v4.0.0
with:
webhook: ${{ secrets.SLACK_WEBHOOK_URL_PR }}
webhook-type: incoming-webhook

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review

[P1] Use the webhook-trigger transport for these payloads. An incoming webhook requires text, blocks, or attachments, but all three jobs send only custom workflow variables, so Slack will reject the message as no_text or deliver nothing. Set webhook-type: webhook-trigger in all three jobs, or convert each payload into a valid incoming-webhook message, and assert the transport in the workflow test.

Comment on lines +93 to +94
normalized = normalized.replace("<", "(").replace(">", ")")
normalized = SLACK_BROADCAST_PATTERN.sub(r"(at \1)", normalized)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review

[P2] Enforce the no-link/plain-text contract in normalization. Model output may ignore the prompt, and the fallback directly includes the untrusted title. Bare URLs and Slack formatting such as *text* survive these replacements, allowing PR or issue authors to inject clickable or formatted content into Slack. Strip or neutralize URLs and Slack markup in normalize_summary, and add tests covering both model and fallback output.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Found two actionable issues, including one that prevents Slack notification delivery. No tests were executed per review constraints.

Reviewed commit cc49f9414d11f4cf6e2f8c58ce4bfedf47268bd9. Workflow run

@zhongkechen
zhongkechen deleted the feat/ai-slack-notification-summary branch August 13, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant