Conversation
ThomasK33
added this pull request to stack #4326
September 22, 2026 14:55
This comment has been minimized.
This comment has been minimized.
ThomasK33
force-pushed
the
thomask33/eval-l6-screening-docs
branch
from
September 22, 2026 15:04
ae0edda to
6bda6f0
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae0eddaeee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: ae0eddaeee
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…te() docs match the contract Codex round 1 on #4333: - screen-github-issue.js: the labeling agent reports through a schema ({ labeled, detail? }); when the label was not applied the run fails with a fixed-template error instead of claiming success. Test covers the schema on the spec and the failure path (red against the previous example). - workflow-authoring.md: state accepts string | JSON array | JSON object; failed admitted attempts are retryable from checkpoint (pre-admission first attempts are not); the runnable command passes --evaluation-model. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8379b672e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…capable agent Codex security review on #4333: the not_detected branch put the issue title and body into an Explore agent prompt, and Explore keeps bash and web_fetch, so a false-negative screen would have exposed a tool-capable agent to the injected text. - screen-github-issue.js: one tool-free evaluate() call answers injection, kind and severity; not_detected returns the evaluator's triage answers and starts no agent. The labeling agent still receives identifiers, the fixed label, the enumerated decision and the digest only. - workflow-authoring.md: evaluate() is tool-free classification, not a sanitizer or security boundary; the "not guaranteed" bullet names the tool grant and states this is a data-flow restriction, not host-enforced isolation. - evaluateScreening.test.ts: not_detected with a sentinel-bearing body starts no agent task; triage comes from the fake evaluator's answers. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…canonical-JSON state digest Codex round 2 on #4333 (docs accuracy): - Replay key = id + canonical JSON of state, questions, per-call model and providerOptions; title and timeoutMs are excluded (a retried step keeps the timeout it was admitted with). - state.sha256 / state.bytes digest the canonical JSON of state (sorted keys, JSON quoting/escaping), not the raw ingested text. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a66c1ae174
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…endent severity Codex round 3 on #4333 (example accuracy): - stateSha256 comment names the canonical JSON of the screened state, not raw title/body bytes. - The severity question measures user impact for every issue kind instead of a bug-only scale that was still published for feature/question/other. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0ca0c67c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
… model too Codex round 4 on #4333: --evaluation-model configures only evaluate(); the labeling agent() inherited the CLI default model, so the advertised command could not label on an OpenAI-only setup. The command now also passes --model openai:gpt-5-mini and the docs say what each flag covers. builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
Member
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82b01a8ecf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…requires the review label Codex round 5 on #4333: - Header comment no longer claims that no agent reads the issue text; it says the text is never placed in an agent prompt and that the labeling agent keeps its tools ("do not read the issue" stays guidance). - The needs-human-review label must exist (--add-label does not create it): runnable instructions in docs and header add a one-time gh label create, and the constant is annotated. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c43599c2a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Member
Author
|
Review-loop status (round budget reached) — paused for maintainer direction on one gate artifact.
Stack: #4330 and #4332 are review-clean with green CI; #4329 remains on the hold described in its pause comment (its round-2 exact-head UAT evidence is posted there). |
This comment has been minimized.
This comment has been minimized.
Member
Author
|
Also minimized (RESOLVED) the Codex task-summary comment posted at 17:59Z — it is the output of the checker-fix task ( |
…ng example Document the evaluate() primitive in the workflow-authoring skill (signature, question/answer shapes, model precedence, replay and attempt semantics, failure vs interruption, the sequential-only rule and the not-guaranteed list) and ship the screening example as a skill asset runnable via skill://workflow-authoring/screen-github-issue.js, exercised through WorkflowService with fake evaluation and agent adapters. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$378.37`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=378.37 -->
…te() docs match the contract Codex round 1 on #4333: - screen-github-issue.js: the labeling agent reports through a schema ({ labeled, detail? }); when the label was not applied the run fails with a fixed-template error instead of claiming success. Test covers the schema on the spec and the failure path (red against the previous example). - workflow-authoring.md: state accepts string | JSON array | JSON object; failed admitted attempts are retryable from checkpoint (pre-admission first attempts are not); the runnable command passes --evaluation-model. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
…capable agent Codex security review on #4333: the not_detected branch put the issue title and body into an Explore agent prompt, and Explore keeps bash and web_fetch, so a false-negative screen would have exposed a tool-capable agent to the injected text. - screen-github-issue.js: one tool-free evaluate() call answers injection, kind and severity; not_detected returns the evaluator's triage answers and starts no agent. The labeling agent still receives identifiers, the fixed label, the enumerated decision and the digest only. - workflow-authoring.md: evaluate() is tool-free classification, not a sanitizer or security boundary; the "not guaranteed" bullet names the tool grant and states this is a data-flow restriction, not host-enforced isolation. - evaluateScreening.test.ts: not_detected with a sentinel-bearing body starts no agent task; triage comes from the fake evaluator's answers. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
…canonical-JSON state digest Codex round 2 on #4333 (docs accuracy): - Replay key = id + canonical JSON of state, questions, per-call model and providerOptions; title and timeoutMs are excluded (a retried step keeps the timeout it was admitted with). - state.sha256 / state.bytes digest the canonical JSON of state (sorted keys, JSON quoting/escaping), not the raw ingested text. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
…endent severity Codex round 3 on #4333 (example accuracy): - stateSha256 comment names the canonical JSON of the screened state, not raw title/body bytes. - The severity question measures user impact for every issue kind instead of a bug-only scale that was still published for feature/question/other. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
… model too Codex round 4 on #4333: --evaluation-model configures only evaluate(); the labeling agent() inherited the CLI default model, so the advertised command could not label on an OpenAI-only setup. The command now also passes --model openai:gpt-5-mini and the docs say what each flag covers. builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
…requires the review label Codex round 5 on #4333: - Header comment no longer claims that no agent reads the issue text; it says the text is never placed in an agent prompt and that the labeling agent keeps its tools ("do not read the issue" stays guidance). - The needs-human-review label must exist (--add-label does not create it): runnable instructions in docs and header add a one-time gh label create, and the constant is annotated. - builtInSkillContent.generated.ts regenerated. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
ThomasK33
force-pushed
the
thomask33/eval-l6-screening-docs
branch
from
September 22, 2026 20:02
c43599c to
acba885
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sixth and last layer of the workflow
evaluate()primitive (stacked on #4332): authoring documentation forevaluate()in the built-inworkflow-authoringskill and a runnable GitHub-issue screening example shipped as a skill asset (skill://workflow-authoring/screen-github-issue.js), exercised end to end throughWorkflowServicewith fake evaluation and agent adapters.Background
Plan layers: L2a #4321 → L2b #4322 → L3a #4324 → L3b #4329 (runner + CLI) → L4 #4330 (timeline rendering) → L5 #4332 (TypeSafe provider) → L6 (this PR). Docs live inline in the skill Markdown (no free-floating docs), and
builtInSkillContent.generated.tsis regenerated from it.Implementation
workflow-authoring.mdgains anevaluate(state, options)section: signature and acceptedstatetypes (string / JSON array / JSON object), question/answer shapes and bounds, the canonical-JSON state digest, model precedence (per-callmodel→--evaluation-model→ persistedevaluationDefaults.model; the Settings card for that default is a separate PR), supported direct routes, the explicit replay key (id+state,questions,model,providerOptions; nottitle/timeoutMs), attempt semantics includingretry_from_checkpointeligibility for admitted failures, failure vs. interruption (fixed-template error text), the sequential-only rule, and what is not guaranteed.evaluate()is described as tool-free classification, not a sanitizer or security boundary.screen-github-issue.js: a screening workflow with flat args (repo,issueNumber,title,body). One tool-freeevaluate()call answersinjection,kindandseverity(impact scale valid for every kind).not_detectedreturns the evaluator's triage answers and starts no agent — issue text is never placed in an agent prompt, not even after a false-negative screen (built-in Explore keepsbash/web_fetch). This is prompt isolation, not read isolation: the labeling agent keeps its tools, and "do not read the issue" stays guidance.suspected/uncertainhand only{ repo, issueNumber, label, reasonCode, stateSha256 }to an exec agent that applies theneeds-human-reviewlabel and reports{ labeled, detail? }through a schema; the run fails when the label was not applied. The runnable command passes--evaluation-model(forevaluate()) and--model(for the labeling agent) and creates theneeds-human-reviewlabel once per repository (--add-labeldoes not create labels).evaluateScreening.test.ts(7): runs the example viaWorkflowServicewith fake evaluation and agent adapters —not_detectedwith a sentinel-bearing body starts no agent task and echoes only triage + digest;suspected/uncertainroute to the labeling agent with identifiers and digest only (never the text) and alabeledschema; a labeling agent reportinglabeled: falsefails the run (no result event, fixed-template error without body bytes); an evaluator failure fails the run before any agent step; resuming after an interrupt in the labeling step replays the evaluation without re-dispatching; a malformedrepois rejected before evaluating.Validation
evaluateScreening.test.ts7/7 (labeling-failure case red against the previous example);agentSkillssuites green after regeneration;make static-checkgreen (includes docs link check).Follow-ups (non-blocking review findings deferred at the round budget)
needs-human-reviewonly when absent (or pin--color) so a re-run of the advertisedgh label create --forcecannot recolor an existing label.scoreanswers as a continuous value in[0, levels − 1](probability-weighted mean when a distribution is returned; an integer only when the provider selects an exact level) and warn against indexing/switching on it.Risks
Generated with
xum• Model:coder:anthropic/claude-fable-5-1• Thinking:xhigh• Cost:$537.54