Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/changes/unreleased/1845-explicit-work-orders.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
kind: fixed
title: automatic task starts distinguish requests from questions
pr: 1845
surface: [chat, engine]
invalidates:
- >-
Questions and product direction could be mistaken for permission to start a
task. Both route judges now distinguish clear polite requests from
feasibility questions, ambiguous questions and future direction; a stopped
task result cannot restart the same request.
---

Clear polite requests can start work even when phrased as questions. Questions
about feasibility, ambiguous questions and future direction stay in the
conversation. A stopped task's result can be reported without starting the same
request again.
24 changes: 18 additions & 6 deletions internal/manual/chat/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -624,10 +624,19 @@ what you asked and the first two lines of the reply, and decides one thing: shou
been work? The same judge also starts reading your message **the moment you send it**,
alongside the reply rather than in front of it, which is its own section below.

**A task starts only when you clearly ask codeaf to do work now.** That can be an imperative
or a polite question whose requested action is clear: "Could you fix the failing tests?" and
"Can you implement headless screenshots and add tests?" both ask for work. A question about
feasibility, like "How would headless screenshots work?", stays in the conversation. "Can we
get screenshots without a display?" is ambiguous without context, so codeaf answers or asks
what you mean before starting. Suggestions and future direction, like "we should support
screenshots eventually", are discussion until you ask codeaf to act.

**A yes is asked twice.** The cheap model only screens — it reads every substantial message
and every wordy turn, which is why it is cheap — and it cannot start anything on its own. Before a task exists, the same
question is put once more to your **planner** model, in the same words, with none of the
first answer in front of it. Only if both say yes does the work start, and then one dim line
first answer in front of it. Both must recognize a clear request to do work now, and both
must agree the work belongs on a task. Only then does the work start, and one dim line
goes into the transcript:

```
Expand Down Expand Up @@ -680,6 +689,9 @@ which works at once when it is the only running task, and needs `alt+t` to walk
first when there are others — or `Stop` on the pointer. Nothing about it is special from the
moment it exists.

If a stopped task's result wakes the conversation, answering that result does not start the
same request again. A new instruction you type is a fresh request and can be judged normally.

**There is no setting that turns this off.** No `/settings` row switches it, and nothing you
type disarms it for the session. What bounds it is the list above, and the stop.

Expand Down Expand Up @@ -710,11 +722,11 @@ four jobs, and reading the request cannot tell them apart. The answer doing the
this moment now only decides **how soon the answer is told what it has run up**, and the
answer decides for itself from there.

**What the judge is looking for is a request whose fastest correct answer is not a
conversation:** several independent deliverables in one message, a sweep over many files or
many sources, or an answer you would otherwise sit and watch a spinner for. A question, a
discussion, one obvious edit and a few tool calls are all answered here, however large the
subject sounds.
**The judge needs both authorization and work worth handing off.** The fastest correct answer
must not be a conversation: several independent deliverables in one message, a sweep over many
files or sources, or an answer you would otherwise sit and watch a spinner for. Feasibility
questions, discussion, suggestions and product direction are answered here, however large the
subject sounds. A clear request whose fastest answer takes only a few tool calls stays here too.

**Everything that bounds the auto-start above bounds this too** — one start every three
turns across both moments, six words, two models agreeing, silence when either cannot
Expand Down
44 changes: 42 additions & 2 deletions internal/session/route_judge.go
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,7 @@ type routeVerdict struct {
// is a condition nobody can check (see [routeVerdict.Acceptance]).
const routeJudgeBrief = `You judge ONE turn of a coding assistant, after the fact. The assistant answered the person in WORDS ALONE — it called no tool. You decide one thing: should that turn have been WORK?

WORDS are a question, a discussion, advice, an opinion, a fact, an explanation, a plan somebody asked to read. SMALL WORK is words too, for this purpose: a few tool calls, one obvious edit, a file read and an answer. Handing small work off is slower than doing it, so it is not work.
WORDS include questions about feasibility, discussion, advice, opinions, facts, explanations and plans somebody asked to read. A polite question that clearly asks you to act now can still authorize work, as the work rule below explains. SMALL WORK is words too, for this purpose: a few tool calls, one obvious edit, a file read and an answer. Handing small work off is slower than doing it, so it is not work.

WORK is research across several sources, changes across several files, a goal with several independent parts, or anything the person would otherwise watch a spinner for.

Expand All @@ -309,6 +309,18 @@ or

{"work": true, "wide": true, "goal": "...", "acceptance": "...", "checks": ["..."], "why": "..."}

work true only when the person clearly asks codeaf to do work now. This can
be an imperative or a polite question whose intended action is clear:
"Could you fix the failing tests?" and "Can you implement headless
screenshots and add tests?" both authorize work. A question about
feasibility or explanation, such as "How would headless screenshots
work?", does not. "Can we get screenshots without a display?" is
ambiguous without context, so answer or clarify; do not start work.
"We should support screenshots eventually" is future direction, not
authorization. Judge intended action and timing, not grammar. Product
direction, suggestions, goals and plans do not authorize work unless
the person clearly asks you to act now. Judge authorization before
effort: a useful or lengthy job is not permission.
wide true when the work is BROAD — many files, many sources, several
independent parts — so the one worker that starts on it is allowed to
hand the parts out once it has opened the material. Leave it out for
Expand Down Expand Up @@ -418,6 +430,11 @@ func (a *Agent) readRouteJudge(ctx context.Context, user userMessage, usedTools
}
asked = steward.Ask()
}
// A stopped task's result wakes a turn so the session can report what
// happened. That reply is not a fresh instruction to start the same work.
if user.wake && a.stoppedTaskResultFor(asked) {
return judgeRuling{}
}
// THE FLOOR IS THE ASK, and it is cheaper than a model call. A commit,
// an undo, a one-line edit or a single read is answered here; paying a
// judge to be told it is work is how F26's commit became a task.
Expand Down Expand Up @@ -469,6 +486,29 @@ func (a *Agent) readRouteJudge(ctx context.Context, user userMessage, usedTools
return judgeRuling{start: true, verdict: verdict, turn: turn, ahead: ahead}
}

// stoppedTaskResultFor reports whether this turn is answering the result of a
// stopped task that was started from the same request. A new message from the
// person is a new instruction and remains eligible for the ordinary judge.
func (a *Agent) stoppedTaskResultFor(asked string) bool {
a.mu.Lock()
results := append([]uint64(nil), a.turnResults...)
a.mu.Unlock()
if len(results) == 0 {
return false
}
graph := a.tasker()
if graph == nil {
return false
}
for _, id := range results {
node := graph.node(id)
if node != nil && node.wasStopped() && strings.TrimSpace(node.request()) == strings.TrimSpace(asked) {
return true
}
}
return false
}

// applyRouteJudge is the other half of [Agent.readRouteJudge]: the decision,
// spent. The TURN calls it, and it calls it at the one moment it knows the turn
// is really over — after the reader that may re-open the turn has said it will
Expand Down Expand Up @@ -766,7 +806,7 @@ func routeJudgeQuestion(asked, answered string) string {
// to surface features is a keyword rule with a bill attached.
const routeAheadBrief = `You judge ONE message a person has just typed to a coding assistant, BEFORE it is answered. Nothing has been done yet: you are reading the REQUEST, not a reply. You decide one thing: is this WORK?

WORDS are a question, a discussion, advice, an opinion, a fact, an explanation, a plan somebody asked to read. SMALL WORK is words too, for this purpose: a few tool calls, one obvious edit, a file read and an answer. Handing small work off is slower than doing it, so it is not work.
WORDS include questions about feasibility, discussion, advice, opinions, facts, explanations and plans somebody asked to read. A polite question that clearly asks you to act now can still authorize work, as the work rule below explains. SMALL WORK is words too, for this purpose: a few tool calls, one obvious edit, a file read and an answer. Handing small work off is slower than doing it, so it is not work.

WORK is research across several sources, changes across several files, a goal with several independent parts, or anything the person would otherwise watch a spinner for. Three shapes of request are almost always work: one message that asks for SEVERAL INDEPENDENT DELIVERABLES, a SWEEP across many files or many sources, and anything whose answer is MINUTES OF TOOL CALLS the person can only sit and watch.

Expand Down
78 changes: 78 additions & 0 deletions internal/session/route_judge_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1902,6 +1902,84 @@ func TestBothBriefsAskForOneObjectAndOnlyOneOfThemMentionsAnAnswer(t *testing.T)
}
}

func TestRouteJudgesDistinguishWorkOrdersFromQuestions(t *testing.T) {
contract := strings.Join(strings.Fields(routeVerdictContract), " ")
for _, example := range routeIntentExamples {
text := strings.Join(strings.Fields(example.request), " ")
if got := strings.Contains(example.rule, "authorize work"); got != example.wantWork {
t.Fatalf("evaluation case %s has rule %q for work=%t", example.request, example.rule, example.wantWork)
}
if !strings.Contains(contract, text) {
t.Errorf("the shared route contract omits %s", example.request)
}
if !strings.Contains(contract, example.rule) {
t.Errorf("the shared route contract does not explain why %s is work=%t: missing %q", example.request, example.wantWork, example.rule)
}
for name, brief := range map[string]string{"pre-turn": routeAheadBrief, "post-turn": routeJudgeBrief} {
brief = strings.Join(strings.Fields(brief), " ")
if !strings.Contains(brief, text) || !strings.Contains(brief, example.rule) {
t.Errorf("the %s route prompt does not carry the %s classification", name, example.request)
}
}
}
for _, want := range []string{
"clearly asks codeaf to do work now",
"Judge intended action and timing, not grammar",
"authorization before effort",
} {
if !strings.Contains(contract, strings.Join(strings.Fields(want), " ")) {
t.Errorf("the shared route contract does not state %q", want)
}
}
if !strings.Contains(routeAheadBrief, routeVerdictContract) || !strings.Contains(routeJudgeBrief, routeVerdictContract) {
t.Fatal("one route judge can start work without the authorization rule")
}
}

// routeIntentExamples is the small, reviewable evaluation set for the prompt
// change. The production judge remains model-backed; these labels are the
// expected contract that both independent readers must follow.
var routeIntentExamples = []struct {
request string
wantWork bool
rule string
}{
{`"Could you fix the failing tests?"`, true, "both authorize work"},
{`"Can you implement headless screenshots and add tests?"`, true, "both authorize work"},
{`"How would headless screenshots work?"`, false, "does not"},
{`"Can we get screenshots without a display?"`, false, "ambiguous without context"},
{`"We should support screenshots eventually"`, false, "future direction, not authorization"},
}

func TestAStoppedTaskResultCannotAutoStartItsRequestAgain(t *testing.T) {
completer := &routeCompleter{answer: "I can report what happened.", verdict: routeYes}
agent, _, _ := routeAgent(t, completer)
graph := agent.graph()
id := graph.reserve()
graph.admit(id, taskSpec{title: "the requested audit", request: routeAsk, brief: "audit", acceptance: "done"})
node := graph.node(id)
node.markStopped()
agent.principal = NewSteward(routeAsk, Budget{Wall: time.Hour}, nil)
agent.mu.Lock()
agent.turnResults = []uint64{id}
agent.mu.Unlock()

ruling := agent.readRouteJudge(context.Background(), userMessage{wake: true}, false, completer.answer)
if ruling.start {
t.Fatal("a stopped task result started the same request again")
}
if completer.asked() != 0 || completer.confirms() != 0 {
t.Fatalf("the route judge spent %d screen calls and %d confirms on a stopped result", completer.asked(), completer.confirms())
}

// The same words in a new human turn are a fresh instruction.
ruling = agent.readRouteJudge(context.Background(), userText(routeAsk), false, completer.answer)
if !ruling.start {
t.Fatal("a new human instruction was suppressed by an earlier stopped result")
}
ruling.release()
}

// ── the order of the two readers at a turn's end ────────────────────────────

// A JUDGE THAT HAS ALREADY ANSWERED HAS STILL STARTED NOTHING.
Expand Down
Loading