A misworded verdict says which way it is wrong

[WaylandYang]

Pitfall 8 of docs/design/prior-work.md: a quote that is a substring is not yet support, and a three-way label is more useful than a binary one (Yue et al. 2023: attributable, extrapolatory, contradictory).

What lands

Each misworded verdict says which way it is wrong. scripts/bench/judge_open.mjs keeps its three verdicts and their wording, and runs a second pass over the statements it judged misworded, asking for a kind: extrapolated (the document states a relationship between the pair, the statement goes beyond it: a qualifier, a time or a detail the document gives for something else or does not give, a phrase that claims more) or contradicted (the document says otherwise: the direction is reversed, the phrase says what the document denies, the value is another row's or column's). The report prints the split beside the misworded count; the judged JSON carries kind per statement. The threshold and every earlier number stay comparable because the verdict pass is byte-identical to before.

Why a second pass and not a fourth verdict. Both were tried on the same 909 statements of the #748 run, with the same fast judge:

judge	misworded	not stated
unchanged, pass 1	5.0%	0.1%
unchanged, pass 2	3.6%	0.0%
three verdicts reworded as four (stated, extrapolated, contradicted, not stated)	0.8%	0.1%
kind asked in the same call	2.1%	0.0%
this change: kind in a second pass	3.7% (extrapolated 8, contradicted 26)	0.0%

The judge is sensitive to its own wording: rewording the verdicts made it pass six of every seven statements it had faulted, and an extra question in the same call took two points off. The second pass sits inside the unchanged judge's own band and agrees with its two passes on 887 and 899 of 909 verdicts.

The judge's own variance is now a number. Two passes of the unchanged judge on the same statements: misworded 5.0% and 3.6%, 97% of verdicts identical; reads-alone 4.8%, 8.6% and 10.7% across three passes. docs/design/extraction.md records it: a difference under two points of misworded between two runs is noise, and reads-alone is reported but not compared between single runs. This narrows what the day's earlier comparisons can claim: #744, #745 and #748 differ from each other by less than the judge's band on misworded, and their reads-alone differences are inside it.

Not in this change

A calibration set labelled with the kind (the existing one is binary and measures agreement on stated versus not); a stability number for the extraction side, which needs a second extraction of the same documents.

🤖 Generated with Claude Code