Repository navigation
Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness - #250
Draft
fanny-riols wants to merge 17 commits into
Draft
fanny-riols wants to merge 17 commits into
fanny-riols wants to merge 17 commits into
Conversation
- The cultural-norms exception now also covers formal register and set service phrases. It allows one such phrase per function (acknowledge, hold, return from hold, thank, close), and a single short courtesy or empathy phrase is never filler on its own. - It does not cover repetition: the same formula twice in one turn, stacked acknowledgements, or a term given together with its synonym or translation are still filler. - Length is judged by how long the turn takes to say and how many information units it carries, not by written length. Polite constructions and numbers written out in words can look long without being long to hear.
- violating_policies: a phrase or clause in another language inside an
otherwise correct-language turn ("one moment", "Let me check that") is
part of "not communicating in the specified language". Established
loanwords, codes, IDs, acronyms, product names and proper nouns are
never a violation.
- failing_to_disambiguate: a value that only contains letters or digits
that sound alike is not by itself a flag. Phonetic confusability counts
only when the trace shows the ambiguity mattered: a failed lookup, a
user correction, two versions of the value, or a real homophone.
- The fabricating_tool_parameters note on confusable letters now points
to the language-specific guidance.
New violation 5: a term immediately followed by its translation, equivalent or abbreviation in parentheses or after a slash, such as "paid time off (PTO)" or "PIN / passcode". TTS does not voice the parentheses or slash, so the listener hears both forms back to back. A word, code or abbreviation used on its own is not a violation.
New block "Spelled codes and mixed-language speech":
- Letters and digits in spelled codes must be said with their names in
the expected language. A letter or digit name from another language
(the English "jay" or "six" in a non-English turn) is wrong_language,
plus entity_error if it sounds like a different letter or digit.
- Spelling-alphabet words ("A as in Anna") are valid.
- Confusable letters and digits are flagged only when the other one is
clearly heard, not because of transcription or poor audio.
- A phrase or sentence in another language inside the turn is
wrong_language, even when the rest of the turn is correct.
- Proper nouns, product names, codes, acronyms and established loanwords
are not wrong language.
- An unusual but unambiguous number, date or time reading is an error
only if a native listener could take it for a different value.
New block "Spelled codes and mixed-language speech":
- Letters and digits of trace entities must be said with their names in
the expected language. A letter or digit name from another language
(the English "jay" or "six" in a non-English turn) means the entity
was not articulated correctly: rating 0. An entity that is itself in
another language in the trace may be said in that language.
- Spelling-alphabet words ("A as in Anna") are valid.
- Confusable letters and digits are flagged only when the other one is
clearly heard, not because of transcription or poor audio.
- A phrase or sentence in another language inside the turn: rating 0,
quoting those words in the explanation.
- Proper nouns, product names, codes, acronyms and established loanwords
are not wrong language.
- An unusual but unambiguous number, date or time reading is rated 0
only if a native listener could take it for a different value.
New section "Spelled Codes and Mixed-Language Speech":
- Letters and digits in spelled codes must be said with their names in
the expected language. A letter or digit name from another language
(the English "jay" or "six" in a non-English turn) rates at most 2
while the value is still unambiguous, and 1 if it sounds like a
different letter or digit.
- Spelling-alphabet words ("A as in Anna") are valid.
- Confusable letters and digits are flagged only when the other one is
clearly heard, not because of transcription or poor audio.
- A phrase or sentence in another language inside the turn rates at
most 2, and 1 if it replaces an entity or the turn no longer makes
sense.
- Proper nouns, product names, codes, acronyms and established loanwords
are not wrong language.
- An unusual but unambiguous number, date or time reading rates at most
2 if it sounds unnatural, and 1 only if a native listener could take it
for a different value.
This was referenced Oct 29, 2026
…hared-multilingual-rules
The shared rules in this branch change the English prompts of six judges. This records their new English renders.
- Expanding an acronym once, on its first mention ("your PTO, that's
your paid time off"), is not repetition. A term given with its synonym
or its translation into another language is still filler. The example
no longer assumes a non-English call.
- The one-phrase-per-function allowance also covers greet / open.
- Polite constructions and numbers written out in words can look long
without adding cognitive load. The previous wording said they don't
take longer to say, which isn't true.
The rating scale had "briefly using the wrong language" at 2 and "repeatedly talking in a language not instructed" at 1, so a recurring English "one moment" in a French call could go either way. Now: - an isolated fixed phrase (a hold, acknowledgement or courtesy formula) is rated 2, even when it recurs across turns; - whole sentences of content in another language, or most turns in another language, are rated 1.
Violation 5 flagged "paid time off (PTO)", which is a normal acronym
expansion in English HR and IT turns. It now covers only a term followed
by its translation into another language or by a synonym that means
exactly the same thing ("PIN / passcode"). Not a violation:
- an acronym expanded once, on its first mention;
- a slash between paired or alternative terms ("AM/PM", "and/or");
- a word, code or abbreviation used on its own.
Also drop the claim that TTS never voices a slash; some engines do.
- Confusable letters and digits: the rule now only stops the judge from
flagging one because of its own transcription. Audio that is ambiguous
between two letters or digits is still an error. The previous "benefit
of the doubt" wording contradicted the entity_error definition.
- A phrase in another language is an error only if it is not in the
intended text; reproducing a phrase the intended text contains is not.
- The wrong_language definition and the rating-0 definition now name the
cases the new rules use: a phrase or clause in another language, and a
letter or digit name from another language inside a spelled code.
- An unusual reading covers only the intended words read in an unusual
order or grouping (digit by digit vs as one number). A different word
for the value ("septante" for "soixante-dix") is still an error, as the
existing rule says.
- The example no longer reads "a English turn" in English runs.
- A phrase in another language around the entities no longer rates the
turn 0. Before, it did only when the turn had entities, so "One moment,
je vérifie." was excluded while "One moment, je vérifie le code
ZK3FFW." was rated 0 with a perfect code. Faithfulness and TTS fidelity
already score the phrase.
- A letter or digit name from another language still rates 0 when it is
inside a trace entity.
- Confusable letters and digits: the rule now only stops the judge from
flagging one because of its own transcription. Audio that is ambiguous
between two letters or digits is still an error, as in the judge's own
examples ("M" sounding like "N", "fifty" vs "fifteen").
- The example no longer reads "a English turn" in English runs.
- Confusable letters and digits: the rule now only stops the judge from
flagging one because of its own transcription. Audio that is ambiguous
between two letters or digits is still an error. The previous "benefit
of the doubt" wording made the judge more lenient on the hardest cases,
so fewer records with a mis-spoken code were rerun.
- A phrase in another language is an error only if it is not in the
intended text; reproducing a phrase the intended text contains is not.
- An unusual reading covers only the intended words read in an unusual
order or grouping. A different word for the value ("septante" for
"soixante-dix") is still flagged, as the existing rule says.
- The example no longer reads "a English turn" in English runs.
…hared-multilingual-rules
…hared-multilingual-rules
The fabricating_tool_parameters note on confusable letters pointed to "the language-specific guidance below". This branch adds no guidance, so the pointer moves to the branch that does.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Some of the problems the native-speaker reviewers found are not specific to one language. They would happen in any language, including English, so they belong in the shared judge prompts rather than in per-language guidance.
This PR changes English scoring for all six judges it touches: conciseness, faithfulness, speakability, tts_fidelity, agent_speech_fidelity and user_speech_fidelity.
Versions are bumped in #251, so this PR must not be run or merged on its own.
What changes
One commit per judge, so each can be reviewed and validated separately. After review, a second commit per judge applies the review fixes; the table below describes the result.
wrong_languagedefinition and the rating-0 definition now name the phrase and letter-name cases.Spelled codes and mixed-language speech:
tests/fixtures/judge_prompts_en.jsonrecords the new English renders of the six judges.Validation
English judge validation datasets (
docs/metrics/judge_validation_datasets/), production judge models, 3 runs per side, mean ± std. Before is Judge prompts: per-language guidance slot #249, which renders the same English prompts asmain.us.anthropic.claude-opus-4-6-v1)gpt-5.2)Both changes are within one standard deviation: no English regression on these datasets. The dataset runner has no audio judges, so agent_speech_fidelity is not covered here.
Not done yet: re-scoring the existing runs with the changed judges, English included. There is no English dataset for speakability, tts_fidelity, user_speech_fidelity or (in this runner) agent_speech_fidelity, so this is their only check.
This PR stays a draft until that is done.
Stack
Versions are bumped in PR 3.