Repository navigation
Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja - #251
Draft
fanny-riols wants to merge 23 commits into
Draft
fanny-riols wants to merge 23 commits into
fanny-riols wants to merge 23 commits into
Conversation
Adds German guidance to the judge prompts. It is rendered only for German conversations, so other languages are unchanged. - agent_speech_fidelity (de): 10 rules - conciseness (de): 2 rules - faithfulness (de): 5 rules - transcription_accuracy_key_entities (de): 2 rules - tts_fidelity (de): 10 rules - user_speech_fidelity (de): 10 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds Dutch guidance to the judge prompts. It is rendered only for Dutch conversations, so other languages are unchanged. - agent_speech_fidelity (nl): 9 rules - conciseness (nl): 2 rules - faithfulness (nl): 4 rules - tts_fidelity (nl): 9 rules - user_speech_fidelity (nl): 9 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds French and Canadian French guidance to the judge prompts. It is rendered only for French and Canadian French conversations, so other languages are unchanged. - agent_speech_fidelity (fr): 9 rules - agent_speech_fidelity (fr-CA): 3 rules - conciseness (fr): 3 rules - conciseness (fr-CA): 1 rule - faithfulness (fr): 3 rules - faithfulness (fr-CA): 2 rules - tts_fidelity (fr): 9 rules - tts_fidelity (fr-CA): 3 rules - user_speech_fidelity (fr): 9 rules - user_speech_fidelity (fr-CA): 3 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds Spanish guidance to the judge prompts. It is rendered only for Spanish conversations, so other languages are unchanged. - agent_speech_fidelity (es): 9 rules - faithfulness (es): 4 rules - tts_fidelity (es): 9 rules - user_speech_fidelity (es): 9 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds Italian guidance to the judge prompts. It is rendered only for Italian conversations, so other languages are unchanged. - agent_speech_fidelity (it): 8 rules - faithfulness (it): 3 rules - tts_fidelity (it): 8 rules - user_speech_fidelity (it): 8 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds Hindi guidance to the judge prompts. It is rendered only for Hindi conversations, so other languages are unchanged. - agent_speech_fidelity (hi): 9 rules - conciseness (hi): 3 rules - faithfulness (hi): 5 rules - speakability (hi): 3 rules - transcription_accuracy_key_entities (hi): 4 rules - tts_fidelity (hi): 9 rules - user_speech_fidelity (hi): 9 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds Korean guidance to the judge prompts. It is rendered only for Korean conversations, so other languages are unchanged. - agent_speech_fidelity (ko): 7 rules - conciseness (ko): 2 rules - faithfulness (ko): 4 rules - transcription_accuracy_key_entities (ko): 2 rules - tts_fidelity (ko): 7 rules - user_speech_fidelity (ko): 7 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
Adds Japanese guidance to the judge prompts. It is rendered only for Japanese conversations, so other languages are unchanged. - agent_speech_fidelity (ja): 10 rules - conciseness (ja): 3 rules - conversation_progression (ja): 4 rules - faithfulness (ja): 5 rules - speakability (ja): 3 rules - transcription_accuracy_key_entities (ja): 3 rules - tts_fidelity (ja): 10 rules - user_speech_fidelity (ja): 10 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.
The guidance slot, the shared rules and the per-language guidance change the prompt or source of these metrics, so their scores are not comparable with earlier versions: - conciseness v0.1 -> v0.2 - conversation_progression v0.2 -> v0.3 - faithfulness v0.3 -> v0.4 - speakability v0.3 -> v0.4 - agent_speech_fidelity v0.4 -> v0.5 - tts_fidelity v0.3 -> v0.4 - user_speech_fidelity v0.2 -> v0.3 - transcription_accuracy_key_entities v0.4 -> v0.5 - user_behavioral_fidelity v0.1 -> v0.2 (source only: shared base class) metrics_version is not changed.
This was referenced Oct 29, 2026
…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json
- agent_speech_fidelity: drop the entries that rated a clearly spoken
but wrong value 0 ('halb drei' for 3:30, soixante-sept for 77, ...).
The judge leaves the value to faithfulness and rates articulation only,
as in its own "$315" example. The reading facts stay.
- tts_fidelity hi and ja: a mispronounced non-entity word is no longer
garbled_hallucination, which means unintelligible noise. It rates 0
only if it changes the meaning.
- Speech fidelity nl, es, it and ko: the first bullet now lists the
English letter and digit names typically heard, as de and fr do,
instead of restating the shared rule in three variants.
- Example values that occur in the datasets are replaced with synthetic
ones (codes, user names, a surname). "M comme Marcel" stays: it is the
standard French spelling alphabet.
- The metric signature fixture is regenerated.
- Speech fidelity: add the regional 'viertel drei' = 2:15 next to 'drei viertel drei' = 2:45. - Speech fidelity: drop "Meier/Meyer-type spelling variants are fine"; the shared prompt already says it. - Conciseness: keep the written-out date example and drop the restated shared rule.
- Conciseness: "Ik ga meteen kijken" and "Ik ga uw reservering er direct bij zoeken" are single hold phrases, so they are fine on their own. They are filler only when stacked with another hold phrase or lead-in. - Speech fidelity: "bee, cee, dee, pee, tee sound like English" was phonetically wrong (Dutch B is /beː/, close to English "bay"). It now says these written names are the Dutch names. - Speech fidelity: the acronym exemption covers acronyms used as words or names (ID, FMLA, I-9). The letters of a spelled code or identifier still need Dutch names, as the rest of the guidance says. - Speech fidelity: the vowel-change example used 'veer', which is not a digit. The rule now says "makes a digit unrecognisable or turns it into another digit". - Speech fidelity: drop "Upper or lower case need not be spoken", which is not specific to Dutch.
French: - Faithfulness: a code with a compound-number sequence no longer always needs digit-by-digit confirmation. Flag the missing confirmation only if the trace shows the ambiguity mattered, as the shared rule says. - Conciseness: "Je peux vous aider avec ça ?" is a calque; use "Puis-je vous aider ?". Vous-forms are described as normal register instead of restating the shared spoken-length rule. - Speech fidelity: dix-huit is liaison, not elision. 'trois heures et quart' = 3:15 moves here from fr-CA, since it is standard French. Canadian French (adds to the French text, which fr-CA also gets): - Faithfulness: drop the compound-number bullet, which only repeated fr. - Speech fidelity: drop the bullet that repeated the fr and shared rules on English letter names and phrases. - Speech fidelity: the time bullet now gives the Québec forms 'trois heures moins quart' (without 'le') and the colloquial 'quart à trois', both 2:45, so the fr 'moins le quart' entry doesn't reject them. - Conciseness: keep the warm-phrase examples, drop the restated rule.
- Speech fidelity: the B/V bullet said 'be' and 've' sound the same, then called 'be' for V a letter error. Now only a clearly distinguishing name is a letter error: 'uve' / 've corta' said for B, or 'be larga' / 'be de Barcelona' said for V. - Faithfulness: the target is European Spanish, so note that with distinción, C before e/i and Z (/θ/) are distinct from S (/s/). The S/C/Z merge applies to seseo speakers.
Speech fidelity: - "a Italian listener" → "an Italian listener". - The realistic wrong date form is "l'uno marzo", not "il uno marzo". - Spell out the 'Gioia' example: the letter J read as the word 'Gioia' instead of 'i lunga'.
EVA supports Hindi–English code-switching, so the judges no longer
penalise English words or phrases inside a Hindi sentence.
- Faithfulness: English loanwords, procedural terms ("Employee ID",
"last name") and short phrases ("six digits") are never a language
violation, in either script. Only a whole English sentence or turn is.
This replaces the rule that rated repeated English procedural phrases
at most 2, which also contradicted the shared rating scale. The list of
Hindi equivalents goes with it, including उपनाम, which usually means
nickname rather than last name.
- Faithfulness: evidence is a whole English sentence or turn, not Latin
script, which on its own says nothing about the language spoken.
- Speech fidelity: English words and phrases inside a Hindi sentence are
not wrong language. For tts_fidelity and user_speech_fidelity, only a
whole English sentence that is not in the intended text is.
- Speech fidelity: the mispronunciation examples now say what the wrong
pronunciation sounds like ('aisle' said as 'es-le'); the others gave
none and are dropped.
Faithfulness: 공 (0) does not sound like the letter O. The real collisions are that the letter O (오) is also the digit 5 (오), and that O is easily confused with 0 in writing.
Speech fidelity: - Counter readings: ななじ is as standard as しちじ for 7時, and じっぷん is a standard reading of 10分; both are added. - く (9) is not confusable with シー (C) or いち (1); drop it from that list. - Keep the katakana examples of a letter group read as a word, and drop the "codes must be read letter by letter" rule, which is not specific to Japanese.
…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json
…-language-guidance-content
…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json
The fabricating_tool_parameters note on confusable letters now points to the language-specific guidance, which this branch adds. The note is in the shared prompt, so the English faithfulness prompt gets the pointer too; "if any" covers languages without guidance.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Native speakers reviewed EVA conversations in 9 languages and pointed out judge errors that depend on the language. Examples:
This PR adds that knowledge as per-language guidance. Guidance is rendered only for its own language, so English and every other language are unchanged by a given language's commit. The one exception is a shared pointer in faithfulness (below).
Review fixes
After review, the branch gets one more commit per language with that language's fixes (same per-language rule), plus one commit for the fixes shared by all languages:
garbled_hallucination.Faithfulness pointer. One more commit adds "(see the language-specific guidance below, if any)" to the fabricating_tool_parameters note on confusable letters in the shared faithfulness prompt, now that the guidance exists. Because the note is shared, this is the only change in this PR that reaches the English prompt: one line in faithfulness.
Hindi code-switching. EVA supports Hindi–English code-switching. The Hindi commit makes the judges accept English words and phrases inside a Hindi sentence (loanwords, "Employee ID", "last name", "six digits"). Only a whole English sentence or turn is a language violation.
Native review needed:
Not changed, on purpose:
Version bump
Bumped once here for the whole stack (guidance slot, shared rules and guidance content), since all three change these metrics' prompts or source:
The review fixes don't change this list.
metrics_versionis not changed.tests/fixtures/metric_signatures.jsonis regenerated.Validation
### <Language>-specific guidanceblock (plus that line in faithfulness), and fr-CA gets the French text followed by its own entry.Stack