[fanny-riols]

Why

Native-speaker linguists reviewed EVA simulated conversations in 9 non-English languages and checked every metric. For stt_wer, most of what they flagged was not an STT error. For example, the normalizer treated two correct ways of writing the same spoken content as different words, so a correct transcript was scored as wrong.

This PR fixes the normalization for de, nl, fr, fr-CA, es, it, hi, ko and ja, and makes Hindi scoring aware of English words mixed into Hindi speech (code-switching). English output is unchanged except for one fix: normalize_text(..., "en") gives identical results on the strings in the three datasets, before and after, apart from the 7 that spell an ID with the word "underscore" (last commit). Those now match the written ID.

Changes by file

Shared: engine.py, wer_utils.py, configs/README.md

Three new optional fields in a language config. A field does nothing unless a language turns it on, so this commit changes no output. test_wer_normalization_hooks.py checks that English and every language that does not opt in are unaffected.

• char_folds: treat two spellings of a character as the same, on both the reference and the transcript (Hindi ज़ = ज, German ü = ue). The folds are also applied to the number-word vocabulary, so German "fünf" still parses as 5.
• join_hyphenated_compounds: treat a hyphenated compound and its closed form as one word (Anmelde-Konto = Anmeldekonto). The hyphen is kept next to a number word, next to an acronym (HR-Fragen, which STT often writes open as "HR Fragen"), and in reduplications (Hindi बार-बार).
• spoken_symbols / spoken_symbols_code_only: when a user spells an ID aloud, they say the name of the symbol, such as "A B C trait d'union 1 2 3". The STT writes ABC-123. Before, only the English word "dash" was recognized, so every other language counted the spoken word as an error. Symbol words that are also ordinary words (Spanish "barra", German "Strich") are only replaced next to something that looks like a code. That way "una barra de pan" is left alone. The last commit makes every language cover -, _ and / (English gets "underscore" and, next to a code only, "slash"). English "at" and Hindi "ऐट" are left out for @: they are ordinary words, and the code rule can't recognize them in an email address.

Per Language

German: configs/de.json

• Hyphenated compounds. The intended text had Anmelde-Konto, the TTS said one word, and the STT wrote Anmeldekonto. That counted as two errors.
• Spoken symbols in IDs. "Bindestrich", "Unterstrich", "Schrägstrich" and "Klammeraffe" now match -, _, / and @. The intended text "read underscore only" was counted as an error when the STT wrote "Read only".
• Umlauts. Krüger = Krueger, Straße = Strasse.
  • Please check: the old normalizer stripped all diacritics, so an STT that drops the umlaut (Kruger) used to match Krüger. With ü = ue it no longer does. We think that is correct, because "Kruger" is a misspelling, but it is a scoring change.

Dutch: configs/nl.json

• Hyphenated compounds match closed compounds (zorg-verlof = zorgverlof, e-mailadres = emailadres).
• "koppelteken", "laag streepje" / "liggend streepje", "schuine streep" and "apenstaartje" match their symbols. "streepje" is replaced only next to a code.

French: configs/fr.json

• Spoken symbols in IDs. "trait d'union", "tiret", "tiret bas", "barre oblique" and "arobase" now match their symbols. Reviewers reported one correct transcript of a spelled code that scored 136% WER.
• The letter M in a code became "monsieur". The rule for the abbreviation "M." also matched a bare "m", so the spelled code M 4 B was normalized to monsieur 4 b on one side and m4b on the other. Only "M." followed by a word (M. Dupont) now expands.
• The pronoun "me" became "maître". "il me faut" was rewritten. The rule is removed.

Canadian French: configs/fr-CA.json (removed), wer_utils.py

Canadian French is now normalized exactly like French, from the same fr.json. fr-CA.json was a near copy of fr.json, and it had drifted in ways that are not Québec-specific:

• a slightly different filler list ("ah" in fr, "mm" in fr-CA)
• mgr, st and ste expanded to monseigneur, saint and sainte only in fr
• "dot" accepted as a thousands separator (1.000) only in fr

Changes:

• fr-CA.json is removed. _make_normalizer now falls back from a regional variant to its base language when the variant has no config of its own (fr-CA → fr.json), the same rule services.py already uses for the STT/TTS language. Among EVA's languages this affects only fr-CA.
• fr.json gains the one filler only fr-CA had ("mm").
• The French tests run for both fr and fr-CA, including curly apostrophes in "trait d’union".

As a result, fr-CA now also gets the st → saint expansion. That rule can fire on codes containing "ST" in both languages; it is listed under "Not in this PR".

Spanish: configs/es.json

• Spoken symbols. "guion", "guion bajo", "barra", "barra oblicua" and "arroba" match their symbols. "guion" and "barra" are replaced only next to a code.
• Single letters rewritten as titles. The letter D in a code became "don". The verb "da" became "doña": "me da igual" turned into "me dona igual". Only "D." followed by a word now expands to "don".

Italian: configs/it.json

"trattino", "trattino basso" and "chiocciola" match their symbols. For example, a date of birth spoken as "1995 trattino 04 trattino 18" now matches 1995-04-18. "barra obliqua" matches /; plain "barra" is replaced only next to a code.

Korean: cjk.py (Korean section)

The number converter turned any syllable that can be a number into a digit:

Input	Before	After
삼월 이십오일 (March 25)	삼월 21	3월 25일
천민수 (surname 천)	1000민수	천민수
백지수 (surname 백)	100지수	백지수
만나서 반갑습니다	10000나서 …	unchanged
인터넷	인터4	unchanged
열두 시	10두 시	12시

It now converts a span only when it is a well-formed number in a numeric context, meaning next to a digit or followed by a counter such as 월, 일, 원, 시 or 명. Also added:

• the irregular months 유월 (6월) and 시월 (10월)
• 만-grouping (1만 2천 원 = 12,000원)
• spoken digit strings (공일공 일이삼사 = 010 1234)
• native counters (스무 살 = 20살). After 한/두/세/네 the counter must end the word or be followed by a particle, so 네 개인정보 ("yes, personal info") is left alone

The fillers 음 and 어 are now removed only as whole words; before, they were also cut out of 음식, 다음 and 영어. The tests include a list of surnames and ordinary words that must not change.

Japanese: cjk.py (Japanese section), configs/ja_loanwords.json (new)

• Spelled codes. Japanese speakers say letters and digits in katakana or kana, and the STT may write either form. イー・エム・ピー、さん now matches EMP3, and 三、一、八、五、ゼロ、四 matches 318504. Only runs of two or more code tokens are mapped, so ordinary words such as エース, ビーフ and ワイシャツ are left alone.
• Spelling variants of the same word. 何とか / なんとか, 振替 / 振り替え, 名字 / 苗字, 合って / あって.
• Product names. Reviewers: "the transcription says コンフュエンス and データドグ which is slightly a misspelling but it should not be fully marked down for WER. All entity names will be said in a Japanese accent so it should be expected for transcription to present it in kana and not in EN."
  • The intended text keeps "Confluence" in Latin script, while the STT writes katakana, so every character was a mismatch (100% error).
  • Both forms are now mapped to one standard katakana spelling (コンフルエンス) and compared in katakana. A near-miss spelling then gets partial credit: コンフュエンス vs コンフルエンス is 0.14 CER, not 1.0.
  • The table lists 22 product names from the ITSM scenarios. Only whole words are mapped: モデル and Dellwood are unchanged.
• Fillers. なんとか was turned into なか, because the filler んと was matched inside the word. Fillers are now removed only at the start of a word.

stt_wer.py: whitespace in Japanese CER

Japanese has no word spacing, but normalize_text re-joins comma-separated chunks with spaces, so every comma in a turn added character errors. Reviewers: "User Turn 1, Turn 4, Turn 8 does not contain any errors, not sure what WER is saying was added. Maybe a space or the comma?" For ja, whitespace is now removed after normalization. Korean keeps its word spaces.

Hindi: configs/hi.json, configs/hi_loanwords.json (new)

• English loanwords in Latin script. Hindi speakers use English words constantly, and the STT often writes them in Latin script. Reviewers: "WER is taking into account English loan words correctly transcribed into English (Roman) script, such as मार्च -> march, कोड -> code". A new lexicon maps the Latin spelling of 54 loanwords to Devanagari (code → कोड, time → टाइम). It was built from the reviewers' examples and the scenario data, with 157 entries at first. Later commits drop the 103 that the phonetic matcher (below) handles (march / मार्च, password / पासवर्ड). The ones kept have fewer than three consonants, too few for the matcher, a Hindi form that differs (january / जनवरी, september / सितंबर), or an unusually spelled first vowel (business / बिज़नेस).
• Nukta and chandrabindu. ज़रूर / जरूर and हाँ / हां are the same word spelled two ways. Both spellings appear in our own scenario data (ऑफ़िस / ऑफिस).
  • Please check: the fold also merges ड़/ढ़ with ड/ढ. These are distinct sounds in native words (बड़ा), not just loanword spellings. Both sides are folded, so the only effect is that the rare pair of words differing only by ड vs ड़ is no longer told apart.
• Hyphenated compounds. ई-मेल = ईमेल. Reduplications such as बार-बार keep the hyphen.
• Spoken symbols. डैश, अंडरस्कोर / underscore and स्लैश match -, _ and / (A B C डैश एक दो तीन = ABC-123). Latin slash is replaced only next to a code.

Hindi code-switching: code_switch.py (new), stt_wer.py

Hindi speakers mix English words into Hindi all the time ("मेरा flight कल है"). The intended text and the STT transcript often write those words differently: in different forms (seven बजे vs 7 बजे, I'm vs I am), or in different scripts (मार्च vs march). The loanword lexicon above and the phonetic matcher below handle the second case. The first case still counted correct English words as errors, because both sides were normalized with Hindi rules only.

How it works. The Hindi alignment is unchanged. Then each error region (a run of mismatched words, up to 4 words per side) is re-checked with an English view of the same text: Latin-script parts are normalized with the English rules ("twenty five" → 25, "I'm" → "i am"), and Devanagari parts keep the Hindi rules. A region stops counting as an error only if:

• both English views are identical, and
• the intended text is itself English there (written in Latin script, or changed by the English rules).

The second condition stops it from hiding real errors. If the intended text has the digit 7 and the STT writes "seven", the user may have said सात, so it stays an error. The STT side must be English too: intended "seven" vs STT सात is a translation and stays an error, even though both normalize to 7. Errors can only go down, never up.

Examples

Intended text	Translation	STT transcript	Before	After	Why
मेरी फ़्लाइट seven बजे है	My flight is at seven o'clock	मेरी फ़्लाइट 7 बजे है	20%	0%	The English view turns "seven" into 7.
मुझे twenty five मार्च की booking चाहिए	I need a booking for March 25	मुझे 25 मार्च की बुकिंग चाहिए	29%	0%	"twenty five" vs "25" is one region; booking / बुकिंग is matched by the phonetic matcher.
I'm not sure, क्या आप check कर सकते हैं	I'm not sure, can you check?	I am not sure क्या आप check कर सकते हैं	10%	0%	English expands "I'm" to "i am".
मुझे 7 बजे की फ़्लाइट चाहिए	I want the 7 o'clock flight	मुझे seven बजे की फ़्लाइट चाहिए	17%	17%	Not rescued: the intended "7" is not English. The user may have said सात.
मेरी flight कल है	My flight is tomorrow	मेरी fight कल है	25%	25%	A real STT error, kept.

Two neighbouring errors form one region, and that region is only rescued as a whole. For example, flight seven vs fight 7 stays two errors. This loses a little recall but never hides the real error next to the rescued word.

Output (Hindi only, so other languages produce exactly the same result as before):

• score / details.wer: the code-switch-aware WER. per_turn_wer, per_turn_errors and error_summary use it too.
• details.strict_wer: the Hindi-only WER as computed before, so runs stay comparable.
• details.code_switch: number of English words in the intended text, number of words rescued, and the WER and word count per language (hi, en). Substitutions and deletions count against the intended word's language, insertions against the inserted word's.
• details.per_turn_code_switch_rescues: every rescued pair, for audit.
• New sub-metrics: wer_hi, wer_en (omitted when a run has no English words), code_switch_word_rate (share of intended words that are English) and code_switch_rescue_rate.

docs/metrics/stt_wer.md has a new code-switching section with a Hindi example output and the meaning of each new field.

Words are tagged English when they are in Latin script, a Devanagari loanword from the lexicon (कोड), or a word the phonetic matcher rescued. Other English words written in Devanagari are counted as Hindi, so the English share is an undercount.

The behaviour is turned on by a new config field, code_switch_languages: ["en"], set only in hi.json.

Phonetic matcher. The lexicon can't list every English word. Commit 15 adds a matcher, code_switch_phonetic, for an English word written in Devanagari on one side and in Latin on the other (in either direction since the last commit). Both are reduced to a consonant key and compared:

Intended text	Translation	STT transcript	Before	After	Why
मेरे लैपटॉप में नेटवर्क प्रॉब्लम है	My laptop has a network problem	मेरे लैपटॉप में network problem है	33%	0%	नेटवर्क / network → ntwrk, प्रॉब्लम / problem → prblm
कृपया मेरा अपॉइंटमेंट कन्फर्म कीजिए	Please confirm my appointment	कृपया मेरा appointment confirm कीजिए	40%	0%	कन्फर्म / confirm → knfrm
मैं कल आऊँगा	I will come tomorrow	मैं call आऊँगा	33%	33%	A real error (कल = tomorrow), blocked by a stoplist
मैं बस से आऊँगा	I will come by bus	मैं bus से आऊँगा	25%	25%	Blocked as well (stoplist, only two consonants): precision over recall

The keys must be identical. The Latin key tries both readings of a soft g (manager = मैनेजर, target = टारगेट) and of an s between vowels (reason = रीज़न, basic = बेसिक). Anusvara before प/ब/भ/म counts as m (नंबर = number). A consonant repeated across a vowel counts twice, so another form of the word doesn't match (refunded / रिफंड, numbers / नंबर). The first vowels must be compatible (बुकिंग matches booking, not backing); words that differ only in a later vowel still match (कन्फर्म / conform). It only compares single words, needs at least three consonants, and skips native Hindi words with an English look-alike (कल/call, बस/bus, मैं/main, पर/par, कमरा/camera, मिलना/million). Commit 17 turns it on for Hindi ("code_switch_phonetic": true in hi.json); it is off for every other language. On existing Hindi runs it is where nearly all code-switch rescues come from, because the simulated user writes English words in Devanagari while the STT writes them in Latin (for example कन्फर्म / confirm, डॉलर / dollar, ट्रांसफर / transfer, स्वैप / swap).

Version bump

Where	Change	Why
STTWERMetric.version (stt_wer.py)	v0.1 → v0.2	Stamped on every stt_wer score, so scores computed before and after this PR can be told apart. This is the main place for a metric-level change.
metrics_version (src/eva/__init__.py)	2.2.4 → 2.2.5	Benchmark-wide version recorded in run provenance. Bumped together with a metric version, as in the last metric change (48704b24).
tests/fixtures/metric_signatures.json	regenerated	Only the STTWERMetric entry changes. Regenerated again in commits 14 and 15.

The code-switching commits keep v0.2 and 2.2.5: neither version is released yet.

Note for future changes: the drift test and the version-bump hook only watch src/eva/metrics/ and judge.yaml. Neither covers src/eva/utils/wer_normalization/, so a normalizer change alone would not trigger them. Here the bump is triggered by the stt_wer.py change, but the language commits before it change scores too. A follow-up could include the normalizer code and configs in STTWERMetric's signature.

Testing

• 62 new test functions (231 cases) across 9 files, built from the reviewers' examples. They check that two renderings of the same speech normalize to the same string, and that names and ordinary words are not changed.
• Code-switching: 43 test cases in test_wer_code_switch.py and 11 in test_stt_wer_metric.py. They cover every example above, the cases that must not be rescued, and the per-language counts.
• Full unit suite: 2,475 passed, 52 skipped, 3 xfailed.
• Hindi sample: 147 records from 19 existing Hindi runs (STT-based pipelines only), re-scored with this branch. Mean stt_wer per record: 0.337 with the Hindi rules alone, 0.302 with code-switching. 128 of the 147 records improve. All 1,146 rescues come from the phonetic matcher. Requiring identical keys removed 21 earlier rescues, most of them real STT errors (foxcrot / फॉक्सट्रॉट, information / कन्फर्मेशन, management / अमेंडमेंट, permanent / परमिनिट, complete / कमपीट).
• English: identical normalize_text output on 18,868 dataset strings. After the code-switching commits, normalize_text and the full stt_wer result for en (score, details, sub-metrics) are also identical to before on every string in data/*_dataset.json. The spoken-symbol commit changes exactly 7 strings, all IDs spelled with "underscore" (mag underscore mutual now matches mag_mutual).
• Not done: re-scoring the reviewed runs end to end.

Not in this PR

• Writing product names in katakana in the Japanese intended text. This is the root cause of the Japanese product-name mismatch, and it is a simulator change (add_user_language_directive). It needs a simulation_version bump and Japanese runs to be re-simulated.
• English-wide issues. Two-letter spelled runs are not collapsed ("F L" vs "FL"). Spaced digit runs don't match ("1 2 3" vs "123").
• Spoken email addresses ("john at gmail dot com", fr "arobase … point com"). "at" and "dot" need an email-context rule, not a symbol-table entry.
• Hindi fractional times (साढ़े, सवा, पौने) and other Hindi spelling variants (सितंबर / सितम्बर).
• Korean letter names in codes (에이 비 씨). They collide with common syllables, and no reviewer reported them.
• Two-letter title abbreviations that still fire on codes (nl mr/hr/ds, de hr/fr, fr st).
• Code-switching for other languages. French, German, Spanish, Italian and Dutch (English words in the same script) and Japanese / Korean (English names in Latin vs katakana or Hangul) will follow in separate PRs.
• Romanized Hindi in the STT output ("mujhe" for मुझे). Whether it should count as correct is a separate decision. The phonetic matcher can accept some romanized Hindi words with three or more consonants ("mujhe" itself is blocked, having only two).