Expose word timestamps for Parakeet

[ArturWierzbicki]

Summary

I have a tool that turns recorded meetings into markdown transcripts with named speakers and screenshots of what was shared on screen. Today it transcribes with Parakeet TDT via parakeet-mlx. I'd love to move to transcribe.cpp to simplify the setup.

To do so, I need word timestamps: I use them to match words to diarized speaker turns and to merge overlapping audio chunks by cutting at silences between words. transcribe.cpp already computes them, but the per-file JSON from --batch-jsonl does not expose them. This PR exposes them and verifies them: the batch JSON gains words and tokens arrays, the parakeet NeMo dumper records word timings in its reference output, and validate.py compares each word's start and end against the reference within a new 80 ms tolerance, the duration of one encoder frame. The comparison code is shared across model families; a family turns it on by dumping reference word timings and adding a timestamps entry to its tolerance file.

The check caught a real bug on its first run. A comma is its own token, and the model predicts a duration for it as it does for every token, even though nothing is spoken there.

Before this PR the decode kept those predicted durations, and word ends inherited them, drifting up to 560 ms past the speech on the validation clip. NeMo gives punctuation zero duration instead: in JFK's "And so, my fellow Americans, ...", the token " so" ends at 800 ms and the "," sits exactly there, so the word "so," ends when the audio does. The offline decode in src/arch/parakeet/model.cpp now applies the same rule; transcript text is unchanged.

Scope

• src/arch/parakeet: punctuation tokens get zero duration in offline decode; the set of punctuation tokens is built once at model load, and vocab entries that are not valid UTF-8 skip the rewrite
• examples/cli: words appears in batch JSONL when a run returns word or token timestamps, tokens only at token level
• scripts: the parakeet reference dumper emits word and token timings; validate.py gains the word-timestamp comparison and unit tests
• tests: a timestamps tolerance entry for parakeet; real-model smoke assertions for zero-duration punctuation
• docs: tools/validate, model-family-testing, and the v2 and v3 model cards

AI Assistance

Yes, heavy Fable usage: code/docs/investigation. I directed and reviewed the work and own the change, happy to iterate more. I'm not an ASR/C++ person, so I also iterated upon a human-digestible :), zero-context explainer of the whole change for me and anyone else who wants the background: https://gist.github.com/ArturWierzbicki/3a5e21740703103e0a89a51428a862a6

Validation

• validate.py all for parakeet-tdt-0.6b-v2: 18/18 tensors within tolerance, exact transcript, and word timestamps that deviate from the reference by 80 ms max and 3.6 ms mean.
• validate.py all for parakeet-tdt-0.6b-v3: exact transcript, and word timestamps that match the reference exactly (0.000 ms max deviation).
• ctest 31/31, pinned clang-format clean, validate.py unit tests pass.
• No WER impact: transcript text is unchanged (the transcript compare stays exact). The batch JSONL change also reaches granite, gigaam, and medasr, which report word or token timestamps too; the WER harness reads only the text field, so their runs are unaffected.
• The punctuation rule changes public token and word timings for every TDT-head variant. v2 and v3 are validated here; tdt-1.1b and the tdt_ctc hybrids are not, and their word-timestamp check turns on automatically the next time their reference dumps are regenerated.
• The opt-in real-model smoke test: every assertion this PR adds passes; 5 pre-existing failures remain (a rejected call leaves the previous result exposed)

[cjpais]

Thanks for this PR. I will take a look at it soon and then try to pull it. And I think this is pretty reasonable