The real-speech numbers in the paper are transcribed from project evaluation logs (docs/eval-ft-context.md), not from checked-in artefacts. artifacts/ is gitignored, so no prediction file backs any row of Table 5.
The paper's Reproducibility section admits this. It is the weakest point in an otherwise reproducible package, and it is cheap to fix.
What's needed
- Commit the prediction JSONLs for every system in Table 5 (they are small — one line per utterance)
- Commit the scorer output alongside them
- A single
make benchmark / run_leaderboard.sh that regenerates the whole table from published checkpoints
- Decide on a home:
04-evaluation/eval-ft/results/ tracked, rather than artifacts/ ignored
Done when
Table 5 can be regenerated with one command, and the Reproducibility section no longer carries the caveat.
The real-speech numbers in the paper are transcribed from project evaluation logs (
docs/eval-ft-context.md), not from checked-in artefacts.artifacts/is gitignored, so no prediction file backs any row of Table 5.The paper's Reproducibility section admits this. It is the weakest point in an otherwise reproducible package, and it is cheap to fix.
What's needed
make benchmark/run_leaderboard.shthat regenerates the whole table from published checkpoints04-evaluation/eval-ft/results/tracked, rather thanartifacts/ignoredDone when
Table 5 can be regenerated with one command, and the Reproducibility section no longer carries the caveat.