Transcription perturbation values (including CIs), perturbation stats pipeline, and plots#165
Merged
Merged
Conversation
… plots - Add new SttTranscription component with collapsible accuracy and delta plots - Create PerturbationMetricValueBarChart for metric value visualization - Extract shared utilities (tierLabel, colorFor, CustomTick) to perturbationChartUtils - Highlight STT model names in yellow on x-axis labels (STT section only) - Add p-value significance legend to both STT and Perturbations sections - Update leaderboardStats.json with transcription accuracy data Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l (main) Brings the perturbation analysis (data_/stats_perturbations, run_perturbations) and the additive, idempotent leaderboardStats regen tool to main. CIs use eva.utils.bootstrap (N_BOOT=2000, run_seed/SHA-256) so they match the leaderboard metrics and are deterministic. Config + data stay local (gitignored). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lindsaydbrin
force-pushed
the
pr/lindsay/transcription-perturbation-plots
branch
from
0cc1943 to
c9d3f9b
Compare
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes derive_pass_metrics and its main() call — EVA-A/X_pass@3/^3 are not consumed by regenerate_perturbation_blocks.py or the website. Config entries removed locally (perturbations_config.yaml is gitignored). Also removes the pull_perturbation_data.py reference from the data_perturbations docstring and replaces hardcoded eva-bench-stats paths in both files' config-format headers with a generic <subdir> placeholder. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lindsaydbrin
force-pushed
the
pr/lindsay/transcription-perturbation-plots
branch
from
b0ad142 to
a30da79
Compare
lindsaydbrin
commented
Jun 25, 2026
Comment on lines
+38
to
+39
| const name = payload.value.startsWith('Scribe v2.2 Realtime') | ||
| ? 'Scribe + Gemini 3 Flash + Conversational v3' |
Collaborator
Author
There was a problem hiding this comment.
@katstankiewicz just confirming that we want this rename in all plots?
CustomTick gains amberFirst (default false) to opt into amber-colored first segment, and dy (default 8) to nudge tick labels away from the x-axis. PerturbationBarChart threads amberFirst through; SttTranscription and PerturbationMetricValueBarChart opt in explicitly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lindsaydbrin
commented
Jun 25, 2026
| fontSize?: number; | ||
| angle?: number; | ||
| textAnchor?: 'start' | 'middle' | 'end' | 'inherit'; | ||
| amberFirst?: boolean; |
Collaborator
Author
There was a problem hiding this comment.
@katstankiewicz I added this so that the STT model can be highlighted for just the STT section, and not all the Perturbation plots above.
lindsaydbrin
commented
Jun 25, 2026
| dy?: number; | ||
| } | ||
|
|
||
| export function CustomTick({ x, y, payload, fill, fontSize = 10, angle = -30, textAnchor = 'end', amberFirst = false, dy = 8 }: CustomTickProps) { |
Collaborator
Author
There was a problem hiding this comment.
I also nudged the tick label away from the x-axis (dy = 8 instead of 4) because I noticed it looked awkwardly close.
Extract DELTA_COLUMNS / METRIC_VALUE_COLUMNS constants (replacing repeated column lists), simplify empty-frame constructions and loop variables, add _as_tuple helper for groupby key normalization, extract ci_row helper in metric_value_cis, fix n_complete evaluation, and align "per-scenario" → "scenario-level" terminology throughout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add (ElevenAgents) suffix to Scribe systems in EVA-A and EVA-X tables - Rename STT section to "STT Performance", drop delta plot, keep accuracy only - Sort STT bar chart by clean audio score descending - Show only STT model name on bar chart x-axis; remove stale Scribe label workaround
Add provider name prefix to STT model labels (e.g. "AssemblyAI / Universal 3.5 Pro") and a disclaimer note below the accuracy chart about ElevenLabs/Scribe's potential home-court advantage when transcribing ElevenLabs TTS audio.
fanny-riols
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Claude-generated, lightly edited.)
Adds a cascade-only "STT Performance: Transcription Accuracy (Key Entities)" section to the results page (clean + 3 perturbations, values + delta plots), and brings the perturbation analysis pipeline to main.
Website (d99cf0f, isolated to easily merge anything useful with #164): new
PerturbationMetricValueBarChart+SttTranscriptionsection; shared chart helpers extracted toperturbationChartUtils(incl.StarMark/SeparatorsLayer);leaderboardStats.jsongains transcriptionperturbation_delta+metric_values.Pipeline (8596c07):
analysis/perturbations/computes perturbation deltas + per-condition metric-value CIs and writes them intoleaderboardStats.json(additive, idempotent, dry-run by default). CIs useeva.utils.bootstrap(N_BOOT=2000, SHA-256 run_seed) so they match the leaderboard metrics and are deterministic. Config + data are local/gitignored.Does not include:
docs/rebuild on website-merge.