fix(audio): carry authoring gain above unity into preview and render - #3304
fix(audio): carry authoring gain above unity into preview and render#3304miguel-heygen wants to merge 4 commits into
Conversation
`HTMLMediaElement.volume` is spec-clamped to [0,1], so both timeline probes lost a clip's authored gain the moment it also carried a fade: the probe seeded the element at the clamped value and every sample read back at or below 0 dB, and the mixer prefers probed keyframes over the static volume. Both probes now shadow the accessor for their own duration and forward the clamped value to the native setter, so the authored gain survives while nothing outside the probe ever sees an illegal volume. Measured on one 6 s composition, first 4 s: unity -32.8 LUFS, boosted -27.0, boosted-with-fade -32.8 before and -27.0 after — +5.8 dB, exactly the gain the clip was authored at. Collapse the other owners of the same decision, each of which kept preview and render disagreeing on their own: the Web Audio master gain had two writers, so unmuting discarded the user's volume; the user's volume was applied twice, once per clip and once on the master; and the control bridge wrote `el.volume` for clips the transport already owned, which the next tick read back as the clip's author gain. `<video data-has-audio>` is never scheduled as a buffer source, so it now routes through a media-element gain node — but only while the audio context is running, since routing into a suspended graph would silence audio the native path is playing.
The unclamped-volume shim reached for `HTMLMediaElement.prototype` before checking that a DOM exists. The renderer's probe body is serialized into the page in production but executed directly by tests that stand in for a Page, so under the bun lane it threw `ReferenceError: HTMLMediaElement is not defined` instead of falling back. Both copies now degrade to the clamped path when the constructor is absent, which is the only correct answer off-browser.
…ngs use Both cases spawn ffmpeg twice — once to build the source, once to mix — under vitest's 5 s default, which the Windows runner routinely outruns. Every other ffmpeg-driven suite in this package already declares 180 s.
jrusso1020
left a comment
There was a problem hiding this comment.
Reviewed at 907362b8. No blockers in the diff — the findings below are in surfaces this PR deliberately leaves alone, plus two test-pinning nits.
Audited end-to-end: packages/core/src/audioGain.ts, runtime/media.ts, runtime/mediaVolumeEnvelope.ts, runtime/webAudioTransport.ts, the three changed regions of runtime/init.ts, engine/src/services/audioMixer.ts, producer/src/services/htmlCompiler.ts. Read for context only: the test files' unchanged neighbours, package.json / package-subpaths.json (mechanical subpath addition). Not executed: I did not run the suites locally, so everything below is a source read, not a test result — CI at this head is the execution evidence.
CI, verified rather than inferred: all 8 required contexts on main are green at 907362b8 (Build, Test, Typecheck, Test: runtime contract, regression, Semantic PR title, Tests on windows-latest, Render on windows-latest). BLOCKED is the reviewer gate alone.
Strengths
- One ceiling, reachable from both sides, and the duplicate does not duplicate it.
MAX_AUDIO_GAINis defined once (audioGain.ts:9-10), imported by the render mixer (audioMixer.ts:64) and passed into the page-serialized probe as a parameter (htmlCompiler.ts:1194) instead of being re-literalled inside the evaluate body. ThewithUnclampedVolumecopy in that body is real duplication, but the number that would actually drift is not part of it. That is the right seam to duplicate on. - The accessor shadow pins its own teardown.
withUnclampedVolumerestores in afinallyand re-asserts the clamped native value on the way out (audioGain.ts:59-62), andmediaVolumeEnvelope.test.tsasserts the own property is gone afterwards and that the spec setter throws again. Monkeypatches usually get reviewed on what they do while installed; this one is reviewable because the restoration is itself a test. applyElementGain's boolean makes double-application unrepresentable rather than merely absent. Returning "the graph carries it" and keyingel.volume = 1off that (media.ts:322-326) removes the class of bug, andmedia.test.tspins both directions (true → element stays at unity, false → the product is written).- The master-gain fix is a second real bug, found and pinned.
_mutedand_userVolumeas two inputs to one formula (webAudioTransport.ts:361-364), with a test that unmuting restores the user's volume instead of resetting the node to 1. - The body's load-bearing claim holds at source.
<video data-has-audio>really is never scheduled as a decoded buffer source — the scheduler selectsaudio[data-start]only (init.ts:3039) — so the media-element gain route is necessary and not redundant with the buffer path. The measured table checks out arithmetically too: 20·log₁₀(1.949845) = 5.80 dB.
important — the Studio volume slider now destroys the gain this PR makes audible
propertyPanelFlatMediaSection.tsx:56-57 computes volumePercent = Math.round(volume * 100) and hands it to a slider declared min={0} max={100} (:216-224). For a clip at data-volume="1.95" that is value={195} on a 0–100 control, displayed as 195%, and onCommit writes formatNumericValue(next / 100) where next ≤ 100. So any touch of that slider silently caps the clip at unity, dropping up to 12 dB.
What makes this a finding now rather than a pre-existing cosmetic quirk: before this PR the boost was clamped to 1.0 in preview and render anyway, so losing it cost nothing audible. After it, the same click is real, silent attenuation of a clip that was playing correctly a moment earlier.
Two things make it cheap to hold. The component already carries the precedent eight lines up — the slider is disabled={volumeAutomated} because "a value set here would be overwritten by the envelope on the next tick", and an above-unity gain is the same the slider cannot represent this level case, so disabled={volumeAutomated || volume > 1} is a one-line hold until the fader lands. Separately, formatAudioGain (added here, audioGain.ts:36) exists precisely to be the serializer for this write, and nothing calls it: the path is still formatNumericValue → roundToCenti (propertyPanelHelpers.ts:232-237), the two-decimal formatter your own doc comment names as the thing that "collapses the whole bottom of the fader onto 0". Worth confirming u2-studio-gain-surface wires it, because at this head the six-decimal serializer is unreferenced.
important — "preview and render" is the apply path; the paths that report level are untouched
The title scopes this accurately and the diff delivers what it says. I am flagging the complement because it is what will read as "the level fix did not work":
- Studio's waveform normalizes per asset.
studio-server/src/helpers/waveform.ts:48-49endscomputePeakswithconst maxPeak = Math.max(...peaks, 0.001); return peaks.map((p) => p / maxPeak)— every asset is scaled to full height independently. Two clips 12 dB apart draw identical waveforms. - And the seam structurally cannot know clip gain. The route is
/projects/:id/waveform/*, keyed on the asset path (routes/waveform.ts:8-22), cache keyed on(path, size, mtime). There is no clip identity in the request, so two clips of one asset at different gains necessarily share peaks. Readingdata-volumeat the current seam cannot fix it. - The agent-facing diagnosis path measures the source file.
skills/hyperframes-audio/references/diagnosis.md:139-195routes level questions throughvolumedetect/astats/ebur128on the asset, and:93-94advises comparing each passage to the track's own median. Both are gain-blind by construction:data-volumeis not in the file.
Net effect after this PR: a boosted clip renders correctly and still reports as unchanged in the timeline and to any tool following the audio skill. Not a defect of this diff — but if the stack's goal is that the level a user sets is the level the product agrees they set, the reporting half needs its own rung, and it is the half a user checks first.
nit — audioLeveller.ts's design rationale is now false
audioLeveller.ts:15-21 explains why the leveller writes a gain node rather than a volume lane: "volume is 0..1 and normaliseEnvelope clamps every keyframe into it, so a volume lane can only ever attenuate." This PR changes that exact function (mediaVolumeEnvelope.ts:36 — the only normaliseEnvelope in the tree) to clamp to 0..+12 dB, which is the same range the comment attributes to the gain node. The module still behaves correctly; its stated reason for existing no longer holds, and u3-normalize-audio builds on it, so the next reader is the one who pays for the stale rationale.
nit — the render-side ceiling has no test
audioMixer.test.ts:275 feeds volume: 3.98, just under MAX_AUDIO_GAIN (3.9810717…), and asserts volume=3.98. That pins the clamp's lower bound — revert clampVolume to min(1, …) and it goes red — but nothing feeds a value above the ceiling, so the engine's upper bound is unpinned: change MAX_AUDIO_GAIN's effect in the mixer and the suite stays green. audioGain.test.ts:109 pins it in core via formatAudioGain(99) === formatAudioGain(MAX_AUDIO_GAIN); the engine copy has no equivalent. One line: volume: 99 asserting toContain("volume=3.981072"), which is formatFilterNumber's output for the ceiling.
note — no limiter in the mix graph
The chain is volume=<gain> per track → amix=inputs=N with normalize on → [mixed]volume=masterOutputGain × N (audioMixer.ts:660-665). No alimiter or acompressor anywhere in it, so a source already near full scale now hard-clips where the old 1.0 clamp made that unreachable — that clamp was incidentally the only headroom guarantee in the render path. Not asking for a change here, and the PR's own measurement sits ~27 dB below full scale so it cannot surface this either way. Is limiting in u3-normalize-audio's scope?
note — blast radius, measured
Because a gain-ceiling change is the shape that usually does move existing output, I checked: no data-volume above 1 exists anywhere in the repo (fixtures, templates, registry, examples, skills), and the only UI writer caps at unity, so nothing shipped here gets retroactively boosted on its next render. The affected population is hand- or agent-authored above-unity attributes, which is small and is exactly the population this PR intends to fix.
Also confirmed no stale callers of the renamed transport API: zero onElementVolume references remain, and the two setElementVolume hits (parsers/src/types.ts:405, init.ts:354) are the unrelated elementId-keyed bridge method and its pre-existing no-op stub, not the removed one.
Verdict: COMMENT
Reasoning: No blockers in the diff and every required check is green, so nothing here should hold this up. I am not converting it to an approval unasked — a second reviewer is going through it in parallel, and an approval from me satisfies this repo's last-push gate, which is a merge authorization nobody requested. Say the word and I will approve at 907362b8.
— Rames Jusso
| * `"0"` (a hard mute) and makes the knob jump on release everywhere below | ||
| * unity. Six decimals round-trip every integer fader stop back to itself. | ||
| */ | ||
| export function formatAudioGain(gain: number): string { |
There was a problem hiding this comment.
Nothing calls this at the current head. The volume write path is still formatNumericValue -> roundToCenti (packages/studio/src/components/editor/propertyPanelHelpers.ts:232-237), i.e. the two-decimal formatter this doc comment names as the thing that collapses the bottom of the fader onto 0.
So the six-decimal serializer ships correct and unreferenced, while the write it exists to fix still rounds to two. Presumably u2-studio-gain-surface wires it — worth confirming, because the test suite here reads as though data-volume already round-trips through it.
| .map((k) => ({ | ||
| time: Math.max(0, k.time - trackStart), | ||
| volume: Math.max(0, Math.min(1, k.volume)), | ||
| volume: clampAudioGain(k.volume), |
There was a problem hiding this comment.
This is the line that invalidates a design rationale elsewhere. packages/core/src/audioLeveller.ts:15-21 justifies riding a gain node instead of a volume lane with: "volume is 0..1 and normaliseEnvelope clamps every keyframe into it, so a volume lane can only ever attenuate."
This is the only normaliseEnvelope in the tree, and it now clamps to 0..+12 dB — the same range that comment attributes to the gain node. Behaviour is fine; the stale reason is what u3-normalize-audio will be read against.
| ); | ||
|
|
||
| expect(result.success).toBe(true); | ||
| expect(capturedFilterScripts[1]).toContain("volume=3.98"); |
There was a problem hiding this comment.
3.98 sits just under MAX_AUDIO_GAIN (3.9810717…), so this pins the clamp's lower bound — revert clampVolume to min(1, …) and it reds — but leaves the upper bound unpinned on the render side. Nothing in this suite feeds a gain above the ceiling.
audioGain.test.ts:109 covers it in core (formatAudioGain(99) === formatAudioGain(MAX_AUDIO_GAIN)); the engine has no equivalent. One line closes it: volume: 99 asserting toContain("volume=3.981072") — that is formatFilterNumber's output for the ceiling.
terencecho
left a comment
There was a problem hiding this comment.
Approving. The core mechanism is right, the measurement in the description is the right kind of evidence, and CI is green across all required checks at 907362b8.
I independently verified Rames's three findings before deciding. All three reproduce, but one of them reads differently once you put it against the counterfactual — details below.
On the Studio fader (Rames's #1) — real, but a strict improvement over base
Confirmed at head: propertyPanelFlatMediaSection.tsx:216-224 is min={0} max={100} over volumePercent, writing formatNumericValue(next / 100), with disabled={volumeAutomated} sitting right there as precedent. A clip at 1.95 does arrive as value={195} on a 0-100 control, and any touch writes it back at or below unity.
The part I'd push back on is the severity framing. The counterfactual matters here:
- Before this PR,
data-volume="1.95"was clamped away on every path. The authored boost was lost unconditionally. - After this PR, it survives preview and render, and is lost only if someone touches the fader.
So this doesn't introduce a way to lose authored gain — it removes the unconditional loss and leaves one conditional one. I also confirmed the retroactive blast radius is zero: at the merge-base (f7d2260f) the only data-volume values above unity in the entire repo are 1.0, and the sole 1.95 at head lives inside this PR's own test and doc comment. No existing composition regresses.
And the write-path fix is real and in flight: #3305 fix(studio): make the volume fader tell the truth about the gain it writes is based directly on audio-gain-above-unity, and touches exactly propertyPanelFlatMediaSection.tsx plus a new packages/lint/src/rules/media.ts. That's the u2 the description promises, and it's what consumes the fader-coordinate half of audioGain.ts that's dead at this head.
One ask, not a blocker: #3305 is still a draft. If it's going to trail this by more than a release, the one-line disabled={volumeAutomated || volume > 1} stopgap Rames suggested is cheap insurance. If it lands close behind, skip it.
On the limiter (Rames's #3) — worth a correction
A limiter does exist in the codebase: audioFx.ts:288 registers one, audioFxWorklets.ts:104 implements HfLimiter, and audioFxPresets.test.ts:217 pins it last in every preset that has one. What's absent is a limiter in the default master chain, which is the accurate version of the concern.
More usefully, I traced where clipping actually lands on each path, because the thing this PR is defending is preview/render agreement:
- Render: per-track
volume=→amix=inputs=N(normalize=true, divides by N) →[mixed]volume=masterOutputGain*N. The compensating multiply ataudioMixer.ts:663cancels the amix division exactly, so a boosted track arrives at the AAC encode at its full authored gain and clips there. - Preview: clip gain node →
_masterGain(clamped to [0,1]) →ctx.destination, clipping at the device.
Both clip at full scale, so preview and render still agree — which is this PR's actual invariant. That makes the missing master limiter an audio-quality follow-up rather than a parity regression, and I'd not gate on it.
Waveform drift (Rames's #2) — confirmed
studio-server/src/helpers/waveform.ts:48-49 normalizes per asset (const maxPeak = Math.max(...peaks, 0.001); return peaks.map((p) => p / maxPeak)), and the cache key at buildWaveformCacheKey is asset path + size/mtime with no clip identity in it — so the route structurally cannot know a clip's gain. Two clips 12 dB apart off one asset draw identical. Agreed this is scope-defined-out, and agreed it's the right follow-up.
Things I'd add
1. The render-path ceiling isn't actually pinned. audioMixer.test.ts:260 is the only test covering the widened clamp, and it uses volume: 3.98 asserting toContain("volume=3.98"). But MAX_AUDIO_GAIN is 3.9810717055 — so 3.98 passes through unclamped and the ceiling is never exercised. The test pins that gain isn't lost; it does not pin that gain is bounded. Since clampVolume in audioMixer.ts:63 just widened from [0,1] to [0, 3.981], and that two-line change is the entire production delta in packages/engine, the above-ceiling case is exactly the boundary worth holding. A second case at volume: 99 expecting volume=3.981072 would close it. (I did confirm the clamp itself is correctly applied on both the static path at :178 and the keyframe path at :183, so this is a test gap, not a live hole.)
2. media.ts:308-312 — the comment now argues against the code beneath it. The comment explains that el.volume is trusted on the first tick precisely because falling back to data-volume "would clobber the GSAP-seeked position." The new line does exactly that clobbering whenever fallbackAuthorVolume > 1. It's the right trade — a clamped read is worse than a stale attribute for a boosted clip — but the comment should say so, because as written the next reader will take the > 1 branch for a bug and revert it.
3. The GSAP-tracking branch is structurally blind above unity. currentElementVolume at :280 is clampNativeMediaVolume(el.volume), and outside the probe el.volume is spec-clamped, so the "GSAP changed el.volume between ticks" branch at :314 can never observe a value above 1. For a boosted clip with a fade whose upper half sits above unity, that branch cannot distinguish "tween at 1.95" from "tween at 1.0". In practice the probed-keyframe branch takes priority and this never runs — which is why the measured LUFS result holds — but it means the fallback path silently degrades for exactly the clips this PR exists to serve. Finding 2 is the one reachable instance of it. Worth a comment noting the fallback is deliberately approximate above unity.
4. Minor: five exports are dead at this head, not one — formatAudioGain, audioFaderPositionToGain, audioGainToFaderPosition, audioGainToText, and AUDIO_GAIN_FADER_MIN/MAX are referenced only from audioGain.test.ts. All are consumed by #3305, so this is deliberate staging rather than dead code, but it's the whole fader-coordinate half of the module riding along a release early.
5. Minor: onSetVolume at init.ts:3150-3166 now ends in a synchronous syncMediaForCurrentState(), which re-runs document.querySelectorAll("video, audio") plus a per-element ancestor walk via resolveAbsoluteMediaStartSeconds. That matches what the other bridge handlers do, so it's consistent — but this is the one handler that fires continuously during a fader drag rather than once per gesture. If master-fader drags get janky on media-heavy compositions, this is the line.
Nice work on the accessor-shadow approach specifically — forwarding to the native setter so nothing outside the probe ever observes an illegal volume, and restoring in a finally, is the right shape for this, and the guard for non-DOM runtimes means the inlined htmlCompiler copy degrades to the old behavior instead of throwing.
— Review by tai (pr-review)
…unity
Review follow-up on the gain work.
The percent slider tops out at 100%, and a boosted clip renders `value={195}`
on it. Committing from that control writes `next / 100` with `next <= 100`, so
one touch silently caps the clip and drops up to 12 dB — which only became
audible damage once this stack made above-unity gain actually render. Both
panels now hold the control above unity, the way the flat panel already holds
it while an automation lane owns the level. A hold, not a fix: the dB fader
that can represent these levels replaces the control outright.
Pin the render-side ceiling too. The existing case feeds 3.98, just under
MAX_AUDIO_GAIN, which proves the clamp is not min(1, ...) but leaves the upper
bound free — remove the clamp entirely and the suite stayed green. Verified by
mutation: the new case reds without it.
Correct audioLeveller's rationale, which justified riding a gain node with
"volume is 0..1 and normaliseEnvelope clamps every keyframe into it". Both
now span the same range, so the real reason is ownership: the volume lane is
the author's envelope, and a leveller writing into it would redraw their work.
|
Thanks — all four addressed at The percent slider capping a boosted clip — you're right that this crosses from cosmetic to destructive at exactly this PR, so it should not wait for the fader. Held in both panels, not just the flat one: The render-side ceiling — added, and I checked it the way you'd want rather than trusting it: with
The reporting half (waveform normalization, the diagnosis skill) — agreed and out of scope for this rung. Both are structural rather than incidental: the waveform route is keyed on asset path with no clip identity in the request, so it cannot know clip gain at the current seam, and the skill measures the source file where |
HTMLMediaElement.volumeis spec-clamped to[0,1]. Both timeline volume probes — the preview one and the copy the renderer serializes into the page — seeded the element fromdata-volumeand sampled it back, so a clip authored above unity lost its whole boost the moment it also carried a fade: every sample read at or below 0 dB, and the mixer prefers probed keyframes over the static volume.Both probes now shadow the accessor for the duration of the probe, forwarding the clamped value to the native setter, so the authored gain survives while nothing outside the probe ever observes an illegal volume.
Measured
One 6 s composition, integrated loudness over the first 4 s:
data-volume="1.949845", no tween+5.8 dB, exactly the gain the clip was authored at. Same composition, same asset, only the probe changed.
Also in this PR
Three other owners of the same decision, each of which kept preview and render disagreeing on its own:
el.volumefor clips the transport already owned, and the next tick read that clamped product back as the clip's author gain. This is reachable from the preview volume slider.<video data-has-audio>is never scheduled as a decoded buffer source, so above-unity gain now routes through a media-element gain node — only while the audio context is running, since routing into a suspended graph would silence audio the native path is playing.Verifying
packages/coreandpackages/producersuites cover the probe, the transport and the envelope. For the end-to-end number: render a composition withdata-volumeabove 1 and avolumetween, and measure withffmpeg -af ebur128.Stack
First of three on the audio side.
u2-studio-gain-surface(the fader and the automation lane) andu3-normalize-audioboth build on this and open once it lands.