perf(studio): measure cold load and project-open time - #2940
Draft
miguel-heygen wants to merge 5 commits into
Draft
perf(studio): measure cold load and project-open time#2940miguel-heygen wants to merge 5 commits into
miguel-heygen wants to merge 5 commits into
Conversation
Adds performance marks for boot, shell-ready, project-open-start, preview-loaded and clips-painted, plus the four measures bracketing them, and a budget module owning the load-time thresholds. The open window ends at clips-painted rather than timelineReady: setTimelineReady fires in initializeAdapter before any clip derivation runs, so anchoring there would place the derivation cost outside the measured window entirely.
Generates a real HyperFrames project at a parameterised clip and track count, rather than injecting TimelineElement objects into the store the way the existing timeline fixture does. Only a project on disk exercises the open pipeline the load gate needs to measure. Reproduces the conditions that make that pipeline expensive: most clips carry no authored DOM id, and same-kind clips share a class name. The output is not checked in. At 1,000 clips it is ~9,200 lines of fully derivable markup that nothing reads; the tests generate into temp dirs and the gate generates the size it needs at run time.
Measures cold shell time and project-open time against a real on-disk project, in both the pre-built SPA arm users get and the Vite dev arm contributors run. Reads hf.studio.* performance measures only; an absent measure fails the run rather than falling back to DOM polling. The recorded baseline settles three things. manifest-derive grows 9.6x for a 3x clip increase against a predicted 9x, so the open path is quadratic and is 66% of open time at 3,000 clips. The 7s "time to interactive" on record was the old probe's 400ms x 4 poll debounce: shell is ~46ms and a 1,000-clip project opens in ~959ms. Row virtualization makes opening 14% faster, not slower, with no overlap between the two arms' run ranges. Shell and dev-open budgets are revised down once under KTD1; they sat far enough above reality to gate nothing. The embedded project-open budget is left red on purpose until the quadratic is removed.
…urable Interleaved A/B against the merge base on an identical 3,000-clip fixture: 4,679ms versus 4,733ms medians with fully overlapping ranges. A first pass ran the arms in blocks and reported +8.77%; that was arm ordering, not cost. Both results are recorded so the method is visible, not just the conclusion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of 5 · base
mainStudio had no wall-clock measurement of the two things users feel most: how long the app takes to become usable, and how long a project takes to open. This PR adds one. It changes no product code path — the follow-up PRs do that.
Why this had to come first
The only existing studio perf gate navigates to a running Studio and then injects a synthetic clip set straight into the Zustand store, with a module-level lease that makes the real sync path no-op. It measures scroll on state that was teleported into place; the entire open pipeline is bypassed by design.
The one open-project number on record — 7,050 ms "time to interactive" — came from an untracked probe whose definition of interactive was: set the URL hash, wait for the timeline element, then poll
[data-clip="true"]every 400 ms until it held steady for four polls. That is up to 1.6 s of pure debounce baked into every reading.What lands
generateStudioLoadFixture.mjs) emitting a real on-disk HyperFrames project at any clip and track count. It deliberately reproduces the conditions that make opening expensive: only ~20% of clips carry an authored DOMid(matching a real sample of 13 of 58) and same-kind clips share a class name. A fixture where every clip had anidwould look fine and hide the bug PR 2 fixes. Output is not committed — ~9,200 lines of derivable markup that nothing reads.performance.getEntriesByName. No DOM polling anywhere: an absent measure fails a run rather than falling back to a weaker signal.The seam that looks obvious and is wrong
setTimelineReady(true)is the natural end-of-open signal. It fires insideinitializeAdapterbefore the per-clip loop runs, and its second call site is guarded such that it never fires on a cold open. It means "the playback adapter is ready", not "the timeline has clips". Anchoring there would place the entire cost PR 2 removes outside the measured window — the optimisation would have reported 0% no matter how much faster it made things.The window ends at clips-painted instead: the first animation frame after the element set becomes non-empty. There is a regression test asserting an early
timelineReadydoes not close it.Measured baseline
Three things this settled:
The embedded arm opens in 4,394 ms against the dev arm's 4,219 ms, so a production React build barely moves open time — the cost is algorithmic JS, not dev-server overhead. That weakens a rationale written into the plan, and it is recorded rather than quietly dropped.
Instrumentation overhead
Interleaved A/B against the merge base on an identical fixture: 4,679 ms vs 4,733 ms medians, ranges fully overlapping. No measurable cost. A first attempt ran the arms in blocks and reported +8.77%; that was arm ordering, not cost. Both results are recorded so the method is visible, not just the conclusion.
Gate status
Embedded
project-openat 3,000 clips is red on purpose (4,518 ms p95 against a 2,500 ms budget) and stays red until PR 2. Shell and dev-open budgets were revised down ~4× against the baseline — at 1,000 ms against a 46 ms measurement they gated nothing.