Skip to content

feat(producer): fail a render that cannot finish within its maxRenderMs budget - #6224

Open
miguel-heygen wants to merge 5 commits into
mainfrom
fix/producer-render-eta-guard
Open

miguel-heygen wants to merge 5 commits into
mainfrom
fix/producer-render-eta-guard

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What changes for users

A render request can now carry a time budget (maxRenderMs). If the render is projected to take clearly longer than the budget, the producer stops early and says why, instead of crawling until the caller's own timeout kills it. Without maxRenderMs nothing changes, so local CLI renders are unaffected.

How it works

  • One decision function (checkRenderEta) is used at two points:
    • after worker-count calibration, before any capture (skipped when there is no calibration estimate, for example explicit worker counts or short renders);
    • during capture, from the measured frame rate (needs at least 60 s and 3 frames of sample; restarts when a capture retry resets the frame counter).
  • ETA is time already spent plus remaining frames times ms per frame. It fails only when the ETA is more than 1.2x the budget, so borderline renders survive. Encode and assemble time after capture is not counted.
  • The failure is RENDER_TOO_SLOW (not retryable, owner user), with the stage, frames done and total, ms per frame, ETA and budget in the message and in errorMetadata. It is on the producer server's safe-error-code list, so it reaches the JSON and SSE error events.
  • maxRenderMs is validated on the render routes (non-negative integer, 0 means no guard).

Verification

  • 15 new vitest cases (decision function, server transport) and the capture stage test, run three times in a row each: green. tsc --noEmit clean.
  • The capture stage test drives the real streaming stage with a slow fake capture on a virtual clock: the guard stops it, and the same capture with no budget completes.
  • Mutation: widening the margin makes the decision tests and the stage test fail.

Not covered

  • The swap in renderOrchestrator.ts that turns the guard's abort into the final RENDER_TOO_SLOW error has no test (it needs a real browser render); it is covered by the type check and review only.
  • The distributed plan/chunk path does not read maxRenderMs.
  • The 1.2x margin and the 60 s sample are tuning choices, easy to change.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head 473ecff6

The guard keeps running after capture finishes, so it fails renders during encode and assemble. That contradicts the PR body's "Encode and assemble time after capture is not counted." Details are inline.

Scope

Read the full diff. Checked that every framesRendered writer and every abortSignal: executionSignal site picks up the combined signal (capture HDR, segmented, streaming and disk, plus assemble). Checked that publicMetadata reaches errorMetadata through server.ts. renderEtaGuard.test.ts and captureStreamingStage.test.ts (bun) pass locally. server.renderTooSlow.test.ts could not run here because of unbuilt workspace packages; I left that to CI, which was still in progress. No prior reviews.

— Genesis reviewer

Comment thread packages/producer/src/services/render/renderEtaGuard.ts Outdated
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 138 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 1 resolved, 0 open, 0 new · 473ecff6 → caa5b3dc

My simulated-clock repro no longer trips after capture completes, and the new test fails without the fix. Required checks are green. The non-required Fallow audit fails on 5 items in the new code: 3 unused exports or members, and CRAP 31.6 against a threshold of 30 on the guard's timer.

— Genesis reviewer

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant