Repository navigation
Conversation
added 6 commits
October 15, 2026 00:17
… the game receives it On the headset the head pose is predicted for the instant the frame will be shown, but every object was posed for "now at update time", a varying lag behind: a moving object stepped unevenly from frame to frame and read as judder at full frame rate. The XR loop now converts the compositor's predicted presentation time (through the compositor clock's epoch, the boot, like CACurrentMediaTime) and hands it to updateXR(presentationTime:); the engine's delta is the time between two presentations, not between two updates. FrameTime (presentation time, delta, period, skipped refreshes, frame index) and FrameClock, which learns the period and counts the refreshes a frame skipped, make the frame's time. It reaches the game through setupCallbacks(frameUpdate:handleInput:), renderer.frameTime and the currentFrameTime global; gameUpdate(deltaTime:) keeps its parameter. UpdateEvent, EngineExtensionUpdateContext, RenderGraphBuildContext and RenderPassContext gain a frameTime field with a default. The MTKView and AR loops pose the frame one of the view's periods ahead of the draw, or for the host's display-link target set in predictedPresentationTime. The engine stats report the presentation delta and the skipped refreshes; the XR loop counts the frames whose drawable's presentation time differs from the prediction.
Transform keys with easing, sampled at an absolute time: a frame's pose is a function of the frame's presentation time alone, so a frame the display skipped lands where the timeline says and the frames after it step by one period again. applyTransformTrack poses an entity from a track for a frame. The tests show a repeated frame does not double-step an absolute-time animation while delta-stepped motion drifts, and, with a real renderer, that a pose the game samples in frameUpdate matches the frame's presentation time rather than the wall clock, that the update event and the extensions receive the same frame time, and that a two-period gap reports one skipped frame.
The proposal records what was seen on the headset, where the loop sampled time, the FrameTime data and how it reaches the game without breaking callers, why the game update stays before the wait, every engine path that samples time, the platform side, the tests, the headset check still to do and the change's size and risk to the foveation, temporal anti-aliasing and back-face lighting pull requests. The XR architecture doc and the starter tutorial mention the frame's time.
…nts the skipped refreshes The proposal records the viewer's drafted change (uncommitted in its working tree): the tour's pose taken at the frame's presentation time through the frameUpdate callback, and the skipped refreshes and presentation revisions in its diagnostics and run summary, which the headset check reads. It notes that the viewer's project points at the xr_foveation worktree, which must be on this branch to build.
…eaches the try branch The viewer must build against try/xr_foveation_specular_aa until this branch is merged into it, so its change — the tour's pose taken at the frame's presentation time through the frameUpdate callback, and the skipped refreshes and presentation revisions in its summary — is kept as a unified patch beside the proposal, which applies with patch -p1 from the viewer's directory, instead of in the viewer's tree.
…uest, and the viewer carries its change The proposal's status and its landing note follow the rebase onto feature/xr_headset_rendering (untoldengine#1359), which rewrote the try branch's history, and the viewer's change now applied in the viewer's own pull request.
added 16 commits
October 15, 2026 08:11
…the rendering deadline A frame submitted on time whose GPU work ends after the compositor's rendering deadline is shown a refresh late, posed for an instant already past: a hop in a moving object that neither the loop's skipped-refresh count nor the presentation revisions can see. The primary drawable's command buffer now compares its GPU end time with the frame's deadline; the misses and the longest overrun are readable (deadlineMisses, worstDeadlineOverrunMs) and logged at most once a second.
…timal input time, and a per-second late-frame summary On the headset nearly every frame's GPU work finished a fraction of a millisecond after the compositor's rendering deadline, and the ones that missed by more were the hop a moving object still showed on the tour's sweep. setRendering(.optimalInputWait(.disabled)) is the diagnostic for the cheapest remedy: the frame is encoded as soon as the update is done, so the GPU starts about the wait's length earlier, and the head pose is predicted from earlier for the compositor's reprojection to cover. The deadline counter now reports once a second how many of that second's frames were late, with the worst and mean overrun.
…r the compositor's deadline, and encoding before the wait halves it
…es meet the deadline, and not while they do not setRendering(.optimalInputWait(.always | .never | .automatic)), automatic by default: the loop keeps the wait, and the freshest head pose, while the GPU finishes before the compositor's rendering deadline; once a second had late frames (one in twenty or more) it encodes right after the update, so the GPU starts about the wait's length earlier and the compositor's reprojection covers the older prediction; it waits again once a second's least slack would have covered the longest wait skipped, with a margin. optimalInputWaitActive says what the last frame did. The proposal records the adaptive wait and the design and estimate of the true late latch (one camera buffer per frame and eye, written after the wait, read by every pass), which is a refactor of every pass for after the headset-rendering pull request.
…de and whether the frame waited
… passes of the last frame
…t for it instead of crashing setShadowSoftness dereferenced the shadow system, which the renderer creates, so a setting made with no space open — the viewer re-applies its switches, shadow taps among them, on any change — was a crash. The setting is now kept while no system exists, reported by getShadowSoftness as the system will hold it, and given to the system when the renderer creates it.
…ame's GPU work does not fit one A frame whose GPU work does not fit one refresh is shown a refresh late now and then, which hops; shown two refreshes apart, evenly, and posed for that instant, it moves smoothly at half the rate while the compositor reprojects the head in between. setRendering(.framePacing(.fullRate | .halfRateWhenLate | .halfRate)), half rate when late by default: XRHalfRatePolicy enters when, with no wait for the optimal input time left to skip, half the frames or more were late by over a millisecond on average for two seconds running, and leaves when the rendered frames' GPU work has fitted 80% of the refresh for two seconds running. While active the loop lets every other compositor frame go, its update phase marked and nothing submitted, and poses the rendered frames one refresh later than the compositor's prediction while they land a refresh late. halfRatePacingActive says the state; the per-second line adds "half rate". Eight tests cover the policy; the proposal records the pass costs the pacing answers.
…letting a taken frame go
A compositor frame taken with queryNextFrame and not submitted is never
released, and three of them abort the app ("We reached more than 3 frames
in flight"): the first pacing, which let every other frame go, crashed as
soon as it engaged. The pacing is now in the taking: with it active, the
loop waits for the last rendered frame's GPU work to finish and then for the
next refresh after that, counted from that frame's presentation instant,
before it takes the next frame. Every frame taken is submitted, nothing
queues on the GPU, and the frames land on every other refresh while one
refresh is not enough for a frame's work.
… keeps the GPU queued Waiting for each frame's GPU work to finish before taking the next frame idled the GPU between frames, and it clocked down: the Forward pass went from 9 to 12 ms, the frame no longer fitted two refreshes, and the pacing settled at three, 30 fps. The frames are now taken on the refresh grid (the last rendered frame's presentation instant and its multiples of the period) every two refreshes, or every three or four when the frame's GPU work with 3 ms of encode needs them, with the next frame's work queued behind the previous one's; the policy sets that cadence from the second's longest GPU span, and each frame is posed for the refresh the cadence lands on. The per-second line names the cadence.
…, and the late-frame line shows the GPU span and the pacing wait A frame whose GPU work with its encode sat at the two-refresh boundary had the cadence rounded up and down by turns, a second each way, and a cadence that changes every second is itself a hop. It now rises at once and falls only after two seconds running that fit the shorter one. The per-second line adds the GPU span's mean and longest and the pacing wait's mean, so a run shows where a frame's time goes.
… pose aims where the frames are measured to land Sized from each second's longest GPU span, the cadence stayed at three refreshes with a typical frame of 14 ms: a GPU used on half the refreshes runs at a lower clock, and its longest spans, 19 to 21 ms with the compositor's own work in them, never earned back the shorter cadence that would raise the clock again. The cadence now follows the mean span plus a fifth with the encode, and the pacing ends on the mean too. And the pose no longer assumes a paced frame lands at the next cadence: the frames finished 9 ms past their deadline and were shown one refresh after the prediction while the pose aimed two refreshes after it; the overrun now says the landing refresh, which the per-second line names.
…and the cadence changes only past a deadband Asked for two refreshes, the pacing waited three: the distance to the next grid instant was rounded up, and with the learned period a hair short of the compositor's, two refreshes computed as a fraction over two and became three, every frame, 30 frames a second at a cadence meant for 45. The snap now tolerates a quarter of a refresh. And with the typical frame right at a cadence's boundary, the cadence rose on a slightly heavy second and fell two seconds later, a hop every few seconds; it now changes only when the typical frame is a millisecond past the boundary, either way.
…last frame, and the late-frame line shows the loop's split Asked for two refreshes, the pacing still took three: its wait was aimed at a presentation instant on the refresh grid, and a query made there landed just after the compositor's hand-out for that refresh and blocked until the next one, a refresh lost on every frame. The pacing now waits until the cadence's refreshes after the compositor handed it the last frame, less three milliseconds, and lets the query's own block align the frames to the refreshes. The per-second line adds how long the query blocked, how long the loop took between frames, how far after the query the presentation was predicted and how far before it the deadline sat, so a run shows where a paced frame's time goes.
…art, and the line reports the presentations' spacing The compositor hands out a frame on demand (the query blocked 0.0 ms) and predicts its presentation about three refreshes after the query. A wait aimed at the previous frame's presentation instant therefore made every cadence three refreshes; a wait of a cadence less a three-millisecond lead spaced the frames 16.6 ms apart at a cadence of 20, shown two refreshes apart, then one. The queries now fall on the pacing's own grid, the instant it started from, every cadence exactly, a query up to a quarter refresh late still taking its slot. The per-second line adds the mean, shortest and longest spacing between consecutive predicted presentations, so a run reads its evenness instead of assuming it.
…past the deadline's margin, and the pose lands by that margin
The compositor's rendering deadline sits 15 ms before the presentation on the
headset, so a frame a few milliseconds past it is still shown on time: the
full-rate runs with nearly every frame a millisecond late dropped nothing, and
the heavy run's paced frames, 3 to 6 ms past the deadline, were posed a refresh
ahead of where they were shown ("landing +1"). The policy now takes the deadline's
margin into account, less a 2 ms safety (XRHalfRatePolicy.usableMarginMs):
- The pose lands on the compositor's prediction while the frames finish within
the usable margin, and as many refreshes later as the rest of the mean overrun
reaches.
- The pacing enters when, with no wait left to skip, the display skipped a
refresh for one frame in ten or more (counted in the loop from the spacing of
consecutive predicted presentations, beyond the pacing's own cadence) or one
frame in ten or more finished past the usable margin, for two seconds running.
Frames merely past the deadline no longer count: the heavy run's second before
it paced had 90 frames a millisecond past it, shown on time, and fewer hops
than one in ten are fewer than the steps the half rate adds.
The loop counts both per second (secondMissed, secondSkipped) and prints them on
the per-second line ("N past the deadline's margin, N refreshes skipped"), which
now also appears for a second with skipped refreshes and no late frame; the GPU
completion handler receives the drawable's presentation time for the margin.
The proposal's pacing paragraph describes the final mechanism (the fixed grid,
the landing, the four lessons from the device) and records the by-eye result of
the heavy run: the even 45 fps shows fewer jumps than the hopping full rate, and
a visible stepping remains, the frame rate itself; the cure for that
configuration is the frame's GPU cost, not the timing.
This was referenced Oct 15, 2026
Member
Author
|
Closed without merging here: the frame posed for its presentation time (feature/presentation_time_animation at 27e121e) is on develop since #76 (merge 0b6348e of feature/xr_shader_cost at eac4e3e, which carries #73–#75), and develop now also holds the head of untoldengine#1359 through #78 and #79. The branch stays for the upstream pull request once untoldengine#1359 lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
feature/xr_headset_rendering(untoldengine#1359): six commits on top of it, linear, rebased from the try branch with no conflicts.What was wrong
On the headset the head pose is predicted for the instant the frame will be shown, but every object was posed for "now at update time": the XR loop ran the game update before its wait for the compositor's optimal input time, on a wall-clock delta between two updates, and queried the device anchor at the drawable's presentation time afterwards. The gap between the two instants is the wait plus the encode, and it varies with the previous frame's CPU work, so a moving object stepped unevenly from frame to frame while the head did not: judder at full frame rate, which the viewer's dartboard tour showed and RealityKit did not. A skipped refresh then doubled the step, and motion stepped by the time between updates drifted off the display's timeline after it.
What a frame does now
predictTiming().presentationTime, through the compositor clock's epoch, which is the boot likeCACurrentMediaTime()) and hands it toupdateXR(presentationTime:). The game update stays before the wait, where its cost sits inside the wait anyway; the engine's delta (timeSinceLastUpdate) is the time between two presentations, not between two updates, so every delta-driven path (skeletal clips, the camera path, scripts, the physics accumulator, extensions) moves in step with the display without changing.FrameTimecarries the presentation time, the delta, the period, the refreshes skipped and the frame index;FrameClockmakes it, learning the period and counting the refreshes a frame skipped. The game receives it throughsetupCallbacks(frameUpdate:handleInput:),renderer.frameTimeor thecurrentFrameTimeglobal;gameUpdate(deltaTime:)keeps its parameter.UpdateEvent,EngineExtensionUpdateContext,RenderGraphBuildContextandRenderPassContextgain aframeTimewith a default, so every caller compiles.predictedPresentationTime, so the API is the same on every platform.TransformTrack: keys with easing sampled at an absolute time, so a frame the display skipped lands where the timeline says and the frames after it step by one period again;applyTransformTrackposes an entity from it.presentationTimeRevisions) and prints once how far the compositor clock's epoch is from the media time.Tests
Seventeen new tests: the frame clock (a steady cadence, a skipped refresh counted once and the next frame back to one period, jitter that is not a skip, a stall, a nominal period, time going backwards), the track (and that a repeated frame does not double-step an absolute-time animation while delta-stepped motion drifts), and, with a real renderer, that a pose the game samples in
frameUpdateis the pose for the frame's presentation time rather than the wall clock, that the update event and the extensions receive the same time, and that a two-period gap reports one skipped frame. The XR module builds for the visionOS simulator, and the viewer with its matching change (feature/viewer_demoin untoldengine/UntoldArcade) builds against this branch.Still to do on the device
docs/proposals/PresentationTimeAnimation.mdhas the analysis, the alternative (the update after the wait would cost a third more CPU after the optimal input time for 6.4 ms fresher input, with no margin at 100 Hz) and the headset check: the clock-base line should show only microseconds,presentationTimeRevisionsshould stay near zero, and the skipped refreshes should match the fps shortfall (6 in 100, 5 in 90).