Repository navigation
Conversation
added 2 commits
October 16, 2026 22:27
UntoldEngineXR.swift mixed three jobs: the headset's runtime (the ARKit session and its providers, the device-anchor cache and recovery, the lighting hooks, the spatial input bridge, the PSVR2 observers, the diagnostics), the Compositor Services loop (the layer's frames and their timing, the wait for the optimal input time and the half-rate pacing, the drawables with the capture-target rule, the foveation frames, the presentation, the deadline accounting) and the generic driving of the renderer (the update, the culling once, each eye, the depth pyramid, the stats). Only the second belongs to Compositor Services, and a second presenter (a RealityView, for Spatial Personas) needs the other two as they are. The module is now four files behind the same public class: - XRFrameSource.swift: the frame contract, modelled on the editor's VisionProFrameSource and widened with the eye's viewport, its foveation frame, the target's primary flag and the frame's timing. A source states its formats and eye size, hands out frames; a frame has an update, a wait, a drawing with one target per drawable (XREyes), a presentation. - XRFrameLoop.swift: renderNewFrame and executeXRSystemPass with the drawable and the frame replaced by the protocol, in the same order of calls, with the same semaphore and the same stats. It compiles on every platform, so it is tested on macOS with a texture-backed source. - CompositorFrameSource.swift: everything that touched LayerRenderer. - XRRuntime.swift: everything that touched ARKit and the input bridge. UntoldEngineXR keeps its surface for games (init(layerRenderer:), setupCallbacks, start, stop, runLoop, setImmersionMode, the lighting and gesture entry points, currentDevicePose, the counters) and gains init(frameSource:runtime:device:), which the layer init calls with a CompositorFrameSource; createXR takes the colour and depth formats and the eye size from the source instead of the layer configuration. initUntoldXR, public but called by nothing, is gone with the single-file structure. No behaviour change on the headset, with two details worth knowing: the rate-map parameters are copied as a frame's eyes are acquired, before the pass takes its in-flight slot, so their ring is one slot deeper (five, not four); and lastRenderedPresentation, written and never read, is dropped. Tests/UntoldEngineRenderTests/XRFrameLoopTests.swift drives the loop with frames of its own: the eyes drawn in order from one command buffer, the targets fitted to the eyes' size, the pyramid built for the display and not for a capture, a frame without targets neither drawn nor presented, a frame without a pose presented with no eye drawn, stop and the source's end both ending the loop. The architecture doc follows the new names.
…ealityKit target Stage 2 of docs/proposals/XRRealityKitPresenter.md (untoldengine#1371): a game that wants the system's space (Spatial Personas in a SharePlay session, native entities, SwiftUI attachments) puts an UntoldXRPresenterView in its ImmersiveSpace instead of a CompositorLayer. The engine renders every frame as it always does; RealityKit shows each eye on a LowLevelMesh grid displaced by the engine's own depth. Games on Compositor Services do not import the target and see no change. The target: XRRealityKitPresenter (two LowLevelTextures the eyes are drawn into, two depth textures of its own, two grids and their entities with the eye-switching material graph Resources/XRPresenter.usda; a ticket per SceneEvents.Update from the replace calls on a command buffer of the engine's queue), RealityKitFrameSource (each ticket one frame of XRFrameLoop, posed for the moment it is taken plus a lead, the eyes a nominal distance apart on the headset's pose, drawn in the ticket's own command buffer which then carries the grids' unprojection and hands RealityKit the frame on completion), XRGridUnprojectPass (the kernel, compiled from source once per device: the nearest depth of each vertex's footprint, unprojected into the origin's space, the far shell for what sees nothing, the tear mask), XRTicketMailbox (one slot, never a queue), XRPresenterStatistics, XRPresenterInput (an inverted collision sphere as the gaze's target), UntoldXRPresenterView. The seam gains XRFrame.makeCommandBuffer(for:) with a default of nil (RealityKit binds a frame's textures to the command buffer its replace calls ran on, so the loop draws in it when a frame has one), and XRImmersionMode, a name for the XR module's immersion enum that another module can spell. Tests on the Mac: the grid's topology, the projection, the mailbox, the kernel against a Swift reference over a synthetic depth, the .usda loading with its parameters, and the whole chain drawing the engine's cubes into RealityKit's textures and placing the grids. The package scheme builds for the Vision Pro simulator.
This was referenced Oct 17, 2026
Draft
added 25 commits
October 17, 2026 10:04
…ngles: a two-sided tear mask, cut in the material Stage 3 of docs/proposals/XRRealityKitPresenter.md (untoldengine#1371), the edges item. In the simulator every silhouette was saw-toothed: the grid's triangles that stretch from an object to the far shell showed the object's rim texels smeared along a sliver, and in a view shifted from the eye the frame was drawn for the sliver swings about a degree across the picture. The kernel marks both sides of a depth jump, against the eight neighbours: the mask's x is 1 on the far side (a vertex further from the eye than a neighbour by more than the threshold), its y on the near side. A stretched triangle has both above zero everywhere inside it; a triangle of the object's own surface has x = 0 and one of the far surface has y = 0. The material (XRPresenter.usda) reads the mask from the second uv set and multiplies the opacity by 0 where min(x, y) exceeds 0.001, so the stretched triangles go while the object and the surface behind it keep their outlines up to the jump. The cut is at an epsilon, not at the mask's midpoint: the mask is interpolated linearly in distance along the sliver, whose midpoint lies 50 m out where the sliver already projects to nothing, and a (1 - mask) factor against the 0.5 threshold changed nothing in the picture. The diagonals are in the neighbourhood because a cell's two triangles share its diagonal, and a jump along that diagonal alone left a hairline unmarked. Simulator, the example scene: the teeth are gone from the slab, the cubes and the sphere; what remains is the grid's own staircase at the outlines, one cell (7.5 texels) at 256 x 240 and half that at 512 x 480, with no measurable difference in the numbers on the Mac (replace 8.1-8.4 ms, frame 3.1-3.5 ms, GPU 0.3-0.5 ms with either grid). The default stays 256 x 240: at 512 x 480 the smaller footprint leaves one-texel rows of the sphere's anti-aliased edge, whose colour is above the alpha threshold where the depth holds nothing, on the shell as hairlines; that belongs to the footprint rule. The kernel test keeps its planes and hole and checks both channels: the far plane's side of the seam and the shell inside the hole carry x, the near plane's side and the planes' rims carry y, nothing inside a plane carries either. Its threshold is 1 m, above the 0.56 m a diagonal step covers on the oblique far plane at that coarse grid's corner.
…very texel: a two-pass kernel Stage 3 of docs/proposals/XRRealityKitPresenter.md (untoldengine#1371), the footprint item of the edges work. The kernel took a grid vertex's nearest depth over up to 4 x 4 taps at fixed offsets of the cell, so texel rows and columns went unsampled (every other one at 256 x 240) and adjacent footprints did not meet; and every vertex recomputed its eight neighbours' depths, 9 x 16 texture reads per vertex. The kernel is now two dispatches on one compute encoder with a memory barrier between them. The first writes each vertex's nearest reverse-Z depth over its whole cell plus a one-texel margin into a scratch buffer of one float per vertex, which the presenter owns per eye and reuses across frames (the tickets' command buffers run in order on the engine's queue); the bounds are taken with a tolerance of a thousandth of a texel, so a cell edge that lands on a texel rounds the same way on the GPU and in the Swift reference. The second reads its own and its neighbours' depths from that buffer and writes the position, uv and tear mask as before. Adjacent footprints overlap by two texels, so a thin feature or an edge between two vertices is caught by one of them. Reads per vertex: about 140 texture reads once at 256 x 240 (64 at 512 x 480) plus nine buffer reads, against 9 x 16. The uniforms keep their 160-byte layout; `encode` takes the scratch buffer. The kernel test's Swift reference follows the new footprint texel for texel and checks the first dispatch's output too; a new test draws a one-texel line at every texel column and row of the texture and checks that one of the two grid vertices it lies between reads it, that the first dispatch reads it at exactly the vertices whose footprint holds it, and that the vertex is placed on the line. Simulator, the example scene: at 256 x 240 the sphere's outline loses its one-cell bites (a vertex whose footprint missed the edge) and the slab's staircase is as before; at 512 x 480 the staircase halves, as before. The numbers do not move on the Mac: replace 7.9-8.2 ms, frame 2.9-3.4 ms, GPU 0.3-0.5 ms at either grid. The dashed hairline above the sphere at 512 x 480 stays, and it is not a sampling gap: the eye's textures and grid dumped from the simulator have no texel with colour and no depth, every texel lies in some footprint, and a software rasterization of that grid from the simulator's viewpoint draws no hairline. In RealityKit it is one pixel row along the grid row above the sphere's top, with the near-side mask and fully opaque sphere texels; it is there with the eye at the camera (no parallax), goes with a far shell of 50 m or less or a cut epsilon of 0.01 or more, and moves with 481 rows. That is how RealityKit interpolates a stretched triangle's attributes between a vertex at 1.6 m and one at 100 m, and belongs to the shell and cut settings of the edges work, not to the footprint.
…ices, the diagonal split, the skirt Stage 3 of docs/proposals/XRRealityKitPresenter.md (untoldengine#1371), from the headset run of 2026-10-18. The surfaces were steady on the Apple Vision Pro; every silhouette was a staircase of cell-sized steps that crawled with the predicted pose's jitter, and a sphere was cut inside its outline. The kernel's first pass now places each vertex on a texel instead of at its cell's centre: the texel nearest the vertex's own centre among those on the near side of the footprint's depth jump (within the tear threshold of the nearest depth, in metres along the vertex's ray), so a vertex just outside an object lands on the object's edge texel whatever the surface's slope, and one on a continuous surface stays at its centre. The footprint's margin is half a cell, so the first vertex outside an edge always reaches it and the second never does. A one-texel hysteresis against the previous frame's texel takes the pose jitter out of the choice. A third pass writes each cell's two triangles per frame through LowLevelMesh.replaceIndices(using:), carried in the ticket: the cell keeps the fixed diagonal unless that one crosses a depth jump and the other does not, which removes the notch where a silhouette ran corner to corner against the fixed split. A rim vertex (the texel under its own centre lies off the near surface) is pushed three texels outward past its edge texel (XRRealityKitPresenterConfiguration.skirtTexels), so the silhouette seen is the alpha test's crossing of the filtered texture rather than the mesh boundary through rim texels; the material divides the colour by the alpha first, since the pixels shown between the rim texel and the crossing would otherwise draw a dark line along every edge. The tests follow the rules texel for texel: the two planes and the hole, the one-texel line, every cell's split and winding, the uniforms' layout (padding became skirtTexels), the graph with its two new nodes. In the simulator the staircases and the notches are gone and the slanted edges are smooth filtered edges.
… over the alpha-tested core The engine's XR frame leaves binary alpha at a silhouette under FXAA (one texel per edge at a constant 21/255) and SMAA (none); only MSAA leaves coverage (quarter steps), measured by a probe in the chain test that reads the left eye's alpha along a row through the cubes in mixed immersion, mode by mode. So a second entity per eye draws the same grid with the graph minus the opacity threshold (Resources/XRPresenterEdge.usda), blended and without depth writes, over the opaque core, whose threshold rises from 0.5 to 0.9 so that only fully covered texels are opaque and the band blends the rest (at 0.5 the core took every texel covered by half or more and the band had only the quarter-covered ones, at a quarter strength). `edgeBlending` in the configuration, on by default; `drawsEdgeBand` says whether the graph loaded. The cell split is a setting too: `adaptiveDiagonal`, off by default until the headset judges it (the per-frame index buffer rides in the ticket only when it is on). The simulator cannot show the band: the engine renders single-sampled there, so its alpha is binary in every mode. The headset judges it. 16 presenter tests pass on the Mac.
…kirt, tear, diagonal, band, lead `XRPresenterSettings` (skirt texels, tear threshold, adaptive diagonal, edge band, lead) is the presenter's `settings`, the configuration's values to start with and a game's to change while the space is open. Each ticket carries the settings in force when it was posted, and the engine's thread reads a frame's rules from its ticket only (`RealityKitFrameSource`), so a change never races a frame. The edge band's entities are made whenever their graph loads and follow the setting on every update, ticket or not (`hasEdgeBand` says the graph loaded, `drawsEdgeBand` that it is shown). The index buffers of a split frame start from the fixed split, blitted from a buffer made at init (RealityKit hands them out with no content, and a frame that draws nothing still completes them), and the same blit writes the fixed split back once when the adaptive diagonal is turned off, so no cell stays split as the depth last left it. Why: launching the example with arguments from the Mac fails most of the time (devicectl's connection), so comparisons on the headset need the settings in the window. The configuration's note on the adaptive diagonal no longer cites the simulator pictures (they came from a build without it); it is off by default until the headset judges it. A test takes the tickets itself: the settings ride in them, the split frame's buffers take a written pattern, the restore frame puts the fixed split back, the band's entities follow the toggle. 17 presenter tests pass; the change was reviewed under three lenses with each finding verified, and the four confirmed ones are in.
…and a RealityKit picture of the grids tested against the engine's Each grid vertex carries its uv times its eye-space depth and that depth (a packed float4 at offset 24, stride 40, attribute uv2); both graphs divide them per pixel before the texture read. Both are affine in the point, so RealityKit's perspective-correct interpolation reproduces the engine's projection on every planar triangle, whatever RealityKit's own camera: an affine uv keystones across a cell whose vertices sit at different depths, by up to a third of a texel on a rim-to-face cell, differently cell by cell. The affine uv0 stays for the fallback material. The configuration takes a view offset of both eyes' viewpoints (zero in use): a deliberate few centimetres make RealityKit show the grids from a point the engine did not draw them from, the parallax test of a silhouette defect. Tests: the kernel test checks every vertex's projective uv against its position's eye-space depth; and a first end-to-end test renders the presenter's grids with RealityKit on the Mac (RealityRenderer, a perspective camera at the engine's eye through a symmetric frustum, the eyes at the device's point, mixed immersion) and compares the picture with the engine's eye texture: identical, 88 dB. With an opaque engine background the same test shows the halo of background lost around a silhouette where the stretched triangles are torn away, one cell wide (26 dB): a later stage's. 18 presenter tests pass.
…i-angular rasterization rate map, at the same pixel count RealityKit gives the presenter no gaze, so the compositor's foveation cannot be had; what reads as soft on the headset is the texture's density: 1920 uniform pixels over 2.6 of tangent are 12.9 texels per degree at the centre of view, where a perspective render is sparsest, against a display of some 34. A FIXED, head-centred rasterization rate map, made once by the presenter, spends the same physical pixels evenly in angle instead: h(x) = 1 / (1 + x^2) per axis over the frustum tangents, 64 samples per axis. Metal rounds rates up toward quality, so the screen is scaled down, axis by axis, until the map's physical size fits the configured pixel budget; the textures and the depth are made at the physical size, so the engine's GPU frame is unchanged. On this Mac (Apple M4 Max), for the default 1920 x 1824 budget and the tangents 1.30 1.30 1.15 1.30: screen 2618 x 2429, physical 1920 x 1824, granularity 32 x 32, screen scale 1.36 x 1.33 (the centre of view 1.36 times denser, 12.9 -> 17.6 texels per degree; Metal's rate rounding costs 4 % of the ideal 18.3), 64 samples per axis, 24624 bytes of parameters, grid 241 x 229. - XRRealityKitPresenterConfiguration: `foveation` (none, the default so the headset can compare, or equiAngular; `textureSize` is then the PHYSICAL pixel budget per eye) and `foveatedCellTexels` (8). - XRPresenterFoveation (new): makes the map, fits the screen to the budget, takes the first cell of 8 or 4 texels that divides `physicalGranularity` (else the presenter draws uniformly and says so), copies the parameters once and keeps one XRFoveationFrame, its viewport the whole screen, for both eyes. - XRRealityKitPresenter: the map first; textures, depth and grid from its physical size (a vertex every cell, the last on the last texel); `foveation` and `textureSize`. - RealityKitFrameSource: `eyeSize` is the physical size (the loop sizes the renderer's targets from it); each XREye carries the map with the screen-space viewport, so the loop's scene passes draw through it and the final pass copies physical pixels one to one, as on the compositor path. - XRGridUnprojectPass: the uniforms gain float2 screenSize, float cellTexels and uint foveated (160 -> 176 bytes). The kernels are specialized on a Metal function constant, kFoveated: the foveated kernels take the map's parameters at buffer 3 and map every texel that becomes a ray (the vertex's own, its eight neighbours', a cell's centre) through the decoder's map_physical_to_screen_coordinates(texel + 0.5) over the screen; the vertices sit on min(gid * cell, size - 1), the map's tile knots, where its mapping changes slope, so the grid interpolates it exactly within a cell; the uv written is (placed + 0.5) / size, so ray, uv and the depth read coincide. The uniform kernels are bit-identical to before: checked byte for byte (vertices, scratch, indices) against the previous kernel source on three grids, four settings, two frames each. encode(...) takes rateMapData: and cellTexels:. - Tests: through a map, the kernel's vertices lie on the ray of rateMap.screenCoordinates(physicalCoordinates:) at a tilted plane's depth within 1.2e-6 m (1e-4 asked); the grid's linear interpolation of the shipped map between vertices meets the map at the cells' midpoints within 0.04 screen px (0.3 asked); the shipped profile never exceeds the budget; the chain through the loop with the map (the eyes' map and viewport, the physical textures, the scene passes, the grid's placement and half-texel uv); the reference helpers take the placement and half-texel convention; the uniforms layout. 24 tests of the presenter and foveation filters pass.
…ll, and tells the projective uv from the affine one A review of the previous commit confirmed the mathematics, the kernel and the vertex layout and faulted the end-to-end test four times, each verified: RealityRenderer's output carries alpha 1 everywhere, so the "drew the grids" guard held for an empty picture; nothing checked the engine's picture; a stalled loop became a skip; and the old affine graph passed the 30 dB threshold (64.6 dB measured). Now both pictures must hold coloured pixels, the ratio is finite, a stalled loop fails, the cubes are turned 35 degrees so that their faces run across cells at changing depth, the cells are four texels, and the threshold is 70 dB: the projective graph scores 90 dB (identical at 8 bits), the affine one 58.4 dB on the same scene. The turned scene also showed the absolute tear threshold at the test's scale: one texel is 39 mm at the cubes, a turned cube's grazing side face changes its distance to the eye by half a metre between neighbouring vertices, and the 0.5 m threshold tore it (RealityKit lost a one-cell strip of the face). The test's threshold is one metre; the headset's 2 mm texels never reach that on such a face, and a relative threshold stays on the plan for far oblique surfaces.
… into feature/xr_presenter_rate_map
… grids at the map's screen size against the engine's uniform frame of that size The picture test's chain (the engine's eye drawn by the loop from the picture pose, RealityKit's rendering of the grids through a 90-degree camera, the coloured-pixel count, the PSNR, the PNG dump) moves into helpers, and the original run is unchanged (90.0 dB, threshold 70). The second run draws the eye through the presenter's equi-angular rate map (budget 256 x 256, screen 296 x 296 on this Mac, cells of four texels) and renders the grids at the map's SCREEN size, against the engine's uniform frame of the screen size from a second presenter: through the map the grid puts every physical texel back on its ray, so RealityKit shows the scene as the uniform frame draws it. The cubes' bounds agree within three pixels (a periphery texel is two pixels wide, the alpha ramp's foot one more; measured 2, 0, 0, 0), and the pictures agree at 39.8 dB (threshold 30): the foveated texture is another texel layout, magnified at the periphery, so edges there are a pixel softer than the uniform frame's. A mapping of texels to rays that took the physical texel for the screen pixel would move an edge by up to a quarter of the screen and score near 21 dB.
…of its own loop, so the next session keeps its camera A space closed and opened again drew nothing. The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (the eye textures kept RealityKit's uninitialised content). `XRRealityKitPresenter.stop(after:)` hands the shutdown a once-only completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires, or after a second if no frame comes. The view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)` after cancelling the update subscription. A Mac test closes a session that way (the reset emulated with the engine's own deferred destroyAllEntities and the camera cleared in its completion), makes the engine's camera and the scene again and runs a second session: the same picture as the first, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. 27 presenter and foveation tests pass.
…ation itself and keeps posting until the shutdown completes A review of the previous commit found the window it left: the engine finalizes the destroys at the start of a frame only when a frame's completion handler asked for it, and the last regular frame's completion may land after the close (a few milliseconds of every refresh), or a loop loading assets skips the frame's update; the closing frame then did nothing and the one-second fallback stopped the presenter with the destroys and the completion still pending, for the next session to inherit. `stop(after:)` now sets `needsFinalizeDestroys` itself before each ticket and posts one per refresh until the completion fires, ninety at most. The test clears the flag before closing, the unlucky case, and the second session still draws the first's picture (83 dB).
…se, and the presenter outlives its own close Two more windows the review of the close found. The closing presenter was held weakly by its completion while the view dropped its reference at once, so the exchange could end on nobody: `stop(after:)` now keeps the presenter alive until the end, which it reports through a `then` callback on the main actor (the hand-off from the engine's thread goes through the main queue directly, not a task). And a session opened within the second a close may take would have inherited the destroys and lost its camera: the view's `close()` keeps the close as a task the next `open()` awaits. The test awaits the reported end before opening its second session.
…to leave behind A close has to give up after its second when its loop already ended (a game stopped the engine itself) or a scene is still loading; the old session's destroys and the reset's completion then stay pending for the next frame, which would be the next session's first, and that session would lose its camera. The engine gains `finalizeDestroysNow()`, the deferred destroys and their completions finished at once under the world gate, for a caller that begins a session after the loop that would have finished them has ended; the presenter view calls it before making the engine of a new session. The Mac test closes a session whose loop had already ended, sees the close end by its fallback, and the session after it draws the picture.
…long silhouettes, on a mesh of its own per eye On the headset the band cost about 11 ms of GPU a frame (27 ms with it, 16 without, against an 11.1 ms frame): a second blended entity per eye over the whole grid, so RealityKit shaded every cell twice at the display's resolution, while only the cells along a silhouette have anything to blend. The band now has a LowLevelMesh of its own per eye (`XRPresenterGrid.makeMesh` with the grid's vertex layout and index capacity, starting from every cell degenerate), whose vertex buffer the unprojection's second pass writes alongside the core's (a second output buffer at index 4 under a function constant `kEdgeBand`, so the kernel without a band is what it was; the same bytes) and whose index buffer a fourth pass writes per frame, one thread per cell: the cell's two triangles, split as the core's are (the diagonal rule is one static function for both passes), when any of its four vertices is a rim vertex (the scratch entry's w) or carries a tear-mask channel (the near or far side of a depth jump, read from the core's vertices after a barrier), else six copies of the cell's first index, a pair of degenerate triangles the rasterizer draws nothing for. The ticket carries the band's vertex and index buffers (`replace(bufferIndex:using:)` and `replaceIndices(using:)` on the band meshes while `settings.edgeBlending` is on); each band index buffer starts from a degenerate blit, as the core's starts from the fixed split, so a frame that draws nothing (no pose, the stop path) leaves the band empty; the presenter keeps the band's entities on the band meshes, enabled by the setting as before. The uniforms are unchanged. What the band keeps: on the kernel test's two planes and a hole (33 x 25 grid), 146 of 768 cells, 19.0 % (the seam's three cell columns and the hole's rim; the planes' interiors, the hole and the corners are degenerate, and the fixed diagonal keeps the same cells with the fixed split); on the chain test's three cubes (65 x 65 grid, the picture configuration), 280 of 4096 cells, 6.8 %, in each eye. On the headset's 256 x 240 grid the example's silhouettes should keep a few per cent likewise, so the band's second shading falls to that fraction of the 11 ms. Tests (31 of the presenter and foveation suites pass): the band's index buffer against a Swift reference on the two planes and the hole, and the band's vertices as the core's bytes; the full chain carrying the band buffers in the ticket, the kernel's rule verified on the scratch entries and the vertices of both eyes, and the band meshes holding what the kernel wrote; the settings test sees the band's buffers leave the tickets and return with the setting, and the degenerate start of a frame without the kernel; a new picture test under MSAA renders RealityKit's picture with the band's cells, with the band's entities put back on the whole grid (as before this commit) and with no band: the cells and the whole grid agree at 90.0 dB (identical at 8 bits, so the cells cover every partly covered texel, 253 of them), both match the engine's MSAA eye at 90.0 dB, and no band scores 36.4 dB against them. The two existing picture tests are unchanged (90.0 dB and 39.8 dB). Limit: a feature thinner than a cell that puts no grid vertex off the surface (a one-texel hole, a two-texel slit between vertex rows) marks no rim and no tear, so its ramp is not blended; the core shows it alpha-tested, as before.
…'s coverage, not the premultiplied fraction
A mixed-immersion frame is premultiplied: a silhouette texel the multisample resolve partly
covers holds its coverage as alpha and the face's colour times that coverage. The look pass
applied the grade and the tone map to that fraction and passed the alpha through, so the
compositor showed T(a·c) where a·T(c) was meant: on a lit face the tone map's shoulder made
the rim brighter than the face (a bright pulsing rim along near-horizontal edges on the
headset); its toe would make a dark face's rim darker.
Now the look pass divides the scene's colour by its coverage before the grade, the tone map,
the grade LUT and the primaries conversion, and multiplies it back after; the alpha goes out
as it came. The coverage is the texel's alpha, or what is left of it behind the splat layer
when one is out. Under a floor of 1e-3 the texel is graded as it is: a transparent one stays
(0, 0, 0, 0), and nothing divides what is not there.
lookColor is a template on that: fragmentLookPremultipliedShader and
fragmentLookPresentPremultipliedShader instantiate it on, as the pipelines
.lookPremultiplied / .lookPresentPremultiplied the pass picks in mixed immersion; the opaque
pass's functions instantiate it off and are the code they were. A template rather than a
function constant because the front end folds it before it optimizes: with a function
constant (folded by the driver on optimized code) or a live branch, the opaque look pass
itself moved by a half-float ULP on 0.02 % of a frame's texels (alpha 1, grade on). With the
template the look pass is bit-identical given its input: on LookPresentFold's lit cube over
the IBL at 1920 × 1080, for ACES / AgX / none × grade on and off × MSAA / FXAA / no AA, its
output matches the original's wherever its input does (8 of 18 runs whole; in the other 10
the scene composite the pass reads already differs at the same 166 texels of 2 M, by one or
two half ULPs, in the non-MSAA modes only — a codegen shift in a pass upstream of the look
when the one kernels translation unit is rebuilt with the new functions; no output texel
differs that its input did not). The PostFX reference tests (Bloom, ChromaticAberration,
ColorGrading, DepthOfField, FXAA, MSAA, SMAA, SSAO, Vignette) pass as they are.
The new test renders the XR path in mixed immersion through the RealityKit presenter's chain
on the Mac with MSAA: a white cube facing the headset, rolled 3° so its edges cross the rows,
under a sun 30° above its normal, into a 640 × 608 eye texture; it reads the eye texture back
and compares each partly covered edge texel's luminance behind its coverage with the face's
one row in. Before, over the face (linear luminance 0.845):
no tone map quarter 0.996 half 0.994 three quarters 1.004 (the control)
ACES quarter 1.646 half 1.460 three quarters 1.204
AgX quarter 2.715 half 1.693 three quarters 1.247
After, the same 69 / 55 / 67 rim texels per coverage (the alpha is the resolve's, unchanged):
no tone map quarter 0.996 half 0.994 three quarters 1.004
ACES quarter 0.992 half 0.990 three quarters 1.003
AgX quarter 0.996 half 0.995 three quarters 1.000
The kernels are rebuilt (buildkernels.sh) for every platform, with the source hash.
…_grade' into feature/xr_realitykit_presenter
…es from 8 × 8 block maxima and scans the rest in a dense dispatch, the same entries The nearest-depth dispatch read every vertex's footprint, some 240 texels twice, and was most of the presenter's GPU time on the headset (4-5 ms of the 11.1 ms frame at 256 × 240). It is now three dispatches that write the same entries: - Pass 0, xrGridBlockNearest: the nearest depth of each 8 × 8 block of the depth, one thread per block, sixteen gathers of a quad each (a read per texel took ten times as long), the blocks at the texture's edge cut to the texels there; it also resets the scan list's header. - Pass 1, xrGridNearestDepth: a fast path. The nearest depth over the blocks that cover the footprint is at least as near as the footprint's own, so its near-side limit along the vertex's ray is at least as strict; when the texel the scan would choose first (the nearest integer texel, the lower one on an exact half, as the scan's row order picks it) and the rim test's centre texel both pass it, the scan's answer is known: that texel, no rim, and the hysteresis holds the previous texel when it passes the same limit. Otherwise the vertex is appended to a scan list (one atomic add per SIMD-group; an atomic max keeps the scan's threadgroup count in the list's header). - xrGridScanDepth: the footprint scan of the listed vertices alone, dispatched indirectly to the list's length, sixteen lanes per vertex (one quad-row each), the footprint read two texels by two through gather with per-texel masks, the lanes' maxima and best texels folded with SIMD shuffles under the order (distance, row, column), which is the texel the scan in row order chose first. Running the scan inside pass 1 made a SIMD-group of 32 vertices pay the whole scan whenever one of them needed it (1 % of the vertices cost a quarter of the scan); one thread per listed vertex left a short list's dispatch as long as one thread's walk of a footprint. XRGridUnprojectScratch bundles the three per-eye buffers (the entries, the block maxima, the scan list) with the grid and the texture size; encode takes it in place of nearestDepths and grid; the presenter keeps one per eye. The uniforms keep their layout: flag bit 1 lists every vertex and scans it, for the tests to compare against. Threadgroups are capped at 256 threads. Measured on a Mac (Apple M4 Max), GPU timestamps on a command buffer holding the unprojection alone, 60 frames after four of warm-up, 1920 × 1824, grid 256 × 240, the edge band on and the fixed diagonal, the two planes and the hole (test_theKernels_atTheHeadsetsSize_timedByGPUTimestamps): before, median 0.689 ms (min 0.603); after, median 0.045 ms (min 0.043), 98.9 % of the vertices settled from the blocks. A standalone bench on other depth contents, the old chain against the new: a far plane with a disc, a slab, a small disc and a pole 0.60 -> 0.047 ms (2.8 % scanned), a checkerboard of 100-texel squares 0.62 -> 0.083 ms (16.5 % scanned). The Mac's milliseconds are not the headset's; the ratio should carry over. Tests: every kernel test passes with its assertions unchanged (entries and vertices against the Swift reference, the rate map's tilted plane too). New: the block dispatch on a 37 × 29 depth with a value per texel, the partial edge blocks included; the fast path against the all-listed mode over three frames (a one-texel line a texel off a vertex column, then without it so the hysteresis holds the column on the line's texel, then a steady frame), entries, vertices, core and band indices byte for byte the same, 93.2 % of the vertices settled on that scene; and the measurement above, printed, asserting only that the frames complete and that most vertices are settled.
…PU time and the thermal state beside the ticket's GPU span The presenter's GPU number was the ticket command buffer's span, gpuEndTime - gpuStartTime, and on the headset that is not the frame's cost alone: the GPU is shared with the compositor and with RealityKit's own drawing of the grids and the edge band, whose work inside the span counts too, and the span stretches when the GPU runs at a lower clock (a warm device after hours of use, a lighter load). The same code read 3.8 ms in a Debug build and 9.5 ms in a Release build at 01:45 on an Apple Vision Pro, then 2.9 and 3.6 ms the next morning with the settings unchanged; no path of the band, MSAA or the timing is gated on the build. GPUPassTiming now also measures the frame's busy time, the union of its stages' spans from the counter samples (busy()), which is the GPU's time on the engine's own passes whatever else ran in between; the presenter's unprojection encoder is made through it, so its passes are timed with the rest. XRPresenterStatisticsSnapshot carries that time (engineGPUMsAverage, zero while the pass timing is off) and the device's thermal state, and its line names the span as a span. Tests: the union of overlapping, touching and apart intervals; the busy time within the frame's span and no more than the stages added up; the snapshot's fields and line.
…rough the core's opacity cut and the band's weight: the shipped cut and band stay A top face seen as a sliver one to two texels tall, its far edge against the passthrough crossing the texture's rows at a shallow slope (the example scene, the eye 35 mm above the middle cube's top, the head rolled by atan 0.04), drawn through the presenter's grids by RealityKit from the engine's own eye at four picture pixels per texel, over a black and over a white room so that every pixel's opacity and straight colour are known. The edge's position per column is read where its opacity crosses one half and as the opacity missing above it, a line is fitted per stretch (the middle cube's far edge and the two far edges of each turned cube) and the deviation about it reported in eye texels, with the edge's move against a reference and the opaque pixels drawn over the room. The texture's own coverage edge deviates 0.074 texel rms (the quarter steps of 4x MSAA), its staircase read as a hard edge 0.29. Through the shipped core cut of 0.9 and the band at coverage the edge deviates 0.066, the same as the band over the whole grid and as the band without any core, with no move and nothing opaque over the room; the cut alone, without the band, 0.22. Cuts of 0.5, 0.25 and 0.1 read 0.09, 0.13 and 0.22 and grow the object over the room by 0.2 to 0.6 texel (opaque pixels where the coverage shows mostly room from 0.25 down); the band at twice the coverage 0.12, at full weight (every touched texel opaque) 0.30 with a halo of 0.9 texel. The rim pixels' colour behind their opacity is the face's (0.98 of it, as in the texture) in every blended variant. So neither a lower cut nor a heavier band straightens the edge: the blend filters the staircase and the cut only keeps the fully covered texels opaque and depth-writing. The defaults stay; the numbers are recorded in the core graph's header, and the test asserts that the shipped configuration stays at the coverage's floor, that the cut alone shows the staircase, and that no variant beats it.
…me latency, off by default; the numbers carry the lead in force RealityKit gives no presentation time, so the presenter posed each frame a fixed lead past the moment its ticket was taken (two refreshes at 90 Hz). The plan's stage 3 asks for the lead to become adaptive: the ticket's age at pickup plus the engine's frame time of the last frames, plus a constant for what follows the frame's completion. XRPresenterLead holds the estimate. Fixed: age + leadSeconds, as before. Adaptive: age + the frames' time from pickup to the ticket's command buffer completing on the GPU (the CPU frame, the GPU's, whatever waited in between), averaged exponentially with an eighth on the latest frame, + displayLatencySeconds; the fixed lead stands in until a frame has completed; a frame's time is clamped to 0.1 s before it enters the average and the adaptive lead to the same range. XRRealityKitPresenterConfiguration and XRPresenterSettings gain adaptiveLead (false: nothing changes until the headset confirms) and displayLatencySeconds (16 ms: the fixed 22.2 ms less the headset's Release reading of 2026-10-19, age 0.03 + CPU 2.8 + GPU 3.6 = 6.4 ms, rounded, so the two leads agree at those numbers; the headset calibrates it with a head-locked marker). The settings travel in the ticket like the others; the engine's thread reads only the ticket's copy. RealityKitFrameSource keeps the estimate under a lock, notes each drawn frame's completion from the command buffer's GPU end time, and poses the frame for the ticket's posting plus the lead. XRPresenterStatisticsSnapshot carries leadMsAverage/Max (the lead in force, from a ticket's posting to the instant its frame was posed for) and completionMsAverage/Max (pickup to completion, the term the adaptive lead averages); the line says "to completion ... lead ...". UntoldEngineXR.devicePose(forPresentationAt:) (over a side-effect-free XRRuntime.devicePose(predictedFor:)) gives a game the pose the frame being updated will be posed by, for something placed on the head: the example's swim probe uses it for its head-locked dot. Tests: XRPresenterLeadTests (the fixed lead whatever the history, the fallback before a frame completes, the average and the display latency, a stall clamped and forgotten, the range, the default constant making the leads agree at the headset's numbers); the statistics' new fields and line; the settings through the ticket; a frame-source test of the presentation time under the fixed and the measured lead; the chain test checks the completions reach the average. The XR presenter render-test filter: 45 tests, 0 failures.
…s CPU frame, its wait for the GPU and its GPU span, count the frames over the refresh and keep the worst frame's split The loop tells the frame it is about to commit its command buffer (XRFrame.willCommit, nothing by default), and the RealityKit frame notes the instant there and adds its completed handler with it: cpu is the pickup to the commit, wait the commit to the GPU's start (the driver's submission and the GPU's queue, where the compositor's and RealityKit's own work go first), gpu the span, and the three sum to the completion (a GPU that reports no times leaves the rest in the wait). The statistics keep each stage's average and maximum, the refresh period measured from the update rate (100 updates/s is 10 ms, 90 is 11.1; nothing assumes 90 Hz), the frames whose completion exceeded it, and the stages of the longest frame of the span; the log line says "worst 16.6 = cpu 3.1 + wait 9.2 + gpu 4.3". frameMs is now pickup to commit (it was pickup to the end of the presentation, microseconds apart); a frame cleared for want of a pose enters the stages too, though not the adaptive lead; the late count's list of completions is bounded at 65536 a span. Tests: the stages from the instants, each stage's maximum and the worst frame, the late count against 100 and 90 updates/s with an injected clock, the bound, the loop's hook order, and the frame source's stages on the Mac.
…ages and count the late ones, as the RealityKit presenter's numbers do The Compositor Services path could not be read against the RealityKit presenter's numbers line: the engine's stats kept the GPU time beyond the previous frame's and nothing of a frame's wait for the GPU or of its raw span. Now `EngineStatsSnapshot.frameStages` carries, over each span of the loop's frames (a second by default), the raw GPU span's maximum, the frames late against the refresh period measured from the frames' cadence, the frames past the presenter's rendering deadline, each stage's maximum and the worst frame's split: "gpu span max 20.0 ms · late 3 of 100 (refresh 10.0 ms) · worst 24.8 = cpu 4.0 + wait 0.1 + gpu 20.0", in both the compact and the expanded styles and through `formatEngineFrameStages`. `EngineFrameStages` mirrors `XRPresenterFrameStages`: the CPU frame from the loop beginning the frame to committing the display's command buffer, less the pacing (the wait for the compositor's optimal input time, which the RealityKit path has not), the wait from the commit to the GPU's start, the GPU's raw span; the three sum to the completion. `XRFrameLoop` stamps the frame's beginning, the pacing and the commit instant and carries them by value into a completed handler on the primary target's command buffer; a capture target's buffer records nothing. `gpuExecutionMs` is as it was. Everything is under ENGINE_STATS_ENABLED, as the rest of the stats. Tests: the stages from the instants, the spans closed by the loop's timestamps with the refresh from the cadence (100 frames in a second: 10 ms, 90: 11.1 ms), each stage's maximum and the worst frame, the deadline count, the bounded late count, both formatter styles, and the loop test that the display's frames enter the stages and a capture target's do not.
…its the main actor: the runtime calls it from its session monitor's task CompositorFrameSource.init is @mainactor and formed the runtime's presenterStateDescription closure there; with the hook typed as a plain () -> String the closure inherited the actor, and the first anchor-diagnostics print from XRRuntime.startSessionEventMonitor's task tripped the runtime isolation check (EXC_BREAKPOINT in dispatch_assert_queue) on the headset, killing every Compositor Services app on the seam at launch. The hook is @sendable now, and the source answers through itself (it is Sendable) with the layer's state, as the loop reads it from its own thread. A visionOS-only test pins the type.
added 10 commits
October 19, 2026 12:54
…out of the CPU frame, as they leave the wait for the drawing time On the headset every frame counted late (100 of 100, the worst frame's CPU stage a whole refresh): beginDrawing's queryDrawables blocks until the compositor hands out the drawables at its phase of the refresh, and that wait sat inside the CPU stage. The pacing now holds both waits, so the CPU stage is the frame's work (the update and the encode) and the late count against the refresh means the same as the RealityKit presenter's.
…Eyes, where the seam queries the drawables, not around beginDrawing
…so it does not stay taken by the compositor
On the headset a Compositor Services app on the seam aborted inside queryNextFrame with the compositor's "BUG IN CLIENT": a frame taken with queryNextFrame and never submitted is never released, and three of them abort the app. A review of the seam against upstream's loop found the paths that let a taken frame go: a frame handed out after stop() landed while the source waited (the seam checked the loop's state once, before the wait; upstream checked again right before the query), a stop() mid-frame, a drawing the presenter refused, a frame whose drawables were all skipped (a capture-only frame: the loop returned as if the frame were cancelled, with the submission begun and nothing presented), and a target whose command buffer could not be made (its drawable was marked handed out and never presented).
XRFrame gains release(), nothing by default; the Compositor frame's release() runs the phases the frame missed on a running layer, queries its drawables if they were not, presents every drawable not yet presented cleared and ends the submission (a frame the compositor gave no drawables for was cancelled and is left alone; a paused or invalidated layer reclaims its frames). Drawables are marked at presentation, not at hand-out. The loop lets a frame go through release() on every abandon path and says so (the first ten, then one in a hundred), the stats count them ("N let go" on the stages line), and the source reads the layer's state again after its pacing sleep and says when a frame comes without timing.
…gan, beside the last second's, so a frame let go at any moment since the launch shows
…layer's state, as in Apple's sample: a layer paused mid-frame keeps the frame counted Pressing the Digital Crown pauses the layer mid-frame; the frame in hand was then dropped (beginDrawing refused on a layer not running, endDrawing skipped endSubmission, release() left the frame alone), and the first queryNextFrame after the resume aborted with the compositor's "BUG IN CLIENT". Apple's sample checks the layer's state only at the top of its loop: once a frame is taken it calls startSubmission and endSubmission regardless, and skips the end only when the drawable query gives nothing (the cancelled frame). The three guards are gone; a layer that is gone gives no drawables, which still ends the frame without a submission. Upstream's loop carries the same guards.
…rame stretched by up to a refresh still lands on time, for one refresh more lead Off by default (`XRRealityKitPresenterConfiguration.pipelineDepth` 1: nothing changes). At 2 the presenter is made with two sets of eye textures, grids, band meshes and entities, and the tickets alternate between them by update. A ticket's replace calls go on a presentation buffer of their own (`XRPresenterTicket.presentationBuffer`; the frame's own buffer one refresh deep), which the presenter commits at the next update once the engine has committed the frame's buffer, so that on the engine's queue it sits behind the drawing and RealityKit takes the frame only drawn. At each update the set shown is the one whose ticket of two updates ago has its presentation completed, the other hidden; a frame later than that holds the set shown and is skipped, as a late frame is one refresh deep. So a frame posted at an update shows at the render two updates on whether its drawing took one refresh or nearly two. The price is one refresh of lead: each ticket carries the refresh period measured from RealityKit's updates (`XRPresenterRefresh`, never assumed: 10 ms at 100 Hz, 11.1 at 90) and the engine's thread poses the frame that much further ahead, on the fixed lead or the adaptive one. The depth travels in the ticket's settings copy (`XRPresenterSettings.pipelineDepth`), within the sets the presenter was made with; a depth-2 presenter can be switched to 1 at run time and back. The late count is against the refreshes planned (`noteFrameCompleted(_:plannedRefreshes:)`) and the lines say "(plan 2 refreshes)" then; one deep the line reads as it did. Tests: the refresh measure, the lead's extra refresh (fixed and adaptive), the late count against the plan, the settings clamped to the sets made, the sets' alternation with the presentation buffers waiting behind their frames and the set shown by plan (a late frame holds, the next on time skips it), the frame posed one measured refresh further, and RealityKit's picture of the set shown two refreshes deep against the engine's eye.
…y default On the headset (2026-10-19, Release) the second refresh took the late frames of quick head turns from one to five in a hundred to none, the head-locked probe stopped jumping, and the dot held still with the longer lead. The tests that count one set's entities or read the picture at the first render pin depth 1.
…, for more texels per degree than the display's pixels `XRRealityKitPresenterConfiguration.renderScale` (1 by default, clamped to 0.5 ... 2): the eye textures, the depth pair and the kernel's scratch are made at `textureSize` times it (`renderedTextureSize`), and through the rate map that scaled size is the physical pixel budget, so the map's screen grows with it and the centre of view gains the same factor. At 1 every size is what it was: `renderedTextureSize` returns `textureSize` itself. What the scale buys is texels per degree: the 1920-pixel frame gives 12.9 at the centre of view (17.6 through the map) against the headset's some 34 display pixels, so RealityKit's bilinear sampler shows each texel 2.6 display pixels wide and magnifies the engine's own one-texel edge steps with it; at 1.5 (2880 × 2736) the texture gives 19.3 (26.5 through the map). The price is the engine's GPU frame, 2.25 times the pixels, which the pipeline two refreshes deep has the budget for. The grids keep their vertex counts rather than growing with the texels (the kernel's and RealityKit's mesh cost scale with them; the 512 grid cost about 9 ms on the device for no visible gain): a uniform grid stays `gridColumns` × `gridRows`, its cell growing with the texels; a foveated grid's cell is `foveatedCellTexels` times the scale, taken as it is when it divides the map's tile (scale 1 keeps today's 8), else the tile's divisor nearest in ratio (`XRPresenterFoveation.cellTexels(for:granularity:)`: at 1.5, 12 snaps to 16, the default grid 181 × 172 against 241 × 229; 4 snaps to 8 for the denser one). The presenter and the configuration say the texels per degree at the centre of view (`centreTexelsPerDegree`), and the running line names the scale and the density. On this Mac (Apple M4 Max, the test scene, the headset's configuration, two sets) the frame's GPU span goes from 2.76 ms to 5.5 ms median at 1.5 (2.0× for 2.25× the pixels; the engine's busy time 2.9 → 5.1 ms); the eye textures take 27.4 MB per set at 1 and 61.5 MB at 1.5, the depth pair the same again. RealityKit takes a `LowLevelTexture` of 2880 × 2736. Tests: the clamp and the rendered size, the texels per degree, the cell's snap to the tile's divisors, the shipped map at 1.5 (the screen 1.5× larger, the grid's count held), the chain at 1.5 (textures, depth, scratch, block buffer, eye size, viewports, the loop's targets), RealityKit's picture at 1.5 against the engine's eye of that size (90 dB, the floor, as at scale 1), and the measurement above.
…th kept, instead of alpha-tested with a band `XRRealityKitPresenterConfiguration.silhouette`: `.cutAndBand` (the default, the shipped chain, unchanged) or `.blendedCore`, for the next space opened. Under the blended core each eye's grid takes a third graph, `XRPresenterBlended.usda`: the band's blended graph (no opacity threshold, the colour divided by the alpha, the tear mask's cut, the mono view showing the left eye) loaded with `writesDepth` on, so the grid still occludes what RealityKit draws after it and what the system composites by the layer's depth, and with a geometry modifier that collapses the OTHER eye's grid to a point through the Camera Index Switch in the vertex stage: with blending every rasterized fragment writes depth, those of opacity 0 too, and the other eye's grid would otherwise have written its parallax-offset depth into the eye it is invisible to. No band meshes or entities are made in this mode (`hasEdgeBand` and `drawsEdgeBand` are false whatever the band setting says), the ticket carries no band buffers, and the kernel's band pass is not encoded. The presenter's `silhouette` says which is in force: the blended core falls back to the shipped chain when its graph cannot be loaded. The view's running line names it. Why: on the headset the engine's cubes shimmer beside a plain RealityKit cube that does not, through the same device render (the control cube, 2026-10-19), and on the Mac no presenter knob moves the display-scale wobble: what the device renders differently is the suspect, the shipped core's hard alpha cut rasterized without anti-aliasing under its foveated render. The blended core has no cut anywhere along a silhouette; the Mac shows the same picture for both chains (90.0 dB, the PSNR floor, against the engine's MSAA eye and against each other), so what the headset shows with it is the device's part alone. Its depth footprint is the shipped chain's: the skirt's texels and the near part of the stretched triangles write depth at an opacity near 0, as the shipped band's cells do (a shader graph material writes depth unless told not to): a box behind the test's cubes, drawn after the grids in the transparent pass, is hidden under the same 921 texels of halo by either chain. Tests: the default and the labels; the blended graph loads with the two parameters, writing depth, both faces; through the blended core RealityKit's picture is the engine's MSAA eye (90.0 dB) and the shipped chain's (90.0 dB), with no band entities, meshes or buffers and depth-writing materials on the grids; the depth write itself, with a RealityKit box behind the cubes drawn after the grids through a sort group: hidden where the cubes are opaque (0 of 2972 texels show it, the cubes within 0.0003 of the engine's), shown over the room away from the cubes (60126 of 60126 texels), covering the cubes with the depth write turned off, and filling the whole picture with the other eye's grid alone (collapsed, it writes no depth). The presenter filter: 99 tests, 0 failures; the shipped pictures unchanged.
… placement unchanged `XRRealityKitPresenterConfiguration.tears` (and the same in `XRPresenterSettings`, so it changes while the space is open): off, the unprojection kernel leaves every vertex's tear mask zero (flag bit 2), so no cell stretched across a depth jump is discarded and the grid stays one surface; the triangles between an object's rim and what lies behind it read the texels between the two, transparent but for the rim's partly covered ones, which the alpha test cuts. The threshold still places the vertices (the near side of a footprint's depth jump) and marks the rims, so the switch isolates the per-cell tear decision as a cause of the shimmer along the cubes' edges on the headset (2026-10-19), the first knob that moved it. A test runs the kernel twice on the two-planes depth, with and without, and checks the mask is zero without and the placement (depth, texel, rim flag, position, uv) is the same; the ticket test carries the switch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 2 of
docs/proposals/XRRealityKitPresenter.md(untoldengine#1371; the plan and the headset notes are in #80), judged on an Apple Vision Pro on 2026-10-18/19, with the stage 3 work the headset asked for: a second presenter of the engine's XR frame. A game that wants the system's space (Spatial Personas in a SharePlay session, native entities, SwiftUI attachments) puts anUntoldXRPresenterViewin itsImmersiveSpaceinstead of aCompositorLayer; the engine renders every frame as it always does, and RealityKit shows each eye on a grid displaced by the engine's own depth. Games on Compositor Services do not import the new target and see no change.Built on the seam of
feature/xr_frame_source(the frame-source split ofUntoldEngineXR). The try-out app isRealityKitPresenterExample(miolabs/UntoldArcade#18); its window holds the presenter's settings, so every comparison below was made on the device without a relaunch.The target
UntoldEngineXRRealityKitXRRealityKitPresenter(main thread): twoLowLevelTextures the engine draws the eyes into, two depth textures of its own, and per eye twoLowLevelMeshes with their entities: the core grid (256 × 240 by default) under the eye-switching shader graph (Resources/XRPresenter.usda: the Camera Index Switch, an Unlit Surface with the tone map off, the colour divided by the alpha, an opacity threshold of 0.9) and the edge band under the same graph without the threshold, blended, no depth writes (XRPresenterEdge.usda). Once perSceneEvents.Updateit makes a command buffer on the engine's queue, calls RealityKit's replace APIs and posts a ticket.settings(XRPresenterSettings: skirt, tear threshold, adaptive diagonal, edge band, lead) change while the space is open; each ticket carries the settings in force when it was posted, so a change never races a frame.stop(after:)closes a session in one more frame of its own loop, so the engine's deferred destroys and the reset's completion run before the next session (a reopened space drew nothing on the device).RealityKitFrameSource: each ticket is one frame ofXRFrameLoop, posed for the moment it is taken plus a lead; the eyes sit a nominal distance apart on the headset's pose; the frame is drawn in the ticket's own command buffer, which after the eyes carries the grids' unprojection and, committed by the loop, hands RealityKit the frame.XRGridUnprojectPass: the kernels, compiled from source once per device and specialized by function constants (kFoveated,kEdgeBand). Per eye and frame: the nearest depth of each 8 × 8 block of the depth; a vertex settled from the blocks when its own centre texel lies within the tear threshold of their nearest (most vertices), the rest gathered into a dense indirect scan of their footprint (the cell plus half a cell, every texel read once); each vertex placed on the near-side texel nearest its own centre, with a one-texel hysteresis against the previous frame, unprojected into the origin's space, carrying a projective uv (u·z, v·z, z) so RealityKit's interpolation reproduces the engine's projection on every planar triangle; rim vertices pushedskirtTexelsoutward so the silhouette seen is the alpha test's crossing of the filtered texture; what sees nothing on a far shell; the tear mask's two channels (the near and the far side of a jump, the material discards where both are set). Then the index buffers: the core's (the fixed diagonal, or withadaptiveDiagonalthe diagonal of each cell that does not cross a depth jump) and the band's (the cells that touch a rim or tear vertex, every other cell degenerate), throughreplaceIndices(using:).XRPresenterFoveation: an optional fixed, head-centred equi-angular rasterization rate map (configuration.foveation = .equiAngular), one for both eyes and every frame, h(x) = 1/(1+x²) per axis over the frustum tangents, the screen scaled down until the map's physical size fits the configured pixel budget (Metal rounds rates up); the textures, the depth and the grid at the physical size, the kernels mapping texels to rays through the map's decoder. On the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen: the centre of view 1.36 × denser (12.9 → 17.6 texels per degree) at the same pixel count and the same engine GPU frame. RealityKit gives the presenter no gaze, so this is what foveation can be here.XRTicketMailbox(one slot, never a queue),XRPresenterStatistics,XRPresenterInput(an inverted collision sphere so every pinch has a target),UntoldXRPresenterView(open()awaits a close still in flight; a new session finishes the deferred destroys a close had to leave behind).XRFrame.makeCommandBuffer(for:)(a default) andXRImmersionMode;finalizeDestroysNow(); and the look pass grades and tone-maps the colour behind a texel's coverage in mixed immersion (.lookPremultiplied/.lookPresentPremultiplied, a template so the opaque pass is bit-identical given its input), since applying ACES to the premultiplied fraction made a partly covered rim texel brighter than its face, a bright pulsing rim along near-horizontal edges on the device (1.65 × at quarter coverage under ACES, 2.72 × under AgX in the test; 0.99–1.00 now).Judged on the headset (2026-10-18/19, Apple Vision Pro, visionOS 27.0.1)
Every device number in this section is from DEBUG builds of the example and of the engine package (xcodebuild without
-configuration Release): the frame times, the GPU readings, the frame rates and the replace times compare with each other, not with a release build, whose render time is much lower. Release numbers will follow.The morning's build: 90 fps, 90 RealityKit updates per second, no dropped ticket, the replace calls 0.05 ms on the main thread, the picture world-locked and whole in each eye (the shader graph, not the fallback); the engine's GPU frame 10 ms of the 11.1 ms refresh for two 1920 × 1824 eyes of the small example scene; and every silhouette a staircase of cell-sized steps (7.5 texels) that crawled with the predicted pose's jitter. The day's commits, in the order the headset asked for them:
34941bce): vertices on texels, the per-frame diagonal, the skirt. The slab's lower edge against a fitted line went from 0.75 px mean / 2.8 px max off, with two steps of 2 px or more, to 0.32 / 0.84 with none (capture pixels, about 1.7 per eye texel; the sphere's outline 0.31 px rms from a circle, of which the grid holds 0.01–0.05). The adaptive diagonal made no visible difference and stays off by default.f0f5f874): a second entity per eye blends a silhouette's partly covered texels over the room, over a core cut at 0.9 (at 0.5 the core took every texel covered by half and the band had nothing to show). Only MSAA leaves coverage in the frame's alpha (FXAA one texel per edge at a constant 21/255, SMAA none, measured by a probe in the chain test), so the example sets MSAA by default. The simulator cannot show the band (single-sampled there).7ba209f1): launches with arguments from the Mac fail most of the time (devicectl), so the comparisons moved into the example's window.93044eb2): "the wavy rim on the cube tops is reduced". What remains of that rim stayed the same under FXAA against MSAA, under a 3 cm view offset and on the 512 × 480 grid, so it is not the presenter's data path: it is the engine's own one-texel staircase along a grazing near-horizontal edge, which the 0.9 cut lets through nearly unfiltered (measured on the Mac: 0.26–0.31 texel rms read as a hard edge, 0.07 through its coverage).b144a5d0): "a little bit sharper" at the centre of view against the uniform frame, the same pixel count.703290a0…4dded0a8): a reopened space drew nothing; the old loop now runs the destroys and the completion itself. A Mac test: the second session's picture 83 dB against the first's, 1.9 dB with the old order.c111c76c; 6.8 % of the cells around the three cubes, the picture identical to the whole-grid band at 90 dB): 2 ms. The 512 grid reading (four times the vertices, +14 ms) put the kernel's every-texel footprint pass in the frame's critical path; the block fast path (83029e81, byte-identical entries, 0.689 → 0.045 ms on the Mac at the headset's size, 98.9 % of the vertices settled from the blocks) took the device from 16 ms to 3.8 with the band off, so the old pass had been about 12 ms, most of the morning's 10 too. With the band, MSAA and the rate map on together: 3.8 ms, a third of the frame.Before (the morning's build) and after (the first build with the silhouette work), the example's still scene in mixed immersion:
Verified
XRRealityKitPresenterTests31,XRFoveationTests3,LookPassCoverageAlphaTests1): the kernels against a Swift reference texel for texel (the two planes and the hole, the one-texel line, every cell's split and winding, the band's cells, the block pass with its partial edge blocks, the fast path against the all-scanned mode byte for byte over three frames), the uniforms' layout, the mailbox, the settings riding in the tickets and the band's buffers leaving and returning with the toggle, the graphs with their nodes, the rate map's fit to the budget and the kernel through it (a tilted plane within 1.2e-6 m), and RealityKit's own picture of the grids (RealityRenderer) against the engine's eye texture: 90 dB with the projective uv (identical at 8 bits; the affine one 58.4 on the same scene), 39.8 dB through the rate map at its screen size (a wrong texel-to-ray mapping would score about 21), 90 dB for the band's cells against the whole-grid band, 83 dB for a second session after a close.test_theKernels_atTheHeadsetsSize_timedByGPUTimestamps): median 0.045 ms on an M4 Max.Left