From bb061a876b9b99918af95d484ef6ab7deb4ea6c4 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sat, 17 Oct 2026 08:18:18 +0200 Subject: [PATCH 01/72] [Docs] Spatial Personas on visionOS: the research note and the plan for a RealityKit presenter of the engine's XR frame Research (SpatialPersonasResearch.md): Apple's statements that Personas never appear over a Compositor Services layer, unchanged in visionOS 27; the four options; the decision for the hybrid (the engine keeps rendering, a RealityView presents the frame on depth-displaced grids); what the hybrid gives up; the Feedback Assistant draft. Plan (XRRealityKitPresenter.md): the frame-source seam in UntoldEngineXR, a RealityKit presenter target, six stages with tests and decisions, and the progress of stages 0 (the spike) and 1 (the seam, feature/xr_frame_source). --- docs/proposals/SpatialPersonasResearch.md | 228 ++++++++++++++++++++++ docs/proposals/XRRealityKitPresenter.md | 178 +++++++++++++++++ 2 files changed, 406 insertions(+) create mode 100644 docs/proposals/SpatialPersonasResearch.md create mode 100644 docs/proposals/XRRealityKitPresenter.md diff --git a/docs/proposals/SpatialPersonasResearch.md b/docs/proposals/SpatialPersonasResearch.md new file mode 100644 index 000000000..efb793b87 --- /dev/null +++ b/docs/proposals/SpatialPersonasResearch.md @@ -0,0 +1,228 @@ +# Spatial Personas in Untold games on visionOS — Research + +Status: research only, no implementation. Researched 2026-10-16 against the XROS 27.0 SDK (Xcode 27.0, 27A266a), Apple's documentation and forum answers, and the engine as checked out that day. Question: we want FaceTime/SharePlay Spatial Personas inside some games, but the engine draws its immersive space with Metal through Compositor Services, where Personas never appear. What can the engine do about it? + +## TL;DR + +1. **The limit is real, Apple-stated, and unchanged in visionOS 27.** Two Apple Vision Pro engineers wrote in July and September 2025 that "personas are not available while displaying content with Compositor Services" and that "there is no way to use persona apart from a SharePlay experience"; both asked for a Feedback Assistant report. Nothing in visionOS 26 (WWDC25 session 294) or visionOS 27 (WWDC26 sessions 287 and 279, the "What's new in visionOS 27" page) mentions Personas for Metal apps, and the Compositor Services module interface in the XROS 27.0 SDK carries 22 `visionOS 26` availability marks and zero `visionOS 27` ones: the framework did not change this year. +2. **Personas only exist inside a SharePlay group session, placed by the system around a scene that RealityKit draws.** So the engine's picture has to pass through RealityKit for a Persona to stand in it. There are two ways: give RealityKit a texture of our frame, or let RealityKit render the scene itself. +3. **Apple has named the texture route.** Answering a question about the new `LowLevelRenderer` (June 2026): "Composition with RealityView content can be done by presenting [the] output to a LowLevelTexture, sampled from a RealityKit mesh via ShaderGraphMaterial. Depth compositing can also be approximated by using a tessellated grid mesh, again through ShaderGraphMaterial and geometry modifier." That applies to any renderer's output, ours included. A new visionOS 27 sample ("Displaying low-latency connected video") shows the zero-copy texture hand-off, stereo included. +4. **The Google answer is right about the conclusion and wrong about the mechanism.** `CustomMaterial` does not exist on visionOS (Apple, April 2025), so "pass your Metal code into RealityKit using CustomMaterial" cannot be done; the "privacy / out-of-process" explanation is nowhere in Apple's material; `LowLevelMesh` only helps in the sense that RealityKit draws it with RealityKit's materials, not ours. +5. **Four options**, in increasing cost: (A) keep Compositor Services and draw the other player ourselves — works today, no Persona; (B) a RealityKit lobby with Personas, the Metal space for play; (C) the hybrid: the engine renders off screen, RealityKit shows the frame on a depth-displaced grid, Personas stand in the scene with approximate occlusion; (D) RealityKit as the engine's visionOS render backend, which is what Apple's own Godot plugin and Unity's PolySpatial do. +6. **Recommendation:** do A now for CoolSaber (the poses are already synchronised; the opponent only has to be drawn), file the Feedback Apple asked for, and prototype C (two to three weeks) to measure the two things that decide it: the extra frame of latency on moving objects and the edge artifacts of depth-displaced grids. D is a second renderer; only for games whose look fits RealityKit's materials. + +--- + +## 1. What Apple says (verified sources) + +### 1.1 Personas and Compositor Services + +| Source | Statement | +|---|---| +| Forum thread 791591 "Metal Compositor Service & Persona (VisionOS)", Jul–Sep 2025 | OP: a collaborative app shows Personas in a `RealityView` but "when I tried to use Compositor Layer ... the personas disappeared." Apple engineer 1: "There is no way to use persona apart from a SharePlay experience." Apple engineer 2: "personas are not available while displaying content with Compositor Services. If you would like to see this capability, I'd really appreciate it if you could file feedback." | +| Forum thread 801420 "SharePlay on the VisionOS with remote participants", Sep 2025 | A Metal immersive app using SharePlay and shared world anchors. Apple: same sentence about Personas; shared world anchors (`sharedWithNearbyParticipants`) "are for nearby participants only. For remote participants, you'd need to sync the position yourself." The OP reports nearby sharing working in the Metal app. | +| Forum thread 766378 "CompositorServices Or RealityKit", Oct 2024 | Apple: "both technologies cannot be used in the same immersive space." RealityKit's ECS (hover effects, physics, collisions) is not available to a Compositor Services app. | +| Forum thread 778674, Apr 2025 | Apple: "visionOS does not support custom shaders in RealityKit ... Compositor Services does support custom shaders, but is not compatible with RealityKit." | + +### 1.2 Where Personas appear and what an app must do + +From "Adding spatial Persona support to an activity" (GroupActivities), WWDC23 session 10087, WWDC25 session 318 and the visionOS 26 sample "Implementing SharePlay for immersive spaces in visionOS": + +- Personas are system-drawn, arranged by a spatial template (side by side, conversational, surround, custom) around the shared scene; windows and volumes get them automatically. "The system doesn't display spatial Personas in a Full Space by default": the app sets `SystemCoordinator.Configuration.supportsGroupImmersiveSpace = true`, and the system then builds a shared coordinate system whose origin is no longer the user ("participants are no longer at the center"). The app opens its immersive space itself when `groupImmersionStyle` reports one, and uses `localParticipantStates.isSpatial` to know when extra state must be kept in sync. Apple's sample: "Set supportsGroupImmersiveSpace to true to see personas in the immersive space." +- WWDC23: "People in the group immersive space will see each other as Personas"; in a private immersive space the system hides the others and shows you as a contact photo. +- visionOS 26 nearby sharing (WWDC25 318): people in the same room see each other through passthrough, not as Personas; "If they join on visionOS, they'll appear in your space as a spatial Persona" is for remote participants. Shared world anchors align nearby devices; `ParticipantState.pose` is not live (updated at key events). +- The sample immersive space is a `RealityView`. Everything above works in `.mixed` or `.full` immersion as far as the documents say; whether a `.full` group space looks right with our content is a device check (section 6). + +### 1.3 What visionOS 26 and 27 changed + +- Compositor Services, visionOS 26 (WWDC25 294): `queryDrawables()`, hover effects through a tracking-areas texture, dynamic render quality, the progressive immersion style with a portal stencil, Mac spatial rendering (`RemoteImmersiveSpace`). No Persona or RealityKit composition. +- visionOS 27 (WWDC26 287 and 279, "What's new in visionOS 27"): RealityKit gained physical-space lighting, projective textures, cloth, reverb meshes, Gaussian splats (`GaussianSplatComponent`), `LevelOfDetailComponent` (by camera distance or screen area) and lightmaps; Godot got a Compositor Services path and a RealityKit rendering plugin; the Spatial Preview framework arrived. Compositor Services: "you can adopt the CompositorServices API directly — on both visionOS and macOS", nothing new. Personas are not mentioned anywhere. +- New in visionOS 27 and worth knowing: `LowLevelRenderer` and `LowLevelRenderContextStandalone` (RealityKit, module `RealityCoreRenderer`, a sub-framework in the SDK). RealityKit's own renderer encodes into a command buffer we own, with stereo outputs (`Output.renderTargetArrayLength`, per-camera `viewports`, a `rasterizationRateMap`) and user-written Metal surface shaders with the `CustomMaterial` ABI. Apple: "interoperability with CompositorServices API is an intended use case", but it is "intended for use in offscreen draws / exclusive rendering"; drawing inside a `RealityView` with shared depth is not supported (feature request FB23042990 filed by the asker). This is the reverse direction of what we need: RealityKit content (for example its splats or materials) drawn into our frame. It brings no Persona. + +### 1.4 The Google AI Mode answer, checked + +| Claim | Verdict | +|---|---| +| Personas are unavailable with Compositor Services | True (section 1.1). | +| Because of "privacy and out-of-process isolation" / "untrusted shader memory" | Not stated by Apple. The engineers only say it is not available and ask for feedback. A plausible guess, nothing more. | +| "You must shift to RealityKit so the system can inject the avatars" | True in substance: only a RealityKit-drawn scene gets Personas. | +| `LowLevelMesh`: "because the final rendering pass is handled by RealityKit, the OS ... will natively inject your participants' Spatial Personas" | Half true. `LowLevelMesh` (visionOS 2) is a vertex/index buffer that RealityKit draws with RealityKit materials. Personas appear because the space is a `RealityView` one, not because of the mesh. Our shading would not run on it. | +| `CustomMaterial` "via Metal Shader Graph nodes" to run our fragment shaders inside RealityKit | False on visionOS. `CustomMaterial` is iOS/macOS only; visionOS has `ShaderGraphMaterial` (MaterialX graphs) and, since 27, Metal surface shaders only inside the offscreen `LowLevelRenderer`. | +| Unity: Metal app mode vs RealityKit app mode | Consistent with Unity's documentation ("PolySpatial and RealityKit will not be used in applications set to this mode"); the cited Unity thread does not actually discuss SharePlay. | + +--- + +## 2. What the engine has today + +Checked on the main worktrees on 2026-10-16 (engine fork develop, arcade main, editor fork develop). + +- **The visionOS loop is written against Compositor Services.** `Sources/UntoldEngineXR/UntoldEngineXR.swift` (1,683 lines) drives `LayerRenderer` (`queryNextFrame`, `frame.predictTiming/startUpdate/endUpdate/queryDrawables/startSubmission/endSubmission`, `LayerRenderer.Clock` waits for pacing) and reads the drawable in `executeXRSystemPass`: `colorTextures` (7 uses), `depthTextures` (4), `deviceAnchor` (4), `rasterizationRateMaps` (3), `frameTiming` (3), `target` (3), `views` (2), `encodePresent` (2), `depthRange` (2), `computeProjection` (1). It owns its own `ARKitSession` with a `WorldTrackingProvider` (and the PSVR2 accessory provider through `InputSystem`). The games configure `.dedicated` layout with foveation off (for example `SaberLayerConfiguration`), one render pass per eye, immersion `.mixed` or `.full` (`setImmersionMode`, the clear alpha from `getAlphaForImmersionMode`). +- **The core renderer is already texture-bound.** `UntoldEngine.renderXR(commandBuffer:passDescriptor:viewMatrix:projectionMatrix:eyeIndex:)` (`Sources/UntoldEngine/Renderer/UntoldEngine.swift:895`) takes any pass descriptor and matrices; `updateXR`, `buildHZBDepthPyramid`, `setXRCameraWorldPosition`, `finalizeXRStatsAndMonitors` are public. The editor's Vision Pro preview drives exactly these from its own loop through a frame-source abstraction (`VisionProFrameSource` / `VisionProEye`: colour and depth textures, `viewFromOrigin`, `projection`; `Editor/VisionPro/VisionProPreviewLoop.swift:203`). An off-screen stereo frame is therefore a third frame source, not a renderer change. +- **SharePlay already exists in CoolSaber** (`UntoldArcade/CoolSaber/Examples/CoolSaberVisionOS/.../visionOS/`): `SaberActivity` (a `GroupActivity`), `SaberSessionController` with `supportsGroupImmersiveSpace = true`, the `.conversational` template, `isSpatial` from `localParticipantStates`, `groupImmersionStyle` mirrored into open/dismiss, an unreliable messenger for poses and a reliable one for events, and `SaberDuelMailbox` to hand data to the render thread without touching the main actor. The game draws only the opponent's blades (`SaberXRGame.updateRemote`), re-based onto an anchor 2 m ahead when the session is not spatial. This is exactly the setup in which Apple's forum posters saw no Persona: everything about the session works, the avatar is simply not drawn over a Compositor Services layer. + +--- + +## 3. Options + +### A. Keep Compositor Services, draw the other player ourselves (works today) + +- **Same room:** with visionOS 26 nearby sharing the opponent is a real person seen through passthrough in `.mixed` immersion; shared world anchors (`WorldAnchor(originFromAnchorTransform:sharedWithNearbyParticipants:)`, nearby only) align the two devices, and the forum shows this working in a Metal app. Nothing to draw. +- **Remote (FaceTime):** no Persona is possible. Send the head pose and the hand or controller poses over the existing `GroupSessionMessenger` (the blade poses already travel that way) and draw an engine avatar: head, hands, blades. `HandTrackingProvider` joints (26 per hand) at 30 Hz are small enough to send for hand avatars. Voice comes from FaceTime regardless. +- Cost: days, inside the game. Loses the Persona's face, expressions and eye direction. + +### B. Two spaces: a RealityKit lobby with Personas, the Metal space for play + +A `RealityView` immersive space (or a volume) carries the SharePlay session: Personas are present in the lobby, between rounds and while spectating; entering the `CompositorLayer` space hides them for everyone (the switch is a dismiss and an open; `groupImmersionStyle` tells the others). Unity ships this as "Hybrid" mode (Compositor Services for the game, RealityKit volumes alongside). + +- Cost: about a week per game, no engine change. Does not give Personas during play. + +### C. The hybrid: the engine renders off screen, RealityKit presents (Apple's suggested bridge) + +Design sketch: + +1. **The immersive space is a `RealityView`.** The group immersive space, spatial template, Personas and the shared coordinate system are then the system's business, as in Apple's sample. +2. **A third frame source for the engine loop** (the pattern of the editor's preview): head pose from our own `WorldTrackingProvider.queryDeviceAnchor(atTimestamp:)` predicted for the display time; a stereo projection of our own (a nominal IPD, a field of view wider than the display's so the edges never run out); two colour targets created with `MTLDevice.makeSharedTexture` and wrapped in `LowLevelTexture`s via `LowLevelDeviceResource(sharedTextureHandle:)` (zero copy; the visionOS 27 sample "Displaying low-latency connected video" does exactly this for a stereo camera feed, toggling two device resources per frame so RealityKit notices the new content); depth stays on our side. +3. **Presentation on a depth-displaced grid.** Per eye, a tessellated grid (about 256 × 240 vertices, one per 8 × 8 pixels of the 1920 × 1824 target) whose vertices a small compute kernel of ours moves to the unprojected depth of that eye's frame, writing a `LowLevelMesh` vertex buffer in the world space of the pose we rendered with. Material: `ShaderGraphMaterial` with an Unlit Surface, "Apply Post Process Tone Map" off, colour from a Camera Index Switch node between the two textures (the node that Apple's stereoscopic image sample uses), and an opacity threshold so each eye's grid is discarded in the other eye. Apple's wording ("ShaderGraphMaterial and geometry modifier") is the shader-only variant of the same thing; the compute kernel keeps the unprojection in our code and avoids depending on Camera Index Switch inside a geometry modifier. +4. **Mixed immersion** falls out of the alpha the engine already writes: alpha below the threshold leaves passthrough. Personas sort against our content per vertex, so occlusion is approximate but present. + +What it buys: Personas standing inside the scene, with occlusion; the system's foveation, hover effects (`HoverEffectComponent`) and reprojection; and the reprojection works in our favour, because the grid is world-locked geometry built from our depth, so head motion between our render and the display is corrected by RealityKit (a depth-mesh reprojection, like positional time warp). + +Costs and risks, to be measured on a prototype: + +- **Latency on moving things.** Our frame reaches the display through RealityKit's next frame: at least one extra 11 ms at 90 Hz for anything that moves in the world (blades), while head motion is covered by the grid. Pacing has to be ours too: no `LayerRenderer.Clock`; drive the engine thread from `RealityView`'s per-frame update or a timer and feed it ARKit timestamps. +- **Edges.** Depth discontinuities stretch triangles; disocclusions show as smears until the next frame. Transparent content (glass, particles) has one depth per pixel. +- **Quality.** The frame is resampled once more; our texture is still rendered uniformly (as today, foveation is off), but RealityKit rasterises the grid with its own foveation; colour management of an unlit surface needs checking (tone map off is a node input). +- **What Compositor Services gives that goes away:** the system's per-eye view transforms (we render at a nominal IPD, but RealityKit displays the grid from the true eye positions, so depth cues stay right), the tracking-areas hover texture, dynamic render quality, the progressive-immersion stencil. +- **Memory and bandwidth:** two shared colour textures and two grids; negligible next to a frame. + +Cost: a two-to-three-week prototype (engine frame source, the RealityKit presenter, CoolSaber hooked up), several more weeks for production (input and accessories in the new space, both immersion modes, pacing, diagnostics). + +### D. RealityKit as the engine's visionOS render backend + +Keep the engine for everything but drawing (ECS, physics, animation, streaming, scripts, SharePlay) and mirror renderable entities into RealityKit: `ModelComponent` with `MeshResource` or `LowLevelMesh`, `PhysicallyBasedMaterial` / `ShaderGraphMaterial`, lights, `MeshInstancesComponent` (visionOS 26), `LevelOfDetailComponent` (27), `GaussianSplatComponent` (27). This is what Apple's GodotRealityKit does ("RealityKit replaces Godot's renderer": PBR and visual-shader graphs convert, GLSL does not, no shadowed point lights) and what Unity's PolySpatial does. + +- Full integration: Personas with true depth, foveation, hover, system lighting, windows and volumes. +- Loses the engine's renderer: custom Metal shaders and render extensions, post-processing, our LOD chains (they would map onto `LevelOfDetailComponent`), HZB culling, the exact look of every game. +- Cost: months; a second renderer to keep in step with the first. Only for games whose look fits RealityKit's material model. + +### Comparison + +| | Personas during play | Occlusion against our content | Engine renderer kept | Engine work | Risk | +|---|---|---|---|---|---| +| A. Own avatars | no (avatar) | n/a | yes | none (game only) | low | +| B. Lobby + Metal | lobby only | n/a | yes | none (game only) | low | +| C. Hybrid texture bridge | yes | approximate (per-vertex depth) | yes | frame source + presenter | latency, edges | +| D. RealityKit backend | yes | exact | no | a second renderer | months, look changes | + +--- + +## 4. Recommendation + +1. **Now, for CoolSaber:** option A. Draw the remote opponent (head and hands from the synced poses) and rely on passthrough for a nearby one. File the Feedback Assistant report both engineers asked for ("Spatial Personas in a Compositor Services immersive space") and keep its number here. +2. **Next:** prototype option C on the editor's frame-source pattern and measure: motion-to-photon for a blade swing versus today; the look of depth edges; whether the Persona's occlusion is acceptable; whether a `.full` group space works as well as `.mixed`. Decide with numbers. +3. **D:** not now. Revisit only for a game that is designed for RealityKit's materials from the start. + +--- + +## 5. Pieces a prototype of option C would touch + +- Engine: a protocol like the editor's `VisionProFrameSource` promoted into `UntoldEngineXR` or the core (eyes as textures and matrices; the loop already exists in the editor's `VisionProPreviewLoop`); a compute kernel that unprojects the depth texture into a `LowLevelMesh` vertex buffer; the double-buffered shared colour textures. +- App side (reusable across games): the `RealityView` immersive space, the grid entities and material (a Reality Composer Pro graph: Camera Index Switch → Unlit Surface with tone map off and opacity threshold), the ARKit session feeding poses, the per-frame kick. +- CoolSaber: swap the `CompositorLayer` space for the RealityKit one and keep `SaberSessionController` as it is (it already configures the group immersive space). + +## 6. Open questions to verify on the device + +1. Do Personas render in a `.full` group immersive space with our grid in front of them, and does the system's placement (the template origin) land where the game expects? +2. The prediction horizon of `queryDeviceAnchor(atTimestamp:)` versus RealityKit's display time: measure the residual swim, then the lag of moving objects. +3. Does the Unlit Surface with the tone map off reproduce our colours (gamut and exposure) exactly? +4. Shared-texture hand-off: the sample says "not supported in the simulator", so this is headset-only work from day one. +5. If the shader-only depth variant is ever wanted: whether Camera Index Switch is honoured inside a geometry modifier. + +## 7. References + +- Apple forum 791591, "Metal Compositor Service & Persona (VisionOS)": https://developer.apple.com/forums/thread/791591 +- Apple forum 801420, "SharePlay on the VisionOS with remote participants": https://developer.apple.com/forums/thread/801420 +- Apple forum 831767, "LowLevelRenderer: visionOS custom shaders, composition with RealityView": https://developer.apple.com/forums/thread/831767 +- Apple forum 766378, "CompositorServices Or RealityKit": https://developer.apple.com/forums/thread/766378 +- Apple forum 778674 (no CustomMaterial on visionOS): https://developer.apple.com/forums/thread/778674 +- Adding spatial Persona support to an activity: https://developer.apple.com/documentation/groupactivities/adding-spatial-persona-support-to-an-activity +- Implementing SharePlay for immersive spaces in visionOS (sample, visionOS 26): https://developer.apple.com/documentation/visionos/implementing-shareplay-for-immersive-spaces-in-visionos +- WWDC23 10087, Build spatial SharePlay experiences: https://developer.apple.com/videos/play/wwdc2023/10087/ +- WWDC25 318, Share visionOS experiences with nearby people: https://developer.apple.com/videos/play/wwdc2025/318/ +- WWDC25 294, What's new in Metal rendering for immersive apps: https://developer.apple.com/videos/play/wwdc2025/294/ +- WWDC26 287, Build next-generation experiences with visionOS 27: https://developer.apple.com/videos/play/wwdc2026/287/ +- WWDC26 279, Explore advances in RealityKit: https://developer.apple.com/videos/play/wwdc2026/279/ +- What's new in visionOS 27: https://developer.apple.com/visionos/whats-new/ +- Displaying low-latency connected video (sample, visionOS 27; LowLevelTexture + LowLevelDeviceResource, stereo): https://developer.apple.com/documentation/realitykit/displaying-low-latency-connected-video +- Displaying a stereoscopic image (Camera Index Switch): https://developer.apple.com/documentation/visionos/displaying-a-stereoscopic-image-in-visionos +- Camera Index Switch node: https://developer.apple.com/documentation/shadergraph/realitykit/camera-index-switch-(realitykit) +- Unlit Surface node (Apply Post Process Tone Map, Opacity Threshold): https://developer.apple.com/documentation/shadergraph/realitykit/unlit-surface-(realitykit) +- Geometry Modifier node: https://developer.apple.com/documentation/shadergraph/realitykit/geometry-modifier-(realitykit) +- LowLevelRenderer (visionOS 27): https://developer.apple.com/documentation/realitykit/lowlevelrenderer and LowLevelRenderer.Output: https://developer.apple.com/documentation/realitykit/lowlevelrenderer/output-swift.struct +- Apple plugins for Godot (GodotRealityKit, Rendering.md): https://github.com/apple/plugins-for-godot +- Unity, Metal-based apps on visionOS: https://docs.unity3d.com/Packages/com.unity.polyspatial.visionos@3.0/manual/MetalApps.html + +## 8. Decision (2026-10-16) and what the hybrid gives up + +The user chose option C, the hybrid, after the discussion of 2026-10-16. It is an **optional presentation mode per game**: games that do not need Personas keep the Compositor Services path unchanged and lose nothing. For the games that opt in, this is the honest list of what changes against Compositor Services, with how much of it can be won back. + +| What Compositor Services gives today | In the hybrid | Recoverable? | +|---|---|---| +| Content displayed the frame it is rendered | One RealityKit frame later (~11 ms at 90 Hz) for anything that moves in the world; head motion is corrected by the world-locked depth grid | Mostly: extrapolate moving transforms by the known delay (the engine already predicts networked objects); draw the few latency-critical things (the player's blades and hands, UI) as native RealityKit entities on top of the grid, at RealityKit's own latency and with exact depth | +| Exact per-pixel depth for passthrough compositing | Per-vertex depth on a grid: stretched triangles at depth jumps, smears on disocclusion during fast head turns | Partly: finer grid where depth varies, discard triangles whose depth span exceeds a threshold (a tear instead of a stretch), per-eye rendering so parallax disocclusions are already right at the render pose | +| One depth per pixel also for glass | Glass has no depth of its own: a Persona behind a pane is either composited in front of the pane's tint and reflection (opaque depth) or hidden by it (glass depth) | No; choose per scene which depth the grid uses. Matters for the BIM site, not for an arena | +| System foveation (rasterization rate maps) and dynamic render quality | Our texture is rendered uniformly, RealityKit foveates only its raster of the grid | Not really; today the engine renders uniformly anyway (foveation off), so nothing is lost now, only the future gain of phase 0 in the virtualized-geometry note | +| True per-eye view transforms (the user's IPD) | We render at a nominal IPD; RealityKit shows the grids from the true eye positions, so depth cues stay right and only view-dependent shading is slightly off | Minor and permanent | +| Hover effects on our content through the tracking-areas texture | None on engine-rendered pixels | Only for native entities (`HoverEffectComponent`) | +| Progressive immersion through the portal stencil | RealityKit does progressive immersion itself | Regained for free | +| The frame clock and predicted timing of `LayerRenderer` | We pace from RealityView's per-frame update and predict with ARKit timestamps | Mostly; some jitter risk to measure | +| Our tone mapping shown untouched | Unlit surface with the post-process tone map off | To verify (exposure in mixed immersion) | +| GPU and memory | Plus a tiny unprojection kernel, two grids of ~61k triangles each, one resampling of the frame, two shared colour textures | Small | + +What the hybrid adds beyond Personas: the system's reprojection on our frame, SwiftUI attachments and native RealityKit entities in the same space (UI, splats, portals), progressive immersion, and hover effects for the native parts. + +Permanent residuals, then: one frame of lag on engine-rendered moving objects, small edge smears during fast head motion, no foveation gain, no hover on engine pixels, the glass depth compromise. Everything else is recoverable or already true today. The prototype measures the first two. + +--- + +## Appendix A. Feedback Assistant draft + +Filed by the user with Apple on 2026-10-16 (number FB________, to record here and to post in forum thread 791591 as the engineer asked). Type: Suggestion. Area: visionOS > Compositor Services (or SharePlay / Group Activities if the picker has no Compositor Services entry). + +**Title:** Compositor Services: Spatial Personas are not shown in a SharePlay group immersive space rendered with Metal + +**Description** + +Apps that render their immersive space with Metal through Compositor Services (CompositorLayer / LayerRenderer) cannot show the Spatial Personas of the other SharePlay participants. The same GroupActivities setup (SystemCoordinator.Configuration with supportsGroupImmersiveSpace = true, a spatial template preference, groupImmersionStyle observed to open the space) shows Personas when the ImmersiveSpace holds a RealityView, but as soon as the space is a CompositorLayer the Personas are gone for everyone in the group immersive space, in both .mixed and .full immersion. Apple engineers confirmed on the developer forums that "personas are not available while displaying content with Compositor Services" (threads 791591 and 801420) and asked for a feedback report; this is it. + +Why it matters: we maintain the open-source Untold Engine (github.com/untoldengine/UntoldEngine), a Swift and Metal game engine whose visionOS renderer is built on Compositor Services, and we ship games with it (for example a two-player lightsaber duel over SharePlay with PlayStation VR2 Sense controllers). Multiplayer games are a natural SharePlay use case, and every engine-based immersive game on visionOS (Untold, Unity's Metal app mode, Unreal, Godot's CompositorServices path) renders this way. Such a game can synchronise its session today but cannot show the other players. The only workarounds are engine-drawn stand-in avatars, or giving up the engine's renderer and rebuilding the game on RealityKit. + +**Steps to reproduce** + +1. Start from the sample "Implementing SharePlay for immersive spaces in visionOS". +2. Replace the RealityView inside the ImmersiveSpace with a CompositorLayer and a minimal Metal render loop (as in "Drawing fully immersive content using Metal"). Leave the SessionController unchanged: supportsGroupImmersiveSpace = true, the .sideBySide template, session.join(). +3. On two Apple Vision Pro devices on a FaceTime call, start the activity and enter the group immersive space on both. + +**Expected:** the other participant's Spatial Persona is placed by the spatial template and visible in the immersive space, exactly as with the unmodified sample. + +**Actual:** no Persona is shown. The session itself works (messages arrive, localParticipantStates reports isSpatial = true), but the other person is invisible. The same happens with the .mixed and the .full immersion style. + +**Suggestions** (any one of these would unblock engine-based games) + +A. Composite the Personas system-side over the Compositor Services layer, depth-tested against the depth texture the app already submits with every frame. The compositor already uses that depth to blend passthrough behind the app's content in mixed immersion, and in visionOS 26 it applies hover effects from an app-provided tracking-areas texture. Personas could be composited the same way, without exposing any Persona data to the app. If useful, the app could also submit a small environment map so the Persona is lit like the scene. + +B. If depth-correct compositing is not possible, draw the Personas on top of the layer, optionally with an app-provided mask texture (in the spirit of the tracking-areas texture) that tells the compositor where Personas may be visible. + +C. Independently of rendering, let the app draw its own avatar for remote spatial participants from a privacy-filtered stream of their motion: head pose and hand poses at normal precision, plus a deliberately coarse gaze and expression signal (see Privacy below). Today SystemCoordinator.ParticipantState.pose is updated only at key events (sharing start, recentering), so an engine cannot even place a stand-in avatar where the other person actually is. + +**Privacy** + +We understand that another person's Persona data (face, eyes, expressions) is sensitive, and that visionOS never exposes gaze to apps. Either of the following trust models would satisfy us, in order of preference for privacy: + +1. The system renders the Persona for us (A or B). No participant data reaches the app. The app only submits what it submits today (colour and depth, optionally a mask or an environment map), and the Persona is composited out of process, exactly as hover effects are applied to a Metal layer without the app seeing the gaze. + +2. The app renders its own avatar from a degraded stream (C), for remote spatial participants only and with their consent (a prompt like the one for hand tracking). An avatar needs nothing precise: the Persona's eye animation is itself an estimate, not the raw gaze. The gaze could be delivered as a look-at point in the shared coordinate space quantised to around 10 cm, delayed and with noise added, and the expressions as a handful of coarse states (blink, smile, mouth open) at a low rate. We would use it only to animate an avatar; the system may degrade it as much as it likes, and the API could be documented and licensed for that purpose alone. Precedent: ARKit on iOS already gives apps 52 face blend shapes, the eye transforms and a look-at point with camera permission, and visionOS gives apps full hand joint poses. The app never needs the local user's own gaze. + +**Configuration:** visionOS 27.x (build to fill in) on two Apple Vision Pro, Xcode 27.0 (27A266a), FaceTime call between the two devices. Also reproducible on visionOS 26. + +**Attachment:** the modified sample project from step 2. diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md new file mode 100644 index 000000000..b3ac3abfe --- /dev/null +++ b/docs/proposals/XRRealityKitPresenter.md @@ -0,0 +1,178 @@ +# A second presenter for the engine's XR frame: RealityKit — Plan + +Status: plan, nothing built. Written 2026-10-16 against the fork develop checkout of that day (the main `UntoldEngine` folder), after the decision recorded in `SpatialPersonasResearch.md` section 8: the engine keeps rendering every frame itself, and for the games that want Spatial Personas a `RealityView` immersive space presents the frame instead of a `CompositorLayer`. + +Question: how does the engine support a second way of presenting its stereo frame on visionOS, next to Compositor Services, without a second renderer and without touching the games that stay on Compositor Services? + +## TL;DR + +- The engine's XR module is one file that does three jobs at once: the XR runtime (ARKit session and providers, world anchors, environment lighting, spatial input, PSVR2 observers, diagnostics), the Compositor Services frame loop (`LayerRenderer` frames, timing, pacing, drawables, foveation, present, deadlines) and the generic per-frame driving of the renderer (update, cull once, each eye, depth pyramid, stats). Only the second job is specific to Compositor Services. The core renderer itself is already texture-bound (`renderXR` takes a pass descriptor and matrices), and the editor has already driven it from a loop of its own through a small frame-source protocol. +- The plan cuts that seam inside the engine (stage 1), then adds a second frame source that gets its textures from RealityKit (stage 2): the engine draws each eye into a `LowLevelTexture` and a compute kernel of ours turns each eye's depth into a `LowLevelMesh` grid; a `RealityView` shows the two grids with a shader-graph material that picks the eye with the Camera Index Switch node. The system then owns the space, so the group immersive space, the spatial template and the Personas are the system's; the engine's LOD, glass, streaming, batching, culling, splats and every shader keep running unchanged. +- Four unknowns decide the details and are settled first in a throwaway spike on the headset (stage 0): the per-frame texture hand-off, the per-eye material, the pose timing and origin, and the Personas themselves. Then stages 3 to 5 tune quality, wire SharePlay and CoolSaber, and ship templates and docs. +- About six to seven weeks of engine work in all, of which stage 1 (the seam, one week, no headset needed) can start now and is worth doing on its own: it also gives the editor's Mac preview a home in the engine. + +## 1. What the engine has (2026-10-16) + +- `Sources/UntoldEngineXR/UntoldEngineXR.swift` (1,696 lines) is `UntoldEngineXR`: `init(layerRenderer:)` → `initUntoldXR` (render path, spatial-event bridge, PSVR2 observer, lighting observer, session monitor, `startARKitProviders`, `UntoldRenderer.createXR(...)` with the layer's colour and depth formats and a guessed viewport), `runLoop()` → `renderNewFrame()` (queryNextFrame, predictTiming, startUpdate, progressive loader tick, spatial input, `syncStreamingCameraPosition`, `updateXR(presentationTime:)`, endUpdate, the wait for the optimal input time with the half-rate policy, startSubmission, `queryDrawables`, device anchor per drawable with the cached-anchor fallback, `executeXRSystemPass`, stats, endSubmission) and `executeXRSystemPass` (semaphore, command buffer, viewport fit, visible entities snapshot, frustum cull, Gaussian cull/preprocess/sort, per view: camera matrix from `deviceAnchor × view.transform`, `computeProjection`, the pass descriptor on the drawable's textures with reverse Z, `makeFoveationFrame`, `renderer.renderXR`, then `buildHZBDepthPyramid`, `encodePresent`, the deadline accounting in the completion handler, commit). The drawable is read in ten places (colorTextures 7, depthTextures 4, deviceAnchor 4, rasterizationRateMaps 3, frameTiming 3, target 3, views 2, encodePresent 2, depthRange 2, computeProjection 1); the frame in seven; the clock throughout the pacing. +- The other files of the module are presenter-independent already: `UntoldEngineXRWorldAnchors.swift` (280 lines, world anchors incl. `sharedWithNearbyParticipants`), `XREnvironmentLightingSystem.swift` (964, the ARKit light estimation probe), `XRSpatialGestureRecognizer.swift` (514, `SpatialEventCollection` → `XRSpatialInputSnapshot` → `InputSystem`), `UntoldEngineXRShutdown.swift` (54, `resetUntoldWorldForXR`, `shutdownUntoldEngineXR`). +- The core renderer: `UntoldRenderer.createXR(configuration:device:commandQueue:colorPixelFormat:depthPixelFormat:viewPort:)` (Renderer/UntoldEngine.swift:813), `updateXR(presentationTime:useExternalStatsLifecycle:)`, `setXRCameraWorldPosition`, `finalizeXRStatsAndMonitors`, `renderXR(commandBuffer:passDescriptor:viewMatrix:projectionMatrix:eyeIndex:)` (:895), `fitStereoTargets(to:)`, plus the public helpers the loop uses (`performFrustumCulling`, `executeGaussianFrustumCulling`, `executeGaussianPreprocess`, `executeRadixSort`, `buildHZBDepthPyramid`, `acquireUniformFrameSlot`, `tripleVisibleEntities`, `visibleEntityIds`, `cullFrameIndex`, `commandBufferSemaphore`, `renderInfo`). `renderInfo.immersionStyle` (`UntoldImmersionMode`), `isXRStereoMode`, `currentEye`, `viewPort`, `xrFoveation`, `xrEyeCameraPosition`. +- The editor's Mac preview (`UntoldEditor/Sources/UntoldEditor/Editor/VisionPro/`): `VisionProFrameSource.swift` (the protocols `VisionProFrameSource` / `VisionProFrame` and the structs `VisionProEye` / `VisionProFrameEyes`), `CompositorFrameSource.swift` (the `LayerRenderer` adapter for macOS 26 remote spaces) and `VisionProPreviewLoop.swift` (the generic driver: update, wait, acquire eyes, cull once, each eye, pyramid, present, stats; tested with a texture-backed source). This is the seam, already proven, in the wrong repository for a game to use. +- Games: every visionOS example and both build templates (`BuildSystem/BuildTemplates.swift:1254` and `:1594`) do `ImmersiveSpace { CompositorLayer(configuration:) { layerRenderer in UntoldEngineXR(layerRenderer:) ... Thread { xr.runLoop() } } }`. CoolSaber adds SharePlay (`SaberSessionController`, `SaberDuelMailbox`). +- Tests: `Tests/UntoldEngineRenderTests/XRImmersionContractTest.swift`, `UpdateXRRenderingSystemTest.swift`, `XRFoveationTests.swift` cover the XR render graph, immersion alpha and the rate maps; nothing covers the loop itself (it needs a `LayerRenderer`). +- SDK facts (XROS 27.0): `LowLevelMesh` and `LowLevelTexture` since visionOS 2 with GPU-side updates (`LowLevelMesh.replace(bufferIndex:using:)` and `LowLevelTexture.replace(using:)` hand back a Metal buffer or texture to write within a command buffer; both are main-actor calls), `LowLevelDeviceResource(sharedTextureHandle:)` in 27 for a zero-copy shared texture, `SceneEvents.Update` once per RealityKit frame on the main thread, `SpatialEventGesture` delivering the same `SpatialEventCollection` the Compositor bridge already consumes, the Camera Index Switch, Unlit Surface (with "Apply Post Process Tone Map" and "Opacity Threshold") and Geometry Modifier nodes, `ShaderGraphMaterial(named:from:in:)` loading a MaterialX graph from a `.usda` in a bundle. + +## 2. Design + +### 2.1 Three pieces instead of one + +``` +UntoldEngineXR (public API unchanged for games) + ├─ XRRuntime ARKit session + providers, world anchors, environment lighting, + │ spatial input bridge, PSVR2 observers, device-anchor cache, diagnostics + ├─ XRFrameLoop the generic driver: for each frame from a source — update, wait, + │ acquire eyes, cull once, draw each eye, pyramid, unproject (if asked), + │ present, stats; owns nothing platform-specific + └─ XRFrameSource where frames come from + ├─ CompositorFrameSource LayerRenderer: timing, pacing, drawables, foveation, present, deadlines + └─ RealityKitFrameSource (new target) RealityKit textures and grids, the main-thread cadence +``` + +The frame contract is the editor's, moved into the engine and widened for what the Compositor path has that the editor did not need: + +```swift +public protocol XRFrameSource: AnyObject { + func nextFrame() -> XRFrame? // blocks until the presenter wants a frame; nil when it is gone + func stop() +} +public protocol XRFrame: AnyObject { + var timing: XRFrameTiming { get } // predicted presentation (media time), optimal input time, deadline (optional) + func beginUpdate(); func endUpdate() + func waitForDrawingTime() // Compositor: the clock wait; RealityKit: nothing + func beginDrawing() -> Bool + func acquireEyes() -> XREyes? // nil: nothing drawn, nothing presented (the compositor's "no drawable") + func present(commandBuffer: MTLCommandBuffer) // Compositor: encodePresent; RealityKit: the grid unprojection + func endDrawing() +} +public struct XREye { color, depth: MTLTexture; viewFromOrigin, projection: simd_float4x4; viewport: MTLViewport; foveation: XRFoveationFrame? } +public struct XREyes { originFromDevice: simd_float4x4?; eyes: [XREye]; primary: Bool } // primary false: a capture target +``` + +`XRFrameLoop` is `renderNewFrame` + `executeXRSystemPass` with the drawable and the frame replaced by the protocol: the same order of calls, the same semaphore, the same stats. The half-rate policy, the optimal-input wait and the deadline accounting move into `CompositorFrameSource`, where the clock is; the loop only asks the source for the timing it has. + +### 2.2 The RealityKit presenter + +A new library target `UntoldEngineXRRealityKit` (visionOS only, depends on `UntoldEngineXR`), so that games on Compositor Services never import RealityKit. It provides: + +- `RealityKitPresenter`: owns two `LowLevelTexture`s (colour, `bgra8Unorm_srgb`, the render size), two of the engine's own depth textures (`depth32Float`, reverse Z as today), two `LowLevelMesh` grids (N×M vertices, position + uv + one half attribute for the tear mask, a static index buffer, one part with a large bound), the two entities (one per eye, identity transform, `ModelComponent(mesh: MeshResource(from: grid), materials: [eyeMaterial(eye)])`), the material, an input entity, and the ticket mailbox. +- `RealityKitFrameSource: XRFrameSource`: `nextFrame()` blocks on the mailbox. Once per RealityKit frame, on the main thread (`SceneEvents.Update`), the presenter makes a command buffer, calls `replace(using:)` on both colour textures and `replace(bufferIndex:using:)` on both grids, stamps the ticket with a target presentation time (`CACurrentMediaTime()` + a lead, two refresh periods to start, tuned in stage 3) and posts it. If the engine thread is still on the previous ticket, the new one replaces it (dropped tickets are counted). The frame's `acquireEyes()` queries the runtime's device anchor for the ticket's time (with the cached-anchor fallback the Compositor path has), builds the two eyes from a nominal IPD and the presenter's projection (a field of view a few degrees wider than the headset's per-eye view, with the same reverse-Z infinite projection the engine uses), and hands back the ticket's colour textures and the presenter's depth textures. `present(commandBuffer:)` encodes the unprojection kernel for each eye into the ticket's command buffer; the loop commits it as it does today, and RealityKit shows the new content when the command buffer completes. +- The unprojection kernel (`Shaders/XRGridUnproject.metal`): one thread per grid vertex; samples the eye's depth at the vertex's pixel (the nearest depth of its footprint, so that near silhouettes are kept), unprojects with the inverse projection and the inverse view into the origin's space, writes position and uv; depth 0 (reverse-Z far) goes to a far shell (100 m) so that sky and passthrough areas keep a surface; a vertex whose footprint spans a depth jump above a threshold gets the tear mask set (stage 3 decides what the material does with it). +- The material (`Resources/XRPresenter.usda`, a MaterialX graph written by hand and checked in Reality Composer Pro): parameters `colorTexture` (that eye's texture) and `eye` (1 left, 2 right); Camera Index Switch (integer, mono 0, left 1, right 2) compared with `eye` → 1 or 0; opacity = texture alpha × that; Unlit Surface with the colour, the opacity, "Opacity Threshold" 0.5 and "Apply Post Process Tone Map" off. Each entity is therefore discarded in the other eye, and alpha below the threshold leaves passthrough (mixed immersion) or black (full). +- Input: one invisible entity with `InputTargetComponent` and a large collision sphere around the user, so that every pinch is targeted; the `RealityView` carries a `SpatialEventGesture` whose `SpatialEventCollection` goes to the existing bridge (`enqueueSpatialInputSnapshot`), the same type the `LayerRenderer.onSpatialEvent` path feeds today. Hands, PSVR2 accessories, planes, world anchors and the lighting probe are ARKit providers on the runtime's own session and need no change. +- `UntoldXRPresenterView`: the SwiftUI view a game puts in its `ImmersiveSpace`: makes the `RealityView`, builds the presenter, creates the `UntoldEngineXR` with it (`init(presenter:)`), starts the engine thread with `runLoop()`, subscribes to `SceneEvents.Update`, and on disappear stops the source (the loop ends, the thread exits) and calls `shutdownUntoldEngineXR` as the Compositor apps do. + +### 2.3 Coordinates, immersion, pacing + +- Spaces: the grid positions are in the ARKit origin's space, the entities sit at identity in the RealityKit scene. In a plain immersive space the two origins coincide; in a group immersive space the system moves the space origin (`immersiveSpaceDisplacement`), and stage 4 settles whether the device anchor already includes it or the presenter applies it to the entities. +- Immersion: `.mixed` and `.full` as today through `renderInfo.immersionStyle` (the alpha the engine writes); `.progressive` comes from RealityKit for free. The `ImmersiveSpace` modifier `immersionStyle(selection:in:)` stays the game's. +- Pacing: RealityKit runs at the display rate and decides when to render the grids; the engine's loop runs once per ticket, so an engine frame that takes longer than a refresh simply makes RealityKit show the previous frame again, which is what the half-rate policy did by hand. The policy stays with the Compositor source. Stats get a RealityKit set: ticket age at pickup, dropped tickets, engine CPU and GPU time, lead in use. + +## 3. Stages + +### Stage 0 — the headset spike (3–4 days, needs the Apple Vision Pro; a second one or a FaceTime participant with a Vision Pro for the last item) + +A throwaway visionOS app (xcodegen `project.yml` like the arcade examples, under `UntoldArcade/Spikes/RealityKitPresenter`, deleted or kept as the try-out app once stage 2 exists) with no engine: a `RealityView` immersive space, two grids, a test pattern drawn by a small Metal pass into `LowLevelTexture`s at 90 Hz from a thread of its own, the per-eye material, the depth grid fed from a synthetic depth. It answers, in this order: + +1. `LowLevelTexture.replace(using:)` and `LowLevelMesh.replace(bufferIndex:using:)` once per frame from the main thread, the command buffer committed from another thread: frame rate, dropped frames, and what happens when the previous command buffer is still in flight. If this costs frames, the visionOS 27 route (`LowLevelDeviceResource(sharedTextureHandle:)` with two device resources toggled per frame) is tried next; the plan assumes the first. +2. The material: Camera Index Switch inside the graph discarding the other eye; the tone map off reproducing a known colour ramp; alpha below the threshold showing passthrough. +3. Timing and origin: `queryDeviceAnchor(atTimestamp:)` in a `RealityView` space, the lead that makes a head-locked test pattern stable, and whether the ARKit origin and the scene origin coincide (a cube at the device anchor must sit on the head). +4. Personas: with `supportsGroupImmersiveSpace`, a SharePlay session from a FaceTime call: does the other person's Persona render in front of and behind the grid in `.mixed` and in `.full`, with the depth the grid provides. + +Done: a one-page result at the end of this note's "Progress" section with the numbers; the four decisions D2–D5 below taken. + +### Stage 1 — the seam in the engine (1 week, no headset: the simulator and the tests) + +Branch `feature/xr_frame_source` on a worktree `UntoldEngine-xrseam`, from the fork develop (the upstream PR is cut from upstream develop at the end, as always). + +- Split `UntoldEngineXR.swift` into `XRRuntime.swift` (ARKit, anchors, lighting hooks, input bridge, PSVR2, diagnostics, the device-anchor cache and recovery), `XRFrameSource.swift` (the protocols and structs of 2.1), `CompositorFrameSource.swift` (the `LayerRenderer` loop: `nextFrame`, timing, pacing, drawables with the capture-target rule, foveation frames, present, deadline accounting) and `XRFrameLoop.swift` (the generic driver). `UntoldEngineXR` keeps its public surface (`init(layerRenderer:)`, `setupCallbacks`, `start`, `stop`, `runLoop`, `setImmersionMode`, lighting and gesture entry points, `currentDevicePose`, the counters) and gains `init(frameSource:device:)`, which the Compositor init calls with a `CompositorFrameSource`. +- `createXR` gets its colour and depth formats and the viewport from the source instead of the layer configuration (the "VERIFY THIS" viewport guess goes with it: the source states the eye size). +- Tests (`Tests/UntoldEngineRenderTests/XRFrameLoopTests.swift`): a texture-backed `XRFrameSource` on macOS drives the loop for a few frames (frames drawn, eyes in order, the cull once per frame, the pyramid once, a frame without eyes neither drawn nor presented, stop ends the loop), as the editor's preview tests do; the existing XR tests stay green; SwiftFormat 0.60.1 rules (see `upstream-ci-quirks`). +- Verification: the visionOS simulator build of an arcade example and the CoolSaber app unchanged in behaviour; a headset run of one example to confirm the Compositor path still paces (the half-rate policy and the deadline counters are the regression risk). + +Done: PR to upstream (`refactor: the XR frame loop takes its frames from a source`), no behaviour change, the module's file sizes halved. + +### Stage 2 — the RealityKit presenter (2 weeks, headset for the last days) + +Branch `feature/xr_realitykit_presenter` on the seam. + +- The target `UntoldEngineXRRealityKit` in `Package.swift` (visionOS, depends on `UntoldEngineXR`, resources: the `.usda`), with `RealityKitPresenter.swift`, `RealityKitFrameSource.swift`, `XRGrid.swift` (the `LowLevelMesh` and its index buffer), `XRGridUnproject.metal` + `XRGridUnprojectPass.swift` (the kernel and its encoder; the kernel also compiles into the engine's metallib for the tests), `XRPresenterMaterial.swift` (loads the graph, sets the parameters), `XRPresenterInput.swift` (the input entity and gesture), `UntoldXRPresenterView.swift`, `XRPresenterStats.swift`. +- Tests on macOS: the kernel against a CPU reference (a synthetic depth of two planes and a hole: positions, the far shell, the tear mask), the grid's index topology, the `.usda` loading as a `ShaderGraphMaterial` with the two parameters (RealityKit on macOS loads shader graphs), the ticket mailbox (replace-not-queue, counts). +- The try-out app: the stage 0 spike rebuilt on the engine (or the ProceduralGeometry example with its `ImmersiveSpace` body swapped for `UntoldXRPresenterView`), run on the headset: the scene appears on the grids, head motion stable, mixed and full immersion, pinch events reaching the engine, hands and PSVR2 wands tracked, the stats on screen. + +Done: an example app on the presenter in the arcade (`UntoldArcade/ProceduralGeometry` gets a second scheme, or a new `RealityKitPresenterExample`), the fork PR with before/after frame-time numbers against the same scene on Compositor Services. + +### Stage 3 — quality and cost (1–2 weeks, headset) + +- The lead: measured swim of a world-locked object against the Compositor path; the lead becomes adaptive (ticket age + GPU time of the last frames). +- Edges: the tear mask in the material (discard the stretched triangles), grid density per region (denser near the centre), the nearest-depth footprint rule tuned. +- Moving objects: the engine's existing motion prediction (`ObjectMotion`) posed for the ticket time; a measurement of blade lag in CoolSaber against the Compositor build. +- Cost: the mono-plus-depth mode (one centre eye rendered, both grids built from it, half the GPU) measured as an option (D3). +- Glass: the depth the grid takes (opaque only, or with glass) as a per-game setting (D4). + +Done: the numbers in this note; a decision whether the hybrid ships for CoolSaber. + +### Stage 4 — SharePlay and CoolSaber (1 week, two headsets or a FaceTime participant on a Vision Pro) + +- The group immersive space origin (`immersiveSpaceDisplacement`, `localParticipantState.pose`) applied so that the engine's scene and the system's seating agree; `groupActivityAssociation` on the view as the forum advises. +- CoolSaber on the presenter: `CoolSaberVisionOSXRApp` gets a second `ImmersiveSpace` body (the presenter) chosen by a setting, `SaberSessionController` unchanged, the avatar of the option-A card as the fallback when the opponent is not spatial. +- Verification: the Persona in the arena, occlusion against the arena geometry, the opponent's blades in step with the Persona's hands. + +Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersonasResearch.md`. + +### Stage 5 — templates, docs, editor (3–4 days) + +- `BuildTemplates.swift`: a "RealityKit presenter" variant of the visionOS app template (an option of the build system, default stays Compositor Services), its README lines. +- Docs: a page on choosing the presenter and what it costs (from `SpatialPersonasResearch.md` section 8). +- Editor (optional, its own PR): the Mac preview's `VisionProFrameSource` replaced by the engine's `XRFrameSource` (the `CompositorFrameSource` of macOS 26 remote spaces becomes a second platform adapter of the engine's), removing the duplicate loop from the editor. + +## 4. Decisions + +- **D1 A separate target.** `UntoldEngineXRRealityKit` rather than RealityKit inside `UntoldEngineXR`: Compositor games keep their imports and binary; the presenter can require visionOS 2 features without touching the module's minimum. +- **D2 The texture hand-off** (stage 0 decides): `replace(using:)` from the main thread per frame as the baseline (visionOS 2, RealityKit manages the completion), the shared handle of visionOS 27 as the fallback if the main-thread coupling costs frames. +- **D3 Per-eye rendering** by default; the mono-plus-depth mode is measured in stage 3 and offered as a setting if the look holds. +- **D4 The grid's depth**: opaque geometry only by default (glass excluded, as the research note's table explains); a per-game switch to include glass. +- **D5 Render size**: the headset's per-eye drawable size (1920 × 1824 on the current drawables) and the engine's existing render-scale setting; the presenter's projection a few degrees wider than the display's. +- **D6 The frame contract lives in the engine**, the editor adopts it later (stage 5, optional). +- **D7 The spike is throwaway code** outside the engine package; nothing from it is merged except what stage 2 rewrites with tests. +- **D8 The loop's public API does not change** for Compositor games in stage 1; the arcade apps and the templates compile as they are. + +## 5. Risks + +- The main-actor calls per frame (`replace(using:)`, `replace(bufferIndex:using:)`, entity updates) tie the presenter's cadence to the game's main thread; a busy SwiftUI window could drop tickets. Measured in stage 0; the shared-handle route decouples it on visionOS 27. +- RealityKit gives no presentation timestamp: the lead is an estimate (stage 3 makes it adaptive). A wrong lead shows as swim of world-locked content, the symptom to watch. +- The group immersive space origin may not be where the ARKit origin is; a wrong assumption shows as the scene and the Persona seating disagreeing (stage 4). +- The exposure of an unlit surface in mixed immersion may differ from the Compositor path even with the tone map off; compared in stage 0 with a colour ramp. +- Simulator: `LowLevelTexture` and `LowLevelMesh` work there, so the structure can be run without the headset; the shared handles do not; Personas never. +- The seam refactor touches the frame pacing that was tuned on the headset (half rate, deadlines); the stage 1 headset run exists for that. + +## 6. Progress + +### Stage 0 — the spike (2026-10-16: built and checked in the simulator; the headset run is pending) + +- Where: `UntoldArcade/Spikes/RealityKitPresenter/` (xcodegen `project.yml`, uncommitted, visionOS 2 minimum with the visionOS 27 parts behind `#available`), its `README.md` and the headset run in `CHECKLIST.md`. It builds for the simulator and, signed with automatic provisioning, for the Apple Vision Pro (the group-session entitlement is in the app). No engine code. +- What it has: a `RealityView` immersive space with two `LowLevelMesh` grids (256 × 240 vertices, position + uv + a half2 tear mask, static indices, one part bounded at ±200 m) at identity; a Metal pass of its own on a render thread drawing a synthetic scene (a floor on y = 0, a wall 2.5 m ahead with a round hole at head height, sky written with alpha 0) or a head-locked angular checker, plus view overlays (crosshair, 10° ring, a bar sweeping in 4 s, a 14-swatch sRGB ramp mirrored in the window, per-eye marks) into two 1920 × 1824 `bgra8Unorm_srgb` textures; tickets from `SceneEvents.Update` (main thread: command buffer, the replace calls, target time = post time + lead); a compute kernel filling the grids from the synthetic depth in the space of `queryDeviceAnchor(atTimestamp: target)` with the engine's retry-then-cache fallback; the hand-written material of section 2.2 (`Resources/EyeMaterial.usda`); a window with the switches (route, lead, immersion, pattern, anchor cube, passthrough sky, overlays, texture size, grid update, head-locked distance, IPD) and the numbers; a `GroupActivity` with `supportsGroupImmersiveSpace` and the side-by-side template, opening the space from `groupImmersionStyle`. +- Found while building: + 1. **Shared Metal textures: device yes, simulator no.** `makeSharedTexture(descriptor:)` and `makeSharedTextureHandle()` typecheck for the visionOS device target (XROS 27.0 SDK) and are marked unavailable only in the simulator SDK, which is why Apple's sample guards them with `#if targetEnvironment(simulator)`; the spike, built against the simulator, took that for the platform (corrected 2026-10-16 from the main session). D2's fallback therefore has two forms: the shared handle on the device, and the IOSurface-backed `LowLevelDeviceResource(textureDescriptor:iosurface:plane:)` the spike implemented, which runs in both and is the one to measure first. The spike has both as switches (two device resources per eye toggled with `replace(deviceResource:using:)`; the shared handle compiled out in the simulator), so the headset run compares the three routes; the plan's and the research note's wording stands. + 2. RealityKit samples a `LowLevelTexture` with v = 0 at the first (top) row: the grid uv is written unflipped (the USD-style `1 − v` showed the frame upside down). Stage 2's unprojection kernel must do the same. + 3. The graph loads as `ShaderGraphMaterial(named: "/Root/EyeMaterial", from: url)` from a loose `.usda` in the bundle, parameters `colorTexture` and `eye`; `ND_realitykit_geometry_switch_cameraindex_integer` compared with `eye` through `ND_ifequal_floatI`; a mono view (the simulator) shows the left eye's grid through an extra comparison. `MaterialParameters.Value.int` is `Int32`. + 4. A ticket's command buffer must be committed once `replace(using:)` or `replace(bufferIndex:using:)` ran on it, so when the render thread still holds the previous ticket the main thread posts none for that frame (a counted drop) instead of replacing it; closing the mailbox commits a leftover ticket empty. +- Simulator (visionOS 26.5 on the Mac, replace route, full size): 90 fps, 90 updates/s, 0 dropped tickets, ticket age at pickup under 0.1 ms, GPU 0.2 ms, commit→completion 3 ms, up to 2 command buffers in flight at commit, every anchor query answered for a 22 ms lead; the picture right (wall, hole showing passthrough, crosshair and ring centred, floor below). **The main thread spent 8 ms per frame inside the replace calls, with spikes to 19–27 ms, almost all of it in the two `LowLevelTexture.replace(using:)` calls (8.1–8.4 ms average, 20 ms max) and only 0.03 ms in the two `replace(bufferIndex:using:)`** — the risk of section 5, and a hint that RealityKit copies or reallocates the 14 MB texture per replace; the headset numbers decide, and the spike can halve the texture or freeze the grids to isolate it. One simulator screenshot caught the moving bar torn between two frames; whether that is RealityKit or the capture is for the headset to tell. +- D2–D5: pending the headset run (checklist items 1–4; item 3's observations give D5's frustum, item 2's edge notes feed stage 3). + +### Stage 1 — the seam (2026-10-16: BUILT, commit `eec61bdc` on `feature/xr_frame_source`, worktree `UntoldEngine-xrseam`, no PR) + +- The branch is cut from the fork develop `2cc546d2`, not upstream develop: upstream still lacks #1359, which the module's pacing code comes from; the upstream PR is cut once #1359 merges. +- What moved where (`Sources/UntoldEngineXR`): `XRFrameSource.swift` (129 lines, the contract, compiles on every platform: a source states its colour and depth formats and an eye size and hands out frames; a frame has an update, a wait, a drawing with one target per drawable, a presentation per target, an end), `XRFrameLoop.swift` (375, the old `renderNewFrame` from the update phase on plus `executeXRSystemPass` per target, moved verbatim with the drawable replaced by the target; compiles on macOS for the tests), `CompositorFrameSource.swift` (798, everything that touched the layer renderer: the state machine in `nextFrame`, the half-rate pacing, timing prediction and clock measurements, the optimal-input wait with its automatic mode, drawables with the capture-target rule, the foveation frames and their ring, `encodePresent` with the deadline handler, the once-a-second line), `XRRuntime.swift` (756, ARKit session and providers with their recovery, the device-anchor cache, the input bridge's event conversion, lighting hooks, the PSVR2 observer, the plane monitor, diagnostics), `UntoldEngineXR.swift` (177, the facade with the same surface, plus `init(frameSource:runtime:device:)`; `createXR` takes formats and eye size from the source; the public `initUntoldXR`, called by nothing in the arcade or the templates, is gone). +- Deviations from section 2.1: the runtime is created before the source and passed to the init (a source poses its frames by the runtime's anchors); an eye carries `deviceFromView` rather than `viewFromOrigin`, so the loop builds the view matrix with the old expression and the eyes keep their textures when there is no pose; `acquireEyes` returns one target per drawable, empty meaning nothing drawn nor presented; the rate-map ring is one slot deeper because the copy happens when the eyes are acquired, before the pass takes its semaphore slot; the dead `lastRenderedPresentation` is gone. +- Verified: Mac build clean, the strict-concurrency guardrail at zero before and after, lint only the local 0.62.1 rules the old file triggered too; core suite 1,755 + 59 component-kit tests (the one known folder-name failure), render suite 1,377 (18 comparison failures were a PATH python without cv2; rerun with the Homebrew python all pass); the eight new tests in `Tests/UntoldEngineRenderTests/XRFrameLoopTests.swift` pass and the three existing XR test classes stay green; simulator builds of the package scheme, ProceduralGeometry and the CoolSaber app; ProceduralGeometry ran 45 s in the simulator with the same once-only lines in the same order as the old loop (63 frames spaced 16.7 ms, no errors). +- A headset run must still confirm: the half-rate pacing entering and leaving under load with the paced frames an exact cadence apart, the deadline counters and the automatic wait moving as before, the capture target when recording the view, pinches reaching the engine through the bridge the compositor source now installs. +- Side findings, untouched: a world-anchors session is editing the old file in the main folder, and its four hunks belong in `XRRuntime.swift` after this split (expect that conflict when the two meet); a visionOS build template calls `startRenderLoop`, a method the module never had; the arcade's ProceduralGeometry lacks its `GameData` folder, which xcodegen refuses. From 6abf814663651b7f96cf13b83485905ef7540046 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sat, 17 Oct 2026 08:21:45 +0200 Subject: [PATCH 02/72] [Docs] Spatial Personas notes: wording for readers outside the workspace, the spike's branch --- docs/proposals/SpatialPersonasResearch.md | 6 +++--- docs/proposals/XRRealityKitPresenter.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/proposals/SpatialPersonasResearch.md b/docs/proposals/SpatialPersonasResearch.md index efb793b87..f2b99c0c6 100644 --- a/docs/proposals/SpatialPersonasResearch.md +++ b/docs/proposals/SpatialPersonasResearch.md @@ -164,7 +164,7 @@ Keep the engine for everything but drawing (ECS, physics, animation, streaming, ## 8. Decision (2026-10-16) and what the hybrid gives up -The user chose option C, the hybrid, after the discussion of 2026-10-16. It is an **optional presentation mode per game**: games that do not need Personas keep the Compositor Services path unchanged and lose nothing. For the games that opt in, this is the honest list of what changes against Compositor Services, with how much of it can be won back. +Option C, the hybrid, was chosen on 2026-10-16. It is an **optional presentation mode per game**: games that do not need Personas keep the Compositor Services path unchanged and lose nothing. For the games that opt in, this is the honest list of what changes against Compositor Services, with how much of it can be won back. | What Compositor Services gives today | In the hybrid | Recoverable? | |---|---|---| @@ -172,7 +172,7 @@ The user chose option C, the hybrid, after the discussion of 2026-10-16. It is a | Exact per-pixel depth for passthrough compositing | Per-vertex depth on a grid: stretched triangles at depth jumps, smears on disocclusion during fast head turns | Partly: finer grid where depth varies, discard triangles whose depth span exceeds a threshold (a tear instead of a stretch), per-eye rendering so parallax disocclusions are already right at the render pose | | One depth per pixel also for glass | Glass has no depth of its own: a Persona behind a pane is either composited in front of the pane's tint and reflection (opaque depth) or hidden by it (glass depth) | No; choose per scene which depth the grid uses. Matters for the BIM site, not for an arena | | System foveation (rasterization rate maps) and dynamic render quality | Our texture is rendered uniformly, RealityKit foveates only its raster of the grid | Not really; today the engine renders uniformly anyway (foveation off), so nothing is lost now, only the future gain of phase 0 in the virtualized-geometry note | -| True per-eye view transforms (the user's IPD) | We render at a nominal IPD; RealityKit shows the grids from the true eye positions, so depth cues stay right and only view-dependent shading is slightly off | Minor and permanent | +| True per-eye view transforms (the wearer's IPD) | We render at a nominal IPD; RealityKit shows the grids from the true eye positions, so depth cues stay right and only view-dependent shading is slightly off | Minor and permanent | | Hover effects on our content through the tracking-areas texture | None on engine-rendered pixels | Only for native entities (`HoverEffectComponent`) | | Progressive immersion through the portal stencil | RealityKit does progressive immersion itself | Regained for free | | The frame clock and predicted timing of `LayerRenderer` | We pace from RealityView's per-frame update and predict with ARKit timestamps | Mostly; some jitter risk to measure | @@ -187,7 +187,7 @@ Permanent residuals, then: one frame of lag on engine-rendered moving objects, s ## Appendix A. Feedback Assistant draft -Filed by the user with Apple on 2026-10-16 (number FB________, to record here and to post in forum thread 791591 as the engineer asked). Type: Suggestion. Area: visionOS > Compositor Services (or SharePlay / Group Activities if the picker has no Compositor Services entry). +Filed with Apple on 2026-10-16 (number FB________, to record here and to post in forum thread 791591 as the engineer asked). Type: Suggestion. Area: visionOS > Compositor Services (or SharePlay / Group Activities if the picker has no Compositor Services entry). **Title:** Compositor Services: Spatial Personas are not shown in a SharePlay group immersive space rendered with Metal diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index b3ac3abfe..9bf04c152 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -158,10 +158,10 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona ### Stage 0 — the spike (2026-10-16: built and checked in the simulator; the headset run is pending) -- Where: `UntoldArcade/Spikes/RealityKitPresenter/` (xcodegen `project.yml`, uncommitted, visionOS 2 minimum with the visionOS 27 parts behind `#available`), its `README.md` and the headset run in `CHECKLIST.md`. It builds for the simulator and, signed with automatic provisioning, for the Apple Vision Pro (the group-session entitlement is in the app). No engine code. +- Where: `UntoldArcade/Spikes/RealityKitPresenter/` (xcodegen `project.yml`, committed on the arcade branch `spike/realitykit_presenter`, visionOS 2 minimum with the visionOS 27 parts behind `#available`), its `README.md` and the headset run in `CHECKLIST.md`. It builds for the simulator and, signed with automatic provisioning, for the Apple Vision Pro (the group-session entitlement is in the app). No engine code. - What it has: a `RealityView` immersive space with two `LowLevelMesh` grids (256 × 240 vertices, position + uv + a half2 tear mask, static indices, one part bounded at ±200 m) at identity; a Metal pass of its own on a render thread drawing a synthetic scene (a floor on y = 0, a wall 2.5 m ahead with a round hole at head height, sky written with alpha 0) or a head-locked angular checker, plus view overlays (crosshair, 10° ring, a bar sweeping in 4 s, a 14-swatch sRGB ramp mirrored in the window, per-eye marks) into two 1920 × 1824 `bgra8Unorm_srgb` textures; tickets from `SceneEvents.Update` (main thread: command buffer, the replace calls, target time = post time + lead); a compute kernel filling the grids from the synthetic depth in the space of `queryDeviceAnchor(atTimestamp: target)` with the engine's retry-then-cache fallback; the hand-written material of section 2.2 (`Resources/EyeMaterial.usda`); a window with the switches (route, lead, immersion, pattern, anchor cube, passthrough sky, overlays, texture size, grid update, head-locked distance, IPD) and the numbers; a `GroupActivity` with `supportsGroupImmersiveSpace` and the side-by-side template, opening the space from `groupImmersionStyle`. - Found while building: - 1. **Shared Metal textures: device yes, simulator no.** `makeSharedTexture(descriptor:)` and `makeSharedTextureHandle()` typecheck for the visionOS device target (XROS 27.0 SDK) and are marked unavailable only in the simulator SDK, which is why Apple's sample guards them with `#if targetEnvironment(simulator)`; the spike, built against the simulator, took that for the platform (corrected 2026-10-16 from the main session). D2's fallback therefore has two forms: the shared handle on the device, and the IOSurface-backed `LowLevelDeviceResource(textureDescriptor:iosurface:plane:)` the spike implemented, which runs in both and is the one to measure first. The spike has both as switches (two device resources per eye toggled with `replace(deviceResource:using:)`; the shared handle compiled out in the simulator), so the headset run compares the three routes; the plan's and the research note's wording stands. + 1. **Shared Metal textures: device yes, simulator no.** `makeSharedTexture(descriptor:)` and `makeSharedTextureHandle()` typecheck for the visionOS device target (XROS 27.0 SDK) and are marked unavailable only in the simulator SDK, which is why Apple's sample guards them with `#if targetEnvironment(simulator)`; the spike, built against the simulator, took that for the platform (corrected 2026-10-16). D2's fallback therefore has two forms: the shared handle on the device, and the IOSurface-backed `LowLevelDeviceResource(textureDescriptor:iosurface:plane:)` the spike implemented, which runs in both and is the one to measure first. The spike has both as switches (two device resources per eye toggled with `replace(deviceResource:using:)`; the shared handle compiled out in the simulator), so the headset run compares the three routes; the plan's and the research note's wording stands. 2. RealityKit samples a `LowLevelTexture` with v = 0 at the first (top) row: the grid uv is written unflipped (the USD-style `1 − v` showed the frame upside down). Stage 2's unprojection kernel must do the same. 3. The graph loads as `ShaderGraphMaterial(named: "/Root/EyeMaterial", from: url)` from a loose `.usda` in the bundle, parameters `colorTexture` and `eye`; `ND_realitykit_geometry_switch_cameraindex_integer` compared with `eye` through `ND_ifequal_floatI`; a mono view (the simulator) shows the left eye's grid through an extra comparison. `MaterialParameters.Value.int` is `Int32`. 4. A ticket's command buffer must be committed once `replace(using:)` or `replace(bufferIndex:using:)` ran on it, so when the render thread still holds the previous ticket the main thread posts none for that frame (a counted drop) instead of replacing it; closing the mailbox commits a leftover ticket empty. @@ -175,4 +175,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - Deviations from section 2.1: the runtime is created before the source and passed to the init (a source poses its frames by the runtime's anchors); an eye carries `deviceFromView` rather than `viewFromOrigin`, so the loop builds the view matrix with the old expression and the eyes keep their textures when there is no pose; `acquireEyes` returns one target per drawable, empty meaning nothing drawn nor presented; the rate-map ring is one slot deeper because the copy happens when the eyes are acquired, before the pass takes its semaphore slot; the dead `lastRenderedPresentation` is gone. - Verified: Mac build clean, the strict-concurrency guardrail at zero before and after, lint only the local 0.62.1 rules the old file triggered too; core suite 1,755 + 59 component-kit tests (the one known folder-name failure), render suite 1,377 (18 comparison failures were a PATH python without cv2; rerun with the Homebrew python all pass); the eight new tests in `Tests/UntoldEngineRenderTests/XRFrameLoopTests.swift` pass and the three existing XR test classes stay green; simulator builds of the package scheme, ProceduralGeometry and the CoolSaber app; ProceduralGeometry ran 45 s in the simulator with the same once-only lines in the same order as the old loop (63 frames spaced 16.7 ms, no errors). - A headset run must still confirm: the half-rate pacing entering and leaving under load with the paced frames an exact cadence apart, the deadline counters and the automatic wait moving as before, the capture target when recording the view, pinches reaching the engine through the bridge the compositor source now installs. -- Side findings, untouched: a world-anchors session is editing the old file in the main folder, and its four hunks belong in `XRRuntime.swift` after this split (expect that conflict when the two meet); a visionOS build template calls `startRenderLoop`, a method the module never had; the arcade's ProceduralGeometry lacks its `GameData` folder, which xcodegen refuses. +- Side findings, untouched: a world-anchors branch in progress edits the old file, and its four hunks belong in `XRRuntime.swift` after this split (expect that conflict when the two meet); a visionOS build template calls `startRenderLoop`, a method the module never had; the arcade's ProceduralGeometry lacks its `GameData` folder, which xcodegen refuses. From 0fb2551d358f3efdcdda53cde41bccf117e72cab Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sat, 17 Oct 2026 08:22:14 +0200 Subject: [PATCH 03/72] [Docs] Spatial Personas notes: the upstream issue and the fork PR in the status lines --- docs/proposals/SpatialPersonasResearch.md | 2 +- docs/proposals/XRRealityKitPresenter.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/proposals/SpatialPersonasResearch.md b/docs/proposals/SpatialPersonasResearch.md index f2b99c0c6..0eb25ce09 100644 --- a/docs/proposals/SpatialPersonasResearch.md +++ b/docs/proposals/SpatialPersonasResearch.md @@ -1,6 +1,6 @@ # Spatial Personas in Untold games on visionOS — Research -Status: research only, no implementation. Researched 2026-10-16 against the XROS 27.0 SDK (Xcode 27.0, 27A266a), Apple's documentation and forum answers, and the engine as checked out that day. Question: we want FaceTime/SharePlay Spatial Personas inside some games, but the engine draws its immersive space with Metal through Compositor Services, where Personas never appear. What can the engine do about it? +Status: research, concluded; the decision and the plan are in `XRRealityKitPresenter.md`, tracked as untoldengine/UntoldEngine#1371 (fork draft PR miolabs/UntoldEngine#80). Researched 2026-10-16 against the XROS 27.0 SDK (Xcode 27.0, 27A266a), Apple's documentation and forum answers, and the engine as checked out that day. Question: we want FaceTime/SharePlay Spatial Personas inside some games, but the engine draws its immersive space with Metal through Compositor Services, where Personas never appear. What can the engine do about it? ## TL;DR diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 9bf04c152..87deed27c 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -1,6 +1,6 @@ # A second presenter for the engine's XR frame: RealityKit — Plan -Status: plan, nothing built. Written 2026-10-16 against the fork develop checkout of that day (the main `UntoldEngine` folder), after the decision recorded in `SpatialPersonasResearch.md` section 8: the engine keeps rendering every frame itself, and for the games that want Spatial Personas a `RealityView` immersive space presents the frame instead of a `CompositorLayer`. +Status: plan; stages 0 and 1 built (see Progress). Tracked as untoldengine/UntoldEngine#1371; the notes are on the fork as draft PR miolabs/UntoldEngine#80. Written 2026-10-16 against the fork develop checkout of that day (the main `UntoldEngine` folder), after the decision recorded in `SpatialPersonasResearch.md` section 8: the engine keeps rendering every frame itself, and for the games that want Spatial Personas a `RealityView` immersive space presents the frame instead of a `CompositorLayer`. Question: how does the engine support a second way of presenting its stereo frame on visionOS, next to Compositor Services, without a second renderer and without touching the games that stay on Compositor Services? From 9453ef0ab1164bd8716c9bfde625db6f0ad954ba Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sat, 17 Oct 2026 09:27:15 +0200 Subject: [PATCH 04/72] [Docs] XR RealityKit presenter plan: stage 2 built (the target, the tests, the simulator run, what stage 3 and the headset run must look at) --- docs/proposals/XRRealityKitPresenter.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 87deed27c..9f72ca0f5 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -1,6 +1,6 @@ # A second presenter for the engine's XR frame: RealityKit — Plan -Status: plan; stages 0 and 1 built (see Progress). Tracked as untoldengine/UntoldEngine#1371; the notes are on the fork as draft PR miolabs/UntoldEngine#80. Written 2026-10-16 against the fork develop checkout of that day (the main `UntoldEngine` folder), after the decision recorded in `SpatialPersonasResearch.md` section 8: the engine keeps rendering every frame itself, and for the games that want Spatial Personas a `RealityView` immersive space presents the frame instead of a `CompositorLayer`. +Status: plan; stages 0, 1 and 2 built (see Progress). Tracked as untoldengine/UntoldEngine#1371; the notes are on the fork as draft PR miolabs/UntoldEngine#80. Written 2026-10-16 against the fork develop checkout of that day (the main `UntoldEngine` folder), after the decision recorded in `SpatialPersonasResearch.md` section 8: the engine keeps rendering every frame itself, and for the games that want Spatial Personas a `RealityView` immersive space presents the frame instead of a `CompositorLayer`. Question: how does the engine support a second way of presenting its stereo frame on visionOS, next to Compositor Services, without a second renderer and without touching the games that stay on Compositor Services? @@ -176,3 +176,12 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - Verified: Mac build clean, the strict-concurrency guardrail at zero before and after, lint only the local 0.62.1 rules the old file triggered too; core suite 1,755 + 59 component-kit tests (the one known folder-name failure), render suite 1,377 (18 comparison failures were a PATH python without cv2; rerun with the Homebrew python all pass); the eight new tests in `Tests/UntoldEngineRenderTests/XRFrameLoopTests.swift` pass and the three existing XR test classes stay green; simulator builds of the package scheme, ProceduralGeometry and the CoolSaber app; ProceduralGeometry ran 45 s in the simulator with the same once-only lines in the same order as the old loop (63 frames spaced 16.7 ms, no errors). - A headset run must still confirm: the half-rate pacing entering and leaving under load with the paced frames an exact cadence apart, the deadline counters and the automatic wait moving as before, the capture target when recording the view, pinches reaching the engine through the bridge the compositor source now installs. - Side findings, untouched: a world-anchors branch in progress edits the old file, and its four hunks belong in `XRRuntime.swift` after this split (expect that conflict when the two meet); a visionOS build template calls `startRenderLoop`, a method the module never had; the arcade's ProceduralGeometry lacks its `GameData` folder, which xcodegen refuses. + +### Stage 2 — the presenter (2026-10-17: BUILT on the plan's defaults, verified on the Mac and in the simulator; the headset run pending) + +- Where: branch `feature/xr_realitykit_presenter` on the engine fork, cut from the seam (`eec61bdc`), worktree `UntoldEngine-xrpresenter`; the example app `RealityKitPresenterExample` on the arcade fork's `feature/realitykit_presenter_example`. Built without waiting for stage 0's headset numbers: D2 as `replace(using:)`, D3 per eye, D4 the engine's depth as it is, D5 the drawable size 1920 × 1824 with a frustum of tangents (1.30, 1.30, 1.15, 1.30), all in `XRRealityKitPresenterConfiguration`, so the headset run changes settings, not code. +- The target `UntoldEngineXRRealityKit` (section 2.2 as planned, with these differences): the kernel `xrGridUnproject` is a Swift string compiled with `makeLibrary(source:)` once per device, not a `.metal` in the engine's metallibs (nothing to rebuild, and the Mac tests run it); one thread per vertex reads the nearest depth over up to 4 × 4 taps of the vertex's footprint, unprojects through the inverse projection into the eye's space and through the eye's pose into the origin's, puts depth 0 and anything past the far shell (100 m) on the shell, and sets the tear mask when a neighbour's view distance differs by more than the threshold (0.5 m); where a footprint straddles a silhouette the nearer side wins, so a near object keeps its outline whole. The grid is 256 × 240 vertices per eye, 24 bytes each (packed position, packed uv with v = 0 at the top row, half2 mask), a static index buffer, one part bounded at twice the shell. The presenter owns its two depth textures (`depth32Float`, private) next to RealityKit's two `LowLevelTexture`s (`bgra8Unorm_srgb`). `RealityKitFrameSource` takes a pose closure (the runtime's device anchor on the headset; a test's pose on the Mac), poses the frame for the moment its ticket is TAKEN plus the lead (2/90 s), builds the two eyes a nominal 63 mm apart, and hands the loop the ticket's own command buffer: the seam gained `XRFrame.makeCommandBuffer(for:)` (default nil) because RealityKit binds a frame's textures to the command buffer the replace calls ran on, and an empty buffer on another queue could complete before the drawing. That buffer is committed exactly once: by the loop after the presentation, by the frame when it is let go before drawing, or by the mailbox closing on a pending ticket. Without a pose the eyes are cleared to transparent and the grids keep their places. The mailbox never queues: a ticket still pending at the next update is a counted drop. The input target is an inverted sphere (radius 50 m) as a static collision mesh with an `InputTargetComponent`; the view's `SpatialEventGesture(coordinateSpace: .immersiveSpace)` feeds the runtime's existing event conversion. `UntoldXRPresenterView(configuration:immersionMode:onReady:onShutdown:)` builds presenter, runtime, source and `UntoldEngineXR(frameSource:runtime:device:)`, hands the game the engine and the presenter once (callbacks, scene, statistics), runs the loop on its thread, posts a ticket per `SceneEvents.Update`, and shuts down as the compositor apps do. The material graph is the spike's, renamed `XRPresenter.usda`, loaded from the target's resource bundle without `Bundle.module`. +- Tests (`Tests/UntoldEngineRenderTests/XRRealityKitPresenterTests.swift`, 13, all on the Mac): the grid's topology and initial vertices; the projection's frustum edges and reverse Z; the mailbox's one slot, the blocking take, the close committing a pending ticket; the kernel against a Swift reference over a synthetic depth (two planes and a hole, drawn by a depth-only pass) including the shell, the seam and the mask; the uniforms' layout; the `.usda` loading on macOS 15 with its two parameters; and the whole chain, `XRFrameLoop` → `RealityKitFrameSource` → presenter, drawing the engine's cubes into the `LowLevelTexture`s and placing the grids (read back through a blit: the middle vertex on the cube, the corners on the shell), a frame without a pose, a frame let go. The eight loop tests and the three older XR classes stay green. +- Simulator (visionOS 26.5, Xcode 27.0): the package scheme and the example build; the example run with `--open-space --hide-window` shows the engine's slab, cubes and sphere in the simulated room in mixed immersion, the left eye's grid in the mono view (the eye switch loaded). Its numbers over a minute: 90 RealityKit updates/s and 90 engine frames/s, 2 tickets dropped at the start and none after, the ticket's age at pickup 1.0 ms, the engine's CPU frame 2.8 ms, the GPU span 0.4 ms (max 0.9), and **8.2 ms per frame on the main thread inside the replace calls (max 18–21 ms)**, the same figure the spike measured for the simulator, so it is RealityKit's cost there (a copy of the 14 MB texture, presumably) and not the engine's; the headset number decides D2 as before. +- Seen in the simulator picture, for stage 3: the silhouettes are saw-toothed where the grid's triangles stretch from an object to the shell (the mask is computed but the material does not use it yet); marking only the far side of a jump and discarding by the mask in the graph is the first thing to try, then the grid density. +- Still to confirm on the headset: that a gaze from inside the collision sphere hits it (pinches reaching the engine); each eye seeing its own grid; the lead against swim; the scene origin against the immersive space's; the replace cost on the main thread (8 ms in the simulator for the spike). From 2b148e92fd631be46620347061ea8a372ec0b917 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sat, 17 Oct 2026 10:04:29 +0200 Subject: [PATCH 05/72] [Docs] XR RealityKit presenter plan: stage 3 edges done in the simulator (the two-sided tear mask cut in the material, the 512 grid's cost and hairlines, what the footprint rule must do) --- docs/proposals/XRRealityKitPresenter.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 9f72ca0f5..d7fdb9bed 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -113,7 +113,7 @@ Done: an example app on the presenter in the arcade (`UntoldArcade/ProceduralGeo ### Stage 3 — quality and cost (1–2 weeks, headset) - The lead: measured swim of a world-locked object against the Compositor path; the lead becomes adaptive (ticket age + GPU time of the last frames). -- Edges: the tear mask in the material (discard the stretched triangles), grid density per region (denser near the centre), the nearest-depth footprint rule tuned. +- Edges: the tear mask in the material (discard the stretched triangles) — DONE 2026-10-17, see Progress; grid density per region (denser near the centre), the nearest-depth footprint rule tuned. - Moving objects: the engine's existing motion prediction (`ObjectMotion`) posed for the ticket time; a measurement of blade lag in CoolSaber against the Compositor build. - Cost: the mono-plus-depth mode (one centre eye rendered, both grids built from it, half the GPU) measured as an option (D3). - Glass: the depth the grid takes (opaque only, or with glass) as a per-game setting (D4). @@ -185,3 +185,11 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - Simulator (visionOS 26.5, Xcode 27.0): the package scheme and the example build; the example run with `--open-space --hide-window` shows the engine's slab, cubes and sphere in the simulated room in mixed immersion, the left eye's grid in the mono view (the eye switch loaded). Its numbers over a minute: 90 RealityKit updates/s and 90 engine frames/s, 2 tickets dropped at the start and none after, the ticket's age at pickup 1.0 ms, the engine's CPU frame 2.8 ms, the GPU span 0.4 ms (max 0.9), and **8.2 ms per frame on the main thread inside the replace calls (max 18–21 ms)**, the same figure the spike measured for the simulator, so it is RealityKit's cost there (a copy of the 14 MB texture, presumably) and not the engine's; the headset number decides D2 as before. - Seen in the simulator picture, for stage 3: the silhouettes are saw-toothed where the grid's triangles stretch from an object to the shell (the mask is computed but the material does not use it yet); marking only the far side of a jump and discarding by the mask in the graph is the first thing to try, then the grid density. - Still to confirm on the headset: that a gaze from inside the collision sphere hits it (pinches reaching the engine); each eye seeing its own grid; the lead against swim; the scene origin against the immersive space's; the replace cost on the main thread (8 ms in the simulator for the spike). + +### Stage 3 — edges (2026-10-17: DONE in the simulator, commit `e5c7b520` on `feature/xr_realitykit_presenter`; the rest of stage 3 waits for the headset) + +- What the teeth were: a grid triangle that stretches from an object to the far shell carries the object's rim texels (and the background's, which alpha discards) along a sliver up to 98 m long. Seen from the eye it was built for the sliver is a point; from the simulator's mono view, 31.5 mm from the left eye, it swings about 1.1° (2.7 cells) across the picture, to the right of every object, since the near end moves with parallax and the far end does not. The object-coloured saw teeth of the stage 2 picture were those slivers. +- What was tried first, as the plan said: the far side of a jump alone marked, the opacity multiplied by (1 − mask) against the Unlit Surface's threshold of 0.5. The picture did not change at all. The mask is interpolated linearly in distance along the sliver, so its midpoint lies 50 m out, where the sliver already projects to nothing; the visible tooth is the sliver's first metre, which that cut keeps whole. +- What works: the kernel marks both sides of a jump in the mask's two channels (x the far side, y the near side), against the eight neighbours, and the material discards wherever min(x, y) exceeds 0.001. A stretched triangle has both channels above zero everywhere inside it; a triangle of the object's own surface has x = 0 and one of the far surface has y = 0, so both keep their outlines up to the jump, with no band of the far surface lost around the object (the x-only cut would lose one cell of it; the example scene cannot show that, nothing lies behind its objects). The diagonals are in the neighbourhood because a cell's two triangles share its diagonal and a jump along the diagonal alone left hairline slivers at slanted silhouettes. With the threshold at 0 the app did not survive the launch; 0.001 and 0.0001 give the same picture. +- Verdict from the screenshots (the `before`/`after` pairs are with the hand-over note of this session): the slab, the cubes and the sphere have no teeth left at 256 × 240; what remains is the grid's own staircase at every outline, up to one cell (7.5 texels, 0.4°) where a grid vertex falls inside the object and its outward neighbour's footprint misses the edge. At 512 × 480 the staircase halves and the sphere is round, at no measurable cost on the Mac (both grids: replace 8.1–8.4 ms, frame 3.1–3.5 ms, GPU 0.3–0.5 ms, 90 frames/s), but the finer grid's smaller footprint (taps 1.4 texels from the vertex instead of 2.8) leaves one-texel rows of the sphere's anti-aliased edge, whose colour alpha is above the threshold where the depth holds nothing (the MSAA edge: colour coverage without a depth sample), on the shell, as dashed hairlines a cell above the sphere. The default stays 256 × 240 until the footprint rule is done. +- Next, the footprint rule (the remaining edges item): the 4 × 4 taps leave texel rows and columns unsampled (every other one at 256 × 240) and adjacent footprints do not meet, so a thin feature or an edge in the gap lands in a stretched triangle and is lost, or stays on the shell; a two-pass kernel (each vertex's nearest depth over its whole cell plus a one-texel margin into a buffer, then the eight neighbours read from that buffer) covers every texel at fewer reads than now (64–81 per vertex once, against 9 × 16). Two more things for the headset stage: an absolute threshold marks oblique surfaces as tears at distance (a floor at eye height beyond ~10 m, where neighbouring rows differ by more than 0.5 m; a relative term, max(0.5 m, 0.1 × distance), would keep them, and the kernel test's threshold had to be 1 m for the same reason on its coarse grid), and the staircase against the dilation trade (a footprint reaching a full cell outward pulls the object's silhouette whole onto the object's depth at the price of a one-cell fringe of background texels glued to it, invisible over passthrough). From b8e27f7af90a07b618a705474e8409d6b43d794e Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sat, 17 Oct 2026 11:35:01 +0200 Subject: [PATCH 06/72] [Docs] XR RealityKit presenter plan: stage 3 footprint done in the simulator (the two-pass kernel, every texel read; the 512 grid's hairline is RealityKit's interpolation of the stretched triangles, not sampling) --- docs/proposals/XRRealityKitPresenter.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index d7fdb9bed..85c97332f 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -113,7 +113,7 @@ Done: an example app on the presenter in the arcade (`UntoldArcade/ProceduralGeo ### Stage 3 — quality and cost (1–2 weeks, headset) - The lead: measured swim of a world-locked object against the Compositor path; the lead becomes adaptive (ticket age + GPU time of the last frames). -- Edges: the tear mask in the material (discard the stretched triangles) — DONE 2026-10-17, see Progress; grid density per region (denser near the centre), the nearest-depth footprint rule tuned. +- Edges: the tear mask in the material (discard the stretched triangles) — DONE 2026-10-17, and the nearest-depth footprint rule (every texel read, two dispatches) — DONE 2026-10-17, see Progress; grid density per region (denser near the centre) open. - Moving objects: the engine's existing motion prediction (`ObjectMotion`) posed for the ticket time; a measurement of blade lag in CoolSaber against the Compositor build. - Cost: the mono-plus-depth mode (one centre eye rendered, both grids built from it, half the GPU) measured as an option (D3). - Glass: the depth the grid takes (opaque only, or with glass) as a per-game setting (D4). @@ -192,4 +192,10 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - What was tried first, as the plan said: the far side of a jump alone marked, the opacity multiplied by (1 − mask) against the Unlit Surface's threshold of 0.5. The picture did not change at all. The mask is interpolated linearly in distance along the sliver, so its midpoint lies 50 m out, where the sliver already projects to nothing; the visible tooth is the sliver's first metre, which that cut keeps whole. - What works: the kernel marks both sides of a jump in the mask's two channels (x the far side, y the near side), against the eight neighbours, and the material discards wherever min(x, y) exceeds 0.001. A stretched triangle has both channels above zero everywhere inside it; a triangle of the object's own surface has x = 0 and one of the far surface has y = 0, so both keep their outlines up to the jump, with no band of the far surface lost around the object (the x-only cut would lose one cell of it; the example scene cannot show that, nothing lies behind its objects). The diagonals are in the neighbourhood because a cell's two triangles share its diagonal and a jump along the diagonal alone left hairline slivers at slanted silhouettes. With the threshold at 0 the app did not survive the launch; 0.001 and 0.0001 give the same picture. - Verdict from the screenshots (the `before`/`after` pairs are with the hand-over note of this session): the slab, the cubes and the sphere have no teeth left at 256 × 240; what remains is the grid's own staircase at every outline, up to one cell (7.5 texels, 0.4°) where a grid vertex falls inside the object and its outward neighbour's footprint misses the edge. At 512 × 480 the staircase halves and the sphere is round, at no measurable cost on the Mac (both grids: replace 8.1–8.4 ms, frame 3.1–3.5 ms, GPU 0.3–0.5 ms, 90 frames/s), but the finer grid's smaller footprint (taps 1.4 texels from the vertex instead of 2.8) leaves one-texel rows of the sphere's anti-aliased edge, whose colour alpha is above the threshold where the depth holds nothing (the MSAA edge: colour coverage without a depth sample), on the shell, as dashed hairlines a cell above the sphere. The default stays 256 × 240 until the footprint rule is done. -- Next, the footprint rule (the remaining edges item): the 4 × 4 taps leave texel rows and columns unsampled (every other one at 256 × 240) and adjacent footprints do not meet, so a thin feature or an edge in the gap lands in a stretched triangle and is lost, or stays on the shell; a two-pass kernel (each vertex's nearest depth over its whole cell plus a one-texel margin into a buffer, then the eight neighbours read from that buffer) covers every texel at fewer reads than now (64–81 per vertex once, against 9 × 16). Two more things for the headset stage: an absolute threshold marks oblique surfaces as tears at distance (a floor at eye height beyond ~10 m, where neighbouring rows differ by more than 0.5 m; a relative term, max(0.5 m, 0.1 × distance), would keep them, and the kernel test's threshold had to be 1 m for the same reason on its coarse grid), and the staircase against the dilation trade (a footprint reaching a full cell outward pulls the object's silhouette whole onto the object's depth at the price of a one-cell fringe of background texels glued to it, invisible over passthrough). +- The footprint rule, as planned next (done, see below): the 4 × 4 taps leave texel rows and columns unsampled (every other one at 256 × 240) and adjacent footprints do not meet, so a thin feature or an edge in the gap lands in a stretched triangle and is lost, or stays on the shell; a two-pass kernel (each vertex's nearest depth over its whole cell plus a one-texel margin into a buffer, then the eight neighbours read from that buffer) covers every texel at fewer reads than now (64–81 per vertex once, against 9 × 16). Two more things for the headset stage: an absolute threshold marks oblique surfaces as tears at distance (a floor at eye height beyond ~10 m, where neighbouring rows differ by more than 0.5 m; a relative term, max(0.5 m, 0.1 × distance), would keep them, and the kernel test's threshold had to be 1 m for the same reason on its coarse grid), and the staircase against the dilation trade (a footprint reaching a full cell outward pulls the object's silhouette whole onto the object's depth at the price of a one-cell fringe of background texels glued to it, invisible over passthrough). + +### Stage 3 — the footprint (2026-10-17: DONE in the simulator, commit `167144c8` on `feature/xr_realitykit_presenter`, fork PR miolabs/UntoldEngine#81) + +- The kernel is two dispatches on one compute encoder with a memory barrier between them: the first writes each vertex's nearest reverse-Z depth over its whole cell plus a one-texel margin into a scratch buffer of one float per vertex (owned by the presenter per eye, reused across frames: the tickets' command buffers run in order on the engine's queue; the bounds taken with a tolerance of a thousandth of a texel so GPU and reference round alike), the second reads its own and its eight neighbours' depths from that buffer and writes position, uv and mask as before. Every texel is read by some vertex and adjacent footprints overlap by two texels; about 140 texture reads per vertex once at 256 × 240 (64 at 512 × 480) plus nine buffer reads, against 9 × 16. The kernel test's reference follows the footprint texel for texel and a new test draws a one-texel line at every texel column and row and checks that one of the two vertices it lies between reads it (22 tests of the six XR classes). +- Simulator: at 256 × 240 the sphere loses its one-cell bites (vertices whose taps missed the edge) and the slab's staircase is unchanged; at 512 × 480 the staircase halves as before. Numbers unchanged (replace 7.9–8.2 ms, frame 2.9–3.4 ms, GPU 0.3–0.5 ms at either grid). +- The dashed hairline above the sphere at 512 × 480 is NOT a sampling gap, so it stays: the eye's textures and grid dumped from the simulator show no texel with colour and no depth, every texel in some footprint, and a software rasterization of that grid from the simulator's viewpoint draws no hairline. In RealityKit the line is one pixel row (about a quarter covered) along the grid row above the sphere's top, carrying the near-side mask and fully opaque sphere texels: it is there with the eye at the camera (no parallax), it goes with a far shell of 50 m or less or a cut epsilon of 0.01 or more (which brings fins back), and it moves to other cell edges with 481 rows. That is RealityKit's interpolation of a stretched triangle's attributes between a vertex at 1.6 m and one at 100 m, which the footprint cannot touch; the knob is the far shell (30 or 50 m give a clean 512 grid in the simulator), a decision for the headset stage together with the relative tear threshold. The default grid stays 256 × 240 until then. Screenshots, crops, the usda-swap and config-run scripts and the analysis scripts are with the hand-over note of this session (`xr-presenter-footprint`). From 1630b0f3dcf78c7160de693fc75dcc288e830097 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 12:35:57 +0200 Subject: [PATCH 07/72] [Docs] XR RealityKit presenter plan: the headset run of 2026-10-18 (D2 settled, the silhouette work, a stale-build correction, MSAA and the edge band, Apple's Godot plugin read) --- docs/proposals/XRRealityKitPresenter.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 85c97332f..8990f3492 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -199,3 +199,19 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - The kernel is two dispatches on one compute encoder with a memory barrier between them: the first writes each vertex's nearest reverse-Z depth over its whole cell plus a one-texel margin into a scratch buffer of one float per vertex (owned by the presenter per eye, reused across frames: the tickets' command buffers run in order on the engine's queue; the bounds taken with a tolerance of a thousandth of a texel so GPU and reference round alike), the second reads its own and its eight neighbours' depths from that buffer and writes position, uv and mask as before. Every texel is read by some vertex and adjacent footprints overlap by two texels; about 140 texture reads per vertex once at 256 × 240 (64 at 512 × 480) plus nine buffer reads, against 9 × 16. The kernel test's reference follows the footprint texel for texel and a new test draws a one-texel line at every texel column and row and checks that one of the two vertices it lies between reads it (22 tests of the six XR classes). - Simulator: at 256 × 240 the sphere loses its one-cell bites (vertices whose taps missed the edge) and the slab's staircase is unchanged; at 512 × 480 the staircase halves as before. Numbers unchanged (replace 7.9–8.2 ms, frame 2.9–3.4 ms, GPU 0.3–0.5 ms at either grid). - The dashed hairline above the sphere at 512 × 480 is NOT a sampling gap, so it stays: the eye's textures and grid dumped from the simulator show no texel with colour and no depth, every texel in some footprint, and a software rasterization of that grid from the simulator's viewpoint draws no hairline. In RealityKit the line is one pixel row (about a quarter covered) along the grid row above the sphere's top, carrying the near-side mask and fully opaque sphere texels: it is there with the eye at the camera (no parallax), it goes with a far shell of 50 m or less or a cut epsilon of 0.01 or more (which brings fins back), and it moves to other cell edges with 481 rows. That is RealityKit's interpolation of a stretched triangle's attributes between a vertex at 1.6 m and one at 100 m, which the footprint cannot touch; the knob is the far shell (30 or 50 m give a clean 512 grid in the simulator), a decision for the headset stage together with the relative tear threshold. The default grid stays 256 × 240 until then. Screenshots, crops, the usda-swap and config-run scripts and the analysis scripts are with the hand-over note of this session (`xr-presenter-footprint`). + +### The headset run (2026-10-18, Javier's Apple Vision Pro, visionOS 27.0.1; both apps installed and launched from the Mac with `devicectl`, the spike's log read through its console with `OS_ACTIVITY_DT_MODE=YES`) + +- **D2 settled: `replace(using:)` stays.** The spike at full size (1920 × 1824 per eye, grids 256 × 240, the replace route): 90.0 fps, 90.0 RealityKit updates/s, 0 dropped tickets, the ticket's age at pickup 0.03 ms, **the main thread inside the replace calls 0.3 ms per frame (textures 0.10, grids 0.19; max 0.6)**, encode 0.35 ms, GPU 1.6 ms (max spikes 3–11 ms), the lead 22.2 ms. The simulator's 8 ms per frame was the simulator's own cost. The IOSurface and shared-handle routes are not needed; they stay in the spike as switches. +- The eye material loaded on the device (`Eye material loaded; parameters: colorTexture, eye`); the system log also prints a harmless `MaterialX node def file not found` line from ShaderGraph at load. +- The spike's picture (no window; `--open-space --hide-window`): the floor and the wall are there, the green ring around the hole with the room showing through it and beyond the wall's edges (passthrough through the opacity threshold works). **Everything shimmers when the eyes rest on it**: the spike draws a sharp per-pixel checker with no anti-aliasing, so the sub-millimetre jitter of the predicted head pose makes every edge crawl, and the floor at grazing angles worst. Not a verdict on the presenter: the engine's frame is anti-aliased; judged again on the example below. Two things to look at if the example shimmers too: the vertex takes the NEAREST depth of its footprint but keeps its own uv, so on an oblique surface it sits off the surface by the depth range across the cell and that error changes with the head (use the centre texel's depth where the footprint's depth range is under the tear threshold, the nearest only at a jump); and whether RealityKit applies the grid and the texture of one ticket in the same frame. +- Windows in the space: the spike's window landed in front of the user each time the space opened (the launch argument opened the space at start) and, with the space open, showed no grab bar to move it; parked aside with the space closed, it was in front again when the space reopened. A game's control window should be hidden while the space is open or placed by the game (`defaultWindowPlacement`); noted for stage 5. +- **The example on the headset** (`--open-space --hide-window`): the engine's metallib loads, the ARKit providers run (one paused-then-running cycle at start, as on the compositor path), `XR presenter: running, 1920×1824 per eye, grids 256×240, eye material` (the shader graph, not the fallback). Steady over a minute: 90 RealityKit updates/s, 90 engine frames/s, 0 dropped tickets, the ticket's age at pickup 0.0 ms, **replace calls 0.05 ms** on the main thread, the engine's CPU frame 3.8 ms, **GPU 10 ms (max 11–17)** for two unfoveated 1920 × 1824 eyes of the small example scene: nine tenths of the 11.1 ms refresh, which is the hybrid's known cost (no foveation gain, section 8 of the research note); the engine's render-scale setting is the lever when a real scene needs it. +- **What the shimmer is** (from a recording of the example, 8 s at 26.7 fps, frames compared in OpenCV): between consecutive frames the engine's surfaces change less than the passthrough does (mean grey difference 1.4 inside the cubes and the slab against 2.2 on the keyboard), and the change concentrates on the **silhouettes**: the outlines of the cubes and the slab, and the sphere against the bright blinds (3.4). The picture does not swim; its edges crawl. Cause: a vertex takes the nearest depth over its cell but keeps the cell's own position, so an object's edge lies at cell granularity (7.5 texels), and the texel-sized jitter of the predicted head pose moves which cell is "on" the object, so the edge jumps by up to a cell. The fix tried next, with the headset on: move each vertex onto the texel it took its depth from (position and uv from that texel), so edges are texel-accurate and the uv matches the position. +- **The edge work, iterated with the headset on** *(the verdicts in this entry are withdrawn: the builds judged did not contain the changes, see the correction below; the reasoning behind each change stands and the changes are in the branch)* (the example made still, so that screenshots compare; a script scores each one: the slab's lower edge against a fitted line, the sphere's outline against a circle). (1) Each vertex placed on the texel its nearest depth was read at: no better, the edges stepped as before (the nearest texel of a tilted face is a footprint corner, up to a cell from the edge). (2) The vertex placed on the near-side texel nearest its own centre (near side = within the tear threshold of the footprint's nearest depth, in metres along the vertex's ray): smaller steps, "a little improvement". (3) The footprint margin from one texel to half a cell, so that the first vertex outside an edge always reaches it (with one texel a third of the edge's places found no vertex; with a whole cell the second row reached it too and the surface behind lost a band). (4) A one-texel hysteresis against the previous frame's texel, against the predicted pose's jitter. Screenshot after (4): slab edge 0.75 px mean / 2.8 px max off a line, sphere 6.1 / 18.5 px off a circle: the same as before by the numbers. The zoom shows why: **the boundary runs through vertices that sit exactly on the edge but can only follow the grid's own edges, and each cell is split along one fixed diagonal (top-right to bottom-left). Where a silhouette descends to the right, a cell's two on-edge vertices are its top-left and bottom-right corners, which that diagonal does not connect; both of the cell's triangles then hold a far vertex, both are discarded, and the boundary dips to the next interior vertex: a notch every few cells along such edges and all around a circle.** The fix: split each cell along the diagonal whose two vertices are nearer in depth (the standard quad split of depth meshes), with the index buffer written per frame by a third kernel pass through RealityKit's index replacement. +- **The diagonal split and the skirt** (built with a design panel of four independent proposals checked by skeptics; the geometry and material routes survived, the alpha-isoline and temporal routes were refuted on their assumption that the engine's temporal anti-aliasing leaves coverage alpha at silhouettes: the engine's XR frame runs FXAA, whose guard keeps object texels at alpha 1 and gives only the first outside texel a ramp, so the colour's alpha is one-sided). (5) A third kernel pass writes each cell's two triangles per frame through RealityKit's `replaceIndices(using:)`, carried in the ticket: the cell keeps the fixed diagonal unless that one crosses a depth jump (more than the tear threshold) and the other does not, so continuous surfaces keep one topology and only the notch cells flip; a test checks every cell's split and winding on the two planes and the hole. In the simulator the staircase on the slab's descending right edge is gone. (6) The skirt: a rim vertex (the texel under its own centre lies off the near surface) is pushed three texels outward past the edge texel it was placed on, from the texel toward its centre, so the mesh boundary lies in texels the alpha test discards and the silhouette seen is the 0.5 crossing of the bilinearly filtered alpha: continuous where FXAA estimated the outside texel, the texel boundary elsewhere; the position is derived in the second pass from the stored texel and a rim flag, so the hysteresis still compares texels. The material divides the colour by the alpha (`ND_max_float` at 0.001, `ND_divide_color3FA`) before the Unlit Surface, since the pixels now shown between the rim texel and the crossing have alpha under 1 and would draw a dark line otherwise; the threshold stays 0.5. In the simulator the slanted edges that were rows of small teeth are smooth filtered edges. The sphere's measured deviation from a circle was mostly the measurement's own mask losing the sphere's bright top; with the mask fixed it is 1.1–1.4 px mean on the earlier headset frames. Settings: `skirtTexels` (3), the hysteresis, the diagonal rule's threshold = the tear threshold. Left for a later round: the other half of each stair step, where FXAA's guard keeps the object-side texel at alpha 1 (needs the object-side coverage: lift the guard for the presenter's frames, or estimate it in a presenter kernel). +- **The coarse-grid experiment** (the example takes `--grid`, `--skirt`, `--adaptive-diagonal`/`--fixed-diagonal`, `--lead`, `--tear` as launch arguments): WITHDRAWN. Its pictures, which seemed to show the fixed split clean and the adaptive flip stepping at a 48 × 45 grid, came from a binary that had neither the arguments nor the flip (the correction below); whether the adaptive diagonal helps or hurts is unjudged, and it is off by default until the headset says. +- **Correction (2026-10-18, midday): none of the day's headset or simulator example builds carried the day's kernel changes until now.** The example's uncommitted local project spec had been rewritten by a side session to point at that session's scratchpad copy of the engine, frozen at the stage 3 edges state; every build compiled against it, and once the example referenced the new settings those builds failed and the install step put the previous binary back, so "still the same" was literal. The Mac tests, which run on the worktree, were the only valid evidence; the simulator comparisons above that showed one build better than another were run-to-run variation of one stale binary (the simulated pose differs a little per launch). The spec now names this session's copy, and the first genuine build is on the headset. Two further facts from the Mac, with a probe in the chain test that reads the eye texture's alpha along a row through the cubes in mixed immersion: (a) **the engine's alpha at a silhouette is binary under FXAA except for one texel per edge at a constant 21/255, and binary under SMAA; MSAA leaves real coverage (64, 128, 191: quarter steps)**, so only MSAA gives an edge band anything to blend; the example now sets `.msaa` by default (`--aa` chooses). (b) In the simulator the viewing pose equals the rendering pose, so any grid geometry projects back onto the texture exactly: the simulator shows the texture's own edge and nothing of the grid's quality (a 48 × 45 grid gives the same picture as 256 × 240); the grid's quality is the headset's to judge, the texture's edge the simulator's. +- **The edge band**: a second entity per eye draws the same grid with the graph minus the opacity threshold (`Resources/XRPresenterEdge.usda`), blended, over the opaque core, so a silhouette's partly covered texels blend over the room instead of being cut at a texel boundary; `edgeBlending` in the configuration, on by default. +- **The band showed nothing, twice, for two reasons found on the Mac and in the engine (2026-10-18, 12:30).** (1) The core's opacity threshold was 0.5: the core drew every texel covered by half or more opaque, and the band had only the quarter-covered texels to blend, at a quarter of their colour, invisible. The threshold is now 0.9: only a silhouette's fully covered texels are opaque in the core, the band blends the quarter, half and three-quarter ones over the room (the 16 presenter tests pass; the alpha probe is unchanged). (2) The simulator cannot show it at all: `requestedOpaqueSampleCount()` returns 1 under `targetEnvironment(simulator)` because the simulator's two-pass deferred fallback samples the G-buffer as a plain texture, so the eye texture's alpha is binary there in every mode. Three simulator screenshots (MSAA with the band, MSAA without, FXAA with) are identical at the texel level, as they must be. The band is the headset's to judge: the Mac probe is the only evidence that MSAA leaves coverage. +- **Apple's Godot plugin, read for how Apple did it (2026-10-18, apple/plugins-for-godot, `godotrealitykit`, MIT, changelog 1.1.0 of 2026-08-27).** It is not a frame hand-off: "RealityKit handles rendering instead of Godot". A `RealitySceneTree` turns Godot's own viewport rendering off and mirrors the scene graph into RealityKit entities every frame (materials compiled to MaterialX `ShaderGraphMaterial`, meshes copied into `LowLevelMesh`, skinning and vertex decompression by Metal compute into the mesh buffers, one command buffer per frame), so silhouettes, anti-aliasing, depth and occlusion are RealityKit's own rasteriser's. The only per-frame image that crosses is a Godot ViewportTexture, blitted into a `LowLevelTexture` with `replace(using:)` as a material texture; alpha cut-outs go to the surface shader's `opacityThreshold` and nothing softens them; alpha-to-coverage is warned unsupported, premultiplied alpha hard-wired off, no mipmaps are made (the plugin's only aliasing warning is about that). On visionOS it installs no camera: the system's stereo projection is used, so there is no per-eye code at all. What it costs Godot: GLSL text shaders cannot run (visual shaders only), screen-space and depth-reading nodes are unsupported, shadowed point lights are lost. Verdict for the engine: Apple's answer to "a custom engine in the shared space" was to not present the engine's frame, which for the Untold Engine would mean a RealityKit back end for every material and pass (the plan's rejected route); the hybrid presenter stays, and its silhouettes are its own problem, with nothing in the plugin to borrow for them. Two details worth keeping: `LowLevelTexture` has no texture views (the plugin keeps sRGB and linear copies), and its mesh layout (positions float3, octahedral ushort2 normals under the uv2/uv3 semantics) is what RealityKit's own loaders expect. From 0d3d16a64c6ebb241ee5059238d5ca7698f02611 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 13:56:50 +0200 Subject: [PATCH 08/72] [Docs] XR RealityKit presenter plan: the genuine build judged, the settings panel, what is left measured and the ranked plan (rate map, projective uv, depth resolve, 512 grid, the limb's teeth) --- docs/proposals/XRRealityKitPresenter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 8990f3492..8d8c5c45a 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -215,3 +215,5 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The edge band**: a second entity per eye draws the same grid with the graph minus the opacity threshold (`Resources/XRPresenterEdge.usda`), blended, over the opaque core, so a silhouette's partly covered texels blend over the room instead of being cut at a texel boundary; `edgeBlending` in the configuration, on by default. - **The band showed nothing, twice, for two reasons found on the Mac and in the engine (2026-10-18, 12:30).** (1) The core's opacity threshold was 0.5: the core drew every texel covered by half or more opaque, and the band had only the quarter-covered texels to blend, at a quarter of their colour, invisible. The threshold is now 0.9: only a silhouette's fully covered texels are opaque in the core, the band blends the quarter, half and three-quarter ones over the room (the 16 presenter tests pass; the alpha probe is unchanged). (2) The simulator cannot show it at all: `requestedOpaqueSampleCount()` returns 1 under `targetEnvironment(simulator)` because the simulator's two-pass deferred fallback samples the G-buffer as a plain texture, so the eye texture's alpha is binary there in every mode. Three simulator screenshots (MSAA with the band, MSAA without, FXAA with) are identical at the texel level, as they must be. The band is the headset's to judge: the Mac probe is the only evidence that MSAA leaves coverage. - **Apple's Godot plugin, read for how Apple did it (2026-10-18, apple/plugins-for-godot, `godotrealitykit`, MIT, changelog 1.1.0 of 2026-08-27).** It is not a frame hand-off: "RealityKit handles rendering instead of Godot". A `RealitySceneTree` turns Godot's own viewport rendering off and mirrors the scene graph into RealityKit entities every frame (materials compiled to MaterialX `ShaderGraphMaterial`, meshes copied into `LowLevelMesh`, skinning and vertex decompression by Metal compute into the mesh buffers, one command buffer per frame), so silhouettes, anti-aliasing, depth and occlusion are RealityKit's own rasteriser's. The only per-frame image that crosses is a Godot ViewportTexture, blitted into a `LowLevelTexture` with `replace(using:)` as a material texture; alpha cut-outs go to the surface shader's `opacityThreshold` and nothing softens them; alpha-to-coverage is warned unsupported, premultiplied alpha hard-wired off, no mipmaps are made (the plugin's only aliasing warning is about that). On visionOS it installs no camera: the system's stereo projection is used, so there is no per-eye code at all. What it costs Godot: GLSL text shaders cannot run (visual shaders only), screen-space and depth-reading nodes are unsupported, shadowed point lights are lost. Verdict for the engine: Apple's answer to "a custom engine in the shared space" was to not present the engine's frame, which for the Untold Engine would mean a RealityKit back end for every material and pass (the plan's rejected route); the hybrid presenter stays, and its silhouettes are its own problem, with nothing in the plugin to borrow for them. Two details worth keeping: `LowLevelTexture` has no texture views (the plugin keeps sRGB and linear copies), and its mesh layout (positions float3, octahedral ushort2 normals under the uv2/uv3 semantics) is what RealityKit's own loaders expect. +- **The genuine build, judged (2026-10-18, 13:00): "the edges look better, even the sphere, but still not the quality we have with the engine."** The headset screenshot scored: the slab's lower edge deviates 0.32 px mean / 0.84 px max from a line with no step of 2 px or more (the stale build: 0.75 / 2.79, two steps). What is left, in crops: crenellations of one to two pixels at cell spacing on the sphere's outline (top right and bottom); a bright wavy rim along the cubes' TOP faces, which are seen edge-on as a sliver a few texels tall (a feature thinner than a cell), and the same at the slab's right end; and a softer picture than the compositor path's (no foveation, bilinear resampling, no mipmaps). The example's window now has a SETTINGS PANEL (the presenter's `settings`: skirt, tear threshold, adaptive diagonal, edge band, lead; the engine's anti-aliasing), so comparisons no longer need a relaunch (launches from the Mac fail most of the time). First verdict with it: **the adaptive diagonal makes no visible difference on the sphere** (head still), so the crenellations are not the cell split. +- **What is left, measured, and the plan for it (2026-10-18, a 17-agent design panel: four proposals from four angles, three skeptics each, one synthesis; the panel re-measured the screenshot itself).** The capture is ~1.7 capture px per eye texel (the slab's 1.2 m edge = 1136 px, the sphere's 0.12 m = 85–94 px, both ≈ 1230 px per unit tangent against the texture's 738), so a 7.5-texel cell is 12.5 capture px. **At rest the mesh adds at most ~0.2 texel rms to a silhouette**: the sphere's outline deviates 0.31 px rms short-scale from a circle and the cell-period band of that holds 0.01–0.05 px, so the crenellations are not the grid (the adaptive diagonal's "no difference" agrees); the slab's lower edge is 0.18 px rms with a clean five-value MSAA ramp. The cubes' top sliver is 1–2 texels tall and its upper edge wobbles 0.66 px rms at 22–36 px periods: the four candidate causes are the band blending MSAA's quarter-coverage sub-steps of a shallow edge, the depth resolve taking sample 0 (which gives the object's depth to 0 % of a top edge's partial texels and 100 % of a bottom edge's, so the rim vertex's depth plane depends on the edge's orientation and per-column luck), the affine uv across a rim→face triangle (0.13–0.30 texel keystone), and parallax of a wrong rim depth plane; the first two vanish under FXAA, the last two do not, which is the next panel experiment (MSAA ↔ FXAA at one pose). **What the eye reads as "not the engine's quality" is the texture's density**: 1920 texels over a 2.6-tangent frustum = 12.9 texels per degree at the centre of view (a uniform perspective render is sparsest at its centre), against a display of ~34 px per degree, so RealityKit's bilinear sampler magnifies each texel to ~2.6 display px and the compositor resamples once more; the engine's 1-texel MSAA ramp and the sphere's Fresnel rim of a glossy limb (real on the compositor path too, at half a display px there) are shown 2.6× magnified. Ranked plan: **(1) the presenter's own fixed rasterization rate map**, equi-angular profile h(x) = 1/(1+x²) per axis, the same physical pixel count (iterate the screen scale down until `physicalSize` fits 1920 × 1824, since Metal rounds rates up), textures and grid at the physical size, the grid uniform in physical texels with cells dividing `physicalGranularity` (8 on 32), the kernel mapping texels to rays through `rasterization_rate_map_decoder` (uniforms +16 bytes, buffer 3), `XREye.foveation` carrying one `XRFoveationFrame` (the loop already attaches it): centre density 12.9 → 18.3 texels/deg at zero engine GPU cost, every texel-sized residue 1.4× smaller; about 150 lines and a day. **(2) Projective uv per vertex** (float4 uv2 = (u·ze, v·ze, ze), divided per pixel in the graph by `ND_separate4`/`ND_divide_float`/`ND_combine2`): RealityKit's perspective-correct interpolation then reproduces the engine's projection exactly on every planar triangle (API-verified on the Mac at float precision), +16 bytes per vertex, half a day. **(3) MSAA depth resolve by nearest sample**: `depthResolveFilter = .max` (reverse Z) on the offscreen and forward descriptors, one line twice, zero cost, shared with the compositor path (the render suites' references may move). **(4) The 512 × 480 grid** with a 50 m far shell, no code, after 1–3. **(5) The sphere's white teeth**: dump the limb texels on the Mac (extend `probeAlpha`); if the partial-coverage texels are ≥ 2× brighter than the interior, `[[centroid_perspective]]` on the vertex outputs mends it on both paths; if not, they are RealityKit's and unreachable. Judged no: a denser or non-uniform grid, a 1024 grid, a higher render scale (10 of 11.1 ms), mipmaps (always magnified), per-frame eye pose through `setParameter` (lands 1–2 frames late); the two-layer grid with a per-pixel depth test is right for the 8–15-texel halo of lost background around silhouettes (invisible against passthrough) and interior occlusion, 4–5 days, later. **What no change removes**: no gaze (RealityKit gives none, so foveation is head-centred and fixed, while the compositor path with `isFoveationEnabled` follows the eyes), two resamples instead of one, the MSAA ramp magnified; and two readings decide the honest statement: MSAA's GPU cost (unmeasured; the band needs it) and which engine app is the reference (the arcade examples and the build template set `isFoveationEnabled = false`, a uniform 2048 × 1984 per eye at 13.8 texels/deg, within 7 % of the presenter's; only UntoldViewer foveates). From 2185726ced710b6c447df083ea7fe344a0851257 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 19:56:24 +0200 Subject: [PATCH 09/72] [Docs] XR RealityKit presenter plan: the panel readings (diagonal no, the rim survives FXAA), the projective uv done with an end-to-end RealityKit test on the Mac (88 dB), the halo seen --- docs/proposals/XRRealityKitPresenter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 8d8c5c45a..649ef00ca 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -217,3 +217,5 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Apple's Godot plugin, read for how Apple did it (2026-10-18, apple/plugins-for-godot, `godotrealitykit`, MIT, changelog 1.1.0 of 2026-08-27).** It is not a frame hand-off: "RealityKit handles rendering instead of Godot". A `RealitySceneTree` turns Godot's own viewport rendering off and mirrors the scene graph into RealityKit entities every frame (materials compiled to MaterialX `ShaderGraphMaterial`, meshes copied into `LowLevelMesh`, skinning and vertex decompression by Metal compute into the mesh buffers, one command buffer per frame), so silhouettes, anti-aliasing, depth and occlusion are RealityKit's own rasteriser's. The only per-frame image that crosses is a Godot ViewportTexture, blitted into a `LowLevelTexture` with `replace(using:)` as a material texture; alpha cut-outs go to the surface shader's `opacityThreshold` and nothing softens them; alpha-to-coverage is warned unsupported, premultiplied alpha hard-wired off, no mipmaps are made (the plugin's only aliasing warning is about that). On visionOS it installs no camera: the system's stereo projection is used, so there is no per-eye code at all. What it costs Godot: GLSL text shaders cannot run (visual shaders only), screen-space and depth-reading nodes are unsupported, shadowed point lights are lost. Verdict for the engine: Apple's answer to "a custom engine in the shared space" was to not present the engine's frame, which for the Untold Engine would mean a RealityKit back end for every material and pass (the plan's rejected route); the hybrid presenter stays, and its silhouettes are its own problem, with nothing in the plugin to borrow for them. Two details worth keeping: `LowLevelTexture` has no texture views (the plugin keeps sRGB and linear copies), and its mesh layout (positions float3, octahedral ushort2 normals under the uv2/uv3 semantics) is what RealityKit's own loaders expect. - **The genuine build, judged (2026-10-18, 13:00): "the edges look better, even the sphere, but still not the quality we have with the engine."** The headset screenshot scored: the slab's lower edge deviates 0.32 px mean / 0.84 px max from a line with no step of 2 px or more (the stale build: 0.75 / 2.79, two steps). What is left, in crops: crenellations of one to two pixels at cell spacing on the sphere's outline (top right and bottom); a bright wavy rim along the cubes' TOP faces, which are seen edge-on as a sliver a few texels tall (a feature thinner than a cell), and the same at the slab's right end; and a softer picture than the compositor path's (no foveation, bilinear resampling, no mipmaps). The example's window now has a SETTINGS PANEL (the presenter's `settings`: skirt, tear threshold, adaptive diagonal, edge band, lead; the engine's anti-aliasing), so comparisons no longer need a relaunch (launches from the Mac fail most of the time). First verdict with it: **the adaptive diagonal makes no visible difference on the sphere** (head still), so the crenellations are not the cell split. - **What is left, measured, and the plan for it (2026-10-18, a 17-agent design panel: four proposals from four angles, three skeptics each, one synthesis; the panel re-measured the screenshot itself).** The capture is ~1.7 capture px per eye texel (the slab's 1.2 m edge = 1136 px, the sphere's 0.12 m = 85–94 px, both ≈ 1230 px per unit tangent against the texture's 738), so a 7.5-texel cell is 12.5 capture px. **At rest the mesh adds at most ~0.2 texel rms to a silhouette**: the sphere's outline deviates 0.31 px rms short-scale from a circle and the cell-period band of that holds 0.01–0.05 px, so the crenellations are not the grid (the adaptive diagonal's "no difference" agrees); the slab's lower edge is 0.18 px rms with a clean five-value MSAA ramp. The cubes' top sliver is 1–2 texels tall and its upper edge wobbles 0.66 px rms at 22–36 px periods: the four candidate causes are the band blending MSAA's quarter-coverage sub-steps of a shallow edge, the depth resolve taking sample 0 (which gives the object's depth to 0 % of a top edge's partial texels and 100 % of a bottom edge's, so the rim vertex's depth plane depends on the edge's orientation and per-column luck), the affine uv across a rim→face triangle (0.13–0.30 texel keystone), and parallax of a wrong rim depth plane; the first two vanish under FXAA, the last two do not, which is the next panel experiment (MSAA ↔ FXAA at one pose). **What the eye reads as "not the engine's quality" is the texture's density**: 1920 texels over a 2.6-tangent frustum = 12.9 texels per degree at the centre of view (a uniform perspective render is sparsest at its centre), against a display of ~34 px per degree, so RealityKit's bilinear sampler magnifies each texel to ~2.6 display px and the compositor resamples once more; the engine's 1-texel MSAA ramp and the sphere's Fresnel rim of a glossy limb (real on the compositor path too, at half a display px there) are shown 2.6× magnified. Ranked plan: **(1) the presenter's own fixed rasterization rate map**, equi-angular profile h(x) = 1/(1+x²) per axis, the same physical pixel count (iterate the screen scale down until `physicalSize` fits 1920 × 1824, since Metal rounds rates up), textures and grid at the physical size, the grid uniform in physical texels with cells dividing `physicalGranularity` (8 on 32), the kernel mapping texels to rays through `rasterization_rate_map_decoder` (uniforms +16 bytes, buffer 3), `XREye.foveation` carrying one `XRFoveationFrame` (the loop already attaches it): centre density 12.9 → 18.3 texels/deg at zero engine GPU cost, every texel-sized residue 1.4× smaller; about 150 lines and a day. **(2) Projective uv per vertex** (float4 uv2 = (u·ze, v·ze, ze), divided per pixel in the graph by `ND_separate4`/`ND_divide_float`/`ND_combine2`): RealityKit's perspective-correct interpolation then reproduces the engine's projection exactly on every planar triangle (API-verified on the Mac at float precision), +16 bytes per vertex, half a day. **(3) MSAA depth resolve by nearest sample**: `depthResolveFilter = .max` (reverse Z) on the offscreen and forward descriptors, one line twice, zero cost, shared with the compositor path (the render suites' references may move). **(4) The 512 × 480 grid** with a 50 m far shell, no code, after 1–3. **(5) The sphere's white teeth**: dump the limb texels on the Mac (extend `probeAlpha`); if the partial-coverage texels are ≥ 2× brighter than the interior, `[[centroid_perspective]]` on the vertex outputs mends it on both paths; if not, they are RealityKit's and unreachable. Judged no: a denser or non-uniform grid, a 1024 grid, a higher render scale (10 of 11.1 ms), mipmaps (always magnified), per-frame eye pose through `setParameter` (lands 1–2 frames late); the two-layer grid with a per-pixel depth test is right for the 8–15-texel halo of lost background around silhouettes (invisible against passthrough) and interior occlusion, 4–5 days, later. **What no change removes**: no gaze (RealityKit gives none, so foveation is head-centred and fixed, while the compositor path with `isFoveationEnabled` follows the eyes), two resamples instead of one, the MSAA ramp magnified; and two readings decide the honest statement: MSAA's GPU cost (unmeasured; the band needs it) and which engine app is the reference (the arcade examples and the build template set `isFoveationEnabled = false`, a uniform 2048 × 1984 per eye at 13.8 texels/deg, within 7 % of the presenter's; only UntoldViewer foveates). +- **Panel readings (2026-10-18, 13:40):** the adaptive diagonal makes no visible difference on the sphere (judged no, as the panel reasoned). **The wavy rim on the cubes' top faces stays under FXAA, in the same places**: the texture-side causes (the band blending MSAA's sub-steps, the sample-0 depth resolve) are out; what remains is geometry: the affine uv across the rim→face triangle (item 2, projective uv) or parallax of a wrong rim depth plane (the +3 cm eye offset test), or something outside the presenter's data path (unchanged by the 512 grid too). The rate-map work (item 1) started in its own session. +- **Item 2 done on the Mac (2026-10-18, 20:00): the projective uv.** Each vertex carries `uvw = (u·ze, v·ze, ze, 0)` (packed float4 at offset 24, stride 40, attribute uv2); both graphs read it through `ND_texcoord_vector4` index 2, `ND_separate4_vector4`, two `ND_divide_float` and `ND_combine2_vector2` into the texture read; the affine uv0 stays for the fallback material. The kernel test checks every vertex's uvw against its position's eye-space depth. **A first end-to-end test renders the presenter's grids with RealityKit on the Mac** (`RealityRenderer`, a perspective camera at the engine's eye through a symmetric 90° frustum, the eyes at the device's point) and compares the picture with the engine's eye texture: **identical, 88 dB, no differing pixel**, in mixed immersion. With an opaque engine background the same test showed the halo the panel predicted: a black frame one cell wide around the cube group, where the triangles stretched to the far shell are torn away and the background behind them is lost (invisible against passthrough; the two-layer grid's job later; 26 dB). The example's window gained choices for the next space opened (close and reopen it): the 512 × 480 grid, the 50 m shell, and a 3 cm upward view offset (`viewOffsetMetres` in the configuration) for the parallax test of the wavy rim. From ab931ebb1b5b648e0b5592413fa5fe50ce341d06 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 20:35:31 +0200 Subject: [PATCH 10/72] [Docs] XR RealityKit presenter plan: the projective uv judged on the headset (rim reduced), the end-to-end test hardened after review (90 vs 58 dB), the tear threshold at the test's scale --- docs/proposals/XRRealityKitPresenter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 649ef00ca..9dd0f18cf 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -219,3 +219,5 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **What is left, measured, and the plan for it (2026-10-18, a 17-agent design panel: four proposals from four angles, three skeptics each, one synthesis; the panel re-measured the screenshot itself).** The capture is ~1.7 capture px per eye texel (the slab's 1.2 m edge = 1136 px, the sphere's 0.12 m = 85–94 px, both ≈ 1230 px per unit tangent against the texture's 738), so a 7.5-texel cell is 12.5 capture px. **At rest the mesh adds at most ~0.2 texel rms to a silhouette**: the sphere's outline deviates 0.31 px rms short-scale from a circle and the cell-period band of that holds 0.01–0.05 px, so the crenellations are not the grid (the adaptive diagonal's "no difference" agrees); the slab's lower edge is 0.18 px rms with a clean five-value MSAA ramp. The cubes' top sliver is 1–2 texels tall and its upper edge wobbles 0.66 px rms at 22–36 px periods: the four candidate causes are the band blending MSAA's quarter-coverage sub-steps of a shallow edge, the depth resolve taking sample 0 (which gives the object's depth to 0 % of a top edge's partial texels and 100 % of a bottom edge's, so the rim vertex's depth plane depends on the edge's orientation and per-column luck), the affine uv across a rim→face triangle (0.13–0.30 texel keystone), and parallax of a wrong rim depth plane; the first two vanish under FXAA, the last two do not, which is the next panel experiment (MSAA ↔ FXAA at one pose). **What the eye reads as "not the engine's quality" is the texture's density**: 1920 texels over a 2.6-tangent frustum = 12.9 texels per degree at the centre of view (a uniform perspective render is sparsest at its centre), against a display of ~34 px per degree, so RealityKit's bilinear sampler magnifies each texel to ~2.6 display px and the compositor resamples once more; the engine's 1-texel MSAA ramp and the sphere's Fresnel rim of a glossy limb (real on the compositor path too, at half a display px there) are shown 2.6× magnified. Ranked plan: **(1) the presenter's own fixed rasterization rate map**, equi-angular profile h(x) = 1/(1+x²) per axis, the same physical pixel count (iterate the screen scale down until `physicalSize` fits 1920 × 1824, since Metal rounds rates up), textures and grid at the physical size, the grid uniform in physical texels with cells dividing `physicalGranularity` (8 on 32), the kernel mapping texels to rays through `rasterization_rate_map_decoder` (uniforms +16 bytes, buffer 3), `XREye.foveation` carrying one `XRFoveationFrame` (the loop already attaches it): centre density 12.9 → 18.3 texels/deg at zero engine GPU cost, every texel-sized residue 1.4× smaller; about 150 lines and a day. **(2) Projective uv per vertex** (float4 uv2 = (u·ze, v·ze, ze), divided per pixel in the graph by `ND_separate4`/`ND_divide_float`/`ND_combine2`): RealityKit's perspective-correct interpolation then reproduces the engine's projection exactly on every planar triangle (API-verified on the Mac at float precision), +16 bytes per vertex, half a day. **(3) MSAA depth resolve by nearest sample**: `depthResolveFilter = .max` (reverse Z) on the offscreen and forward descriptors, one line twice, zero cost, shared with the compositor path (the render suites' references may move). **(4) The 512 × 480 grid** with a 50 m far shell, no code, after 1–3. **(5) The sphere's white teeth**: dump the limb texels on the Mac (extend `probeAlpha`); if the partial-coverage texels are ≥ 2× brighter than the interior, `[[centroid_perspective]]` on the vertex outputs mends it on both paths; if not, they are RealityKit's and unreachable. Judged no: a denser or non-uniform grid, a 1024 grid, a higher render scale (10 of 11.1 ms), mipmaps (always magnified), per-frame eye pose through `setParameter` (lands 1–2 frames late); the two-layer grid with a per-pixel depth test is right for the 8–15-texel halo of lost background around silhouettes (invisible against passthrough) and interior occlusion, 4–5 days, later. **What no change removes**: no gaze (RealityKit gives none, so foveation is head-centred and fixed, while the compositor path with `isFoveationEnabled` follows the eyes), two resamples instead of one, the MSAA ramp magnified; and two readings decide the honest statement: MSAA's GPU cost (unmeasured; the band needs it) and which engine app is the reference (the arcade examples and the build template set `isFoveationEnabled = false`, a uniform 2048 × 1984 per eye at 13.8 texels/deg, within 7 % of the presenter's; only UntoldViewer foveates). - **Panel readings (2026-10-18, 13:40):** the adaptive diagonal makes no visible difference on the sphere (judged no, as the panel reasoned). **The wavy rim on the cubes' top faces stays under FXAA, in the same places**: the texture-side causes (the band blending MSAA's sub-steps, the sample-0 depth resolve) are out; what remains is geometry: the affine uv across the rim→face triangle (item 2, projective uv) or parallax of a wrong rim depth plane (the +3 cm eye offset test), or something outside the presenter's data path (unchanged by the 512 grid too). The rate-map work (item 1) started in its own session. - **Item 2 done on the Mac (2026-10-18, 20:00): the projective uv.** Each vertex carries `uvw = (u·ze, v·ze, ze, 0)` (packed float4 at offset 24, stride 40, attribute uv2); both graphs read it through `ND_texcoord_vector4` index 2, `ND_separate4_vector4`, two `ND_divide_float` and `ND_combine2_vector2` into the texture read; the affine uv0 stays for the fallback material. The kernel test checks every vertex's uvw against its position's eye-space depth. **A first end-to-end test renders the presenter's grids with RealityKit on the Mac** (`RealityRenderer`, a perspective camera at the engine's eye through a symmetric 90° frustum, the eyes at the device's point) and compares the picture with the engine's eye texture: **identical, 88 dB, no differing pixel**, in mixed immersion. With an opaque engine background the same test showed the halo the panel predicted: a black frame one cell wide around the cube group, where the triangles stretched to the far shell are torn away and the background behind them is lost (invisible against passthrough; the two-layer grid's job later; 26 dB). The example's window gained choices for the next space opened (close and reopen it): the 512 × 480 grid, the 50 m shell, and a 3 cm upward view offset (`viewOffsetMetres` in the configuration) for the parallax test of the wavy rim. +- **Headset verdict on item 2 (2026-10-18, 20:15): "with the projective uv build the wavy rim on the cube tops is reduced"** (skirt 3, MSAA, the panel's settings). The affine keystone was a real part of it; what remains is judged next with the view offset (a rim vertex at a wrong depth would move with it) and the 512 grid. +- **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. From 9c4781a4930e087d7b9d5a2553cf0e012e24f3bd Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 22:04:01 +0200 Subject: [PATCH 11/72] [Docs] XR RealityKit presenter plan: the equi-angular rate map landed (item 1), merged from its session; the one-headset rule --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 9dd0f18cf..baff1c320 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -221,3 +221,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Item 2 done on the Mac (2026-10-18, 20:00): the projective uv.** Each vertex carries `uvw = (u·ze, v·ze, ze, 0)` (packed float4 at offset 24, stride 40, attribute uv2); both graphs read it through `ND_texcoord_vector4` index 2, `ND_separate4_vector4`, two `ND_divide_float` and `ND_combine2_vector2` into the texture read; the affine uv0 stays for the fallback material. The kernel test checks every vertex's uvw against its position's eye-space depth. **A first end-to-end test renders the presenter's grids with RealityKit on the Mac** (`RealityRenderer`, a perspective camera at the engine's eye through a symmetric 90° frustum, the eyes at the device's point) and compares the picture with the engine's eye texture: **identical, 88 dB, no differing pixel**, in mixed immersion. With an opaque engine background the same test showed the halo the panel predicted: a black frame one cell wide around the cube group, where the triangles stretched to the far shell are torn away and the background behind them is lost (invisible against passthrough; the two-layer grid's job later; 26 dB). The example's window gained choices for the next space opened (close and reopen it): the 512 × 480 grid, the 50 m shell, and a 3 cm upward view offset (`viewOffsetMetres` in the configuration) for the parallax test of the wavy rim. - **Headset verdict on item 2 (2026-10-18, 20:15): "with the projective uv build the wavy rim on the cube tops is reduced"** (skirt 3, MSAA, the panel's settings). The affine keystone was a real part of it; what remains is judged next with the view offset (a rim vertex at a wrong depth would move with it) and the 512 grid. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. +- **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. Headset verdict on the map pending. From bdfd6ecc49ee8af468ded26517aa732602a3dd57 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 22:12:08 +0200 Subject: [PATCH 12/72] [Docs] XR RealityKit presenter plan: the rate map judged on the headset (a little sharper at the centre) --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index baff1c320..f9aa66e6b 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -221,4 +221,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Item 2 done on the Mac (2026-10-18, 20:00): the projective uv.** Each vertex carries `uvw = (u·ze, v·ze, ze, 0)` (packed float4 at offset 24, stride 40, attribute uv2); both graphs read it through `ND_texcoord_vector4` index 2, `ND_separate4_vector4`, two `ND_divide_float` and `ND_combine2_vector2` into the texture read; the affine uv0 stays for the fallback material. The kernel test checks every vertex's uvw against its position's eye-space depth. **A first end-to-end test renders the presenter's grids with RealityKit on the Mac** (`RealityRenderer`, a perspective camera at the engine's eye through a symmetric 90° frustum, the eyes at the device's point) and compares the picture with the engine's eye texture: **identical, 88 dB, no differing pixel**, in mixed immersion. With an opaque engine background the same test showed the halo the panel predicted: a black frame one cell wide around the cube group, where the triangles stretched to the far shell are torn away and the background behind them is lost (invisible against passthrough; the two-layer grid's job later; 26 dB). The example's window gained choices for the next space opened (close and reopen it): the 512 × 480 grid, the 50 m shell, and a 3 cm upward view offset (`viewOffsetMetres` in the configuration) for the parallax test of the wavy rim. - **Headset verdict on item 2 (2026-10-18, 20:15): "with the projective uv build the wavy rim on the cube tops is reduced"** (skirt 3, MSAA, the panel's settings). The affine keystone was a real part of it; what remains is judged next with the view offset (a rim vertex at a wrong depth would move with it) and the 512 grid. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. -- **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. Headset verdict on the map pending. +- **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From f498372d975eaf02d2c48f9d75349a6bd3d2f876 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 22:15:44 +0200 Subject: [PATCH 13/72] [Docs] XR RealityKit presenter plan: the view-offset reading (the rim is not a wrong-depth vertex) --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index f9aa66e6b..7376c3413 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -220,5 +220,6 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Panel readings (2026-10-18, 13:40):** the adaptive diagonal makes no visible difference on the sphere (judged no, as the panel reasoned). **The wavy rim on the cubes' top faces stays under FXAA, in the same places**: the texture-side causes (the band blending MSAA's sub-steps, the sample-0 depth resolve) are out; what remains is geometry: the affine uv across the rim→face triangle (item 2, projective uv) or parallax of a wrong rim depth plane (the +3 cm eye offset test), or something outside the presenter's data path (unchanged by the 512 grid too). The rate-map work (item 1) started in its own session. - **Item 2 done on the Mac (2026-10-18, 20:00): the projective uv.** Each vertex carries `uvw = (u·ze, v·ze, ze, 0)` (packed float4 at offset 24, stride 40, attribute uv2); both graphs read it through `ND_texcoord_vector4` index 2, `ND_separate4_vector4`, two `ND_divide_float` and `ND_combine2_vector2` into the texture read; the affine uv0 stays for the fallback material. The kernel test checks every vertex's uvw against its position's eye-space depth. **A first end-to-end test renders the presenter's grids with RealityKit on the Mac** (`RealityRenderer`, a perspective camera at the engine's eye through a symmetric 90° frustum, the eyes at the device's point) and compares the picture with the engine's eye texture: **identical, 88 dB, no differing pixel**, in mixed immersion. With an opaque engine background the same test showed the halo the panel predicted: a black frame one cell wide around the cube group, where the triangles stretched to the far shell are torn away and the background behind them is lost (invisible against passthrough; the two-layer grid's job later; 26 dB). The example's window gained choices for the next space opened (close and reopen it): the 512 × 480 grid, the 50 m shell, and a 3 cm upward view offset (`viewOffsetMetres` in the configuration) for the parallax test of the wavy rim. - **Headset verdict on item 2 (2026-10-18, 20:15): "with the projective uv build the wavy rim on the cube tops is reduced"** (skirt 3, MSAA, the panel's settings). The affine keystone was a real part of it; what remains is judged next with the view offset (a rim vertex at a wrong depth would move with it) and the 512 grid. +- **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 3c2f0b76d1b3fcf89378bba3a460739db636956b Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 23:11:06 +0200 Subject: [PATCH 14/72] [Docs] XR RealityKit presenter plan: the 512 grid reading (the cube-top rim is outside the presenter's path) --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 7376c3413..072fbb3e3 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -221,5 +221,6 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Item 2 done on the Mac (2026-10-18, 20:00): the projective uv.** Each vertex carries `uvw = (u·ze, v·ze, ze, 0)` (packed float4 at offset 24, stride 40, attribute uv2); both graphs read it through `ND_texcoord_vector4` index 2, `ND_separate4_vector4`, two `ND_divide_float` and `ND_combine2_vector2` into the texture read; the affine uv0 stays for the fallback material. The kernel test checks every vertex's uvw against its position's eye-space depth. **A first end-to-end test renders the presenter's grids with RealityKit on the Mac** (`RealityRenderer`, a perspective camera at the engine's eye through a symmetric 90° frustum, the eyes at the device's point) and compares the picture with the engine's eye texture: **identical, 88 dB, no differing pixel**, in mixed immersion. With an opaque engine background the same test showed the halo the panel predicted: a black frame one cell wide around the cube group, where the triangles stretched to the far shell are torn away and the background behind them is lost (invisible against passthrough; the two-layer grid's job later; 26 dB). The example's window gained choices for the next space opened (close and reopen it): the 512 × 480 grid, the 50 m shell, and a 3 cm upward view offset (`viewOffsetMetres` in the configuration) for the parallax test of the wavy rim. - **Headset verdict on item 2 (2026-10-18, 20:15): "with the projective uv build the wavy rim on the cube tops is reduced"** (skirt 3, MSAA, the panel's settings). The affine keystone was a real part of it; what remains is judged next with the view offset (a rim vertex at a wrong depth would move with it) and the 512 grid. - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). +- **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From e51d0cbcf3ae9a2621604764ba3a9fc4ad165b03 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 23:28:06 +0200 Subject: [PATCH 15/72] [Docs] XR RealityKit presenter plan: the dark reopen found and fixed on the Mac; the device GPU reading with the map and MSAA --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 072fbb3e3..2cbad4dbc 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -222,5 +222,6 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Headset verdict on item 2 (2026-10-18, 20:15): "with the projective uv build the wavy rim on the cube tops is reduced"** (skirt 3, MSAA, the panel's settings). The affine keystone was a real part of it; what remains is judged next with the view offset (a rim vertex at a wrong depth would move with it) and the 512 grid. - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. +- **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From f7b8e2ec2a1efd66101573c307dc8a6b5b4c8989 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 23:47:40 +0200 Subject: [PATCH 16/72] [Docs] XR RealityKit presenter plan: the device's frame cost after the day's changes, and the rim measured in the engine's own texture (the staircase, the 0.9 cut, the tone map on premultiplied colour) --- docs/proposals/XRRealityKitPresenter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 2cbad4dbc..336a17abe 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,5 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. +- **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 762b42a64198d3fc4a725946802ee1ecdf0aee90 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Sun, 18 Oct 2026 23:50:25 +0200 Subject: [PATCH 17/72] [Docs] XR RealityKit presenter plan: the band's cost on the device (11 ms), the next two pieces --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 336a17abe..0960b2923 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Next: the band over the rim cells only (a per-frame index buffer of the cells that touch a rim or near-side vertex, on a mesh of its own, the rest degenerate), expected under a millisecond; and the look pass's tone map on un-premultiplied colour. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From d7b411c49e7e038fd1807bf7dee704a726e24b2c Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 01:53:24 +0200 Subject: [PATCH 18/72] [Docs] XR RealityKit presenter plan: the band over rim cells and the look pass on un-premultiplied colour merged --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 0960b2923..e3af5d1ab 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Next: the band over the rim cells only (a per-frame index buffer of the cells that touch a rim or near-side vertex, on a mesh of its own, the rest degenerate), expected under a millisecond; and the look pass's tone map on un-premultiplied colour. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). The device's GPU with the cheap band is the next reading. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 7e227b7052f707fd3b2ee6132b6a0ea890248efd Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 02:05:48 +0200 Subject: [PATCH 19/72] [Docs] XR RealityKit presenter plan: the rim-cell band costs 2 ms on the device; the 16 ms baseline is the open item --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index e3af5d1ab..07a8ec85d 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). The device's GPU with the cheap band is the next reading. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From dc800e093458231fc8090005822d2999a418b634 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 02:08:48 +0200 Subject: [PATCH 20/72] [Docs] XR RealityKit presenter plan: the kernel's share of the device frame (4-5 ms) from the 512 grid reading, and its fast path --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 07a8ec85d..d6ff00b36 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 89717506d6cdb517b4444c64a94e177f2f2d6f32 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:33:21 +0200 Subject: [PATCH 21/72] [Docs] XR RealityKit presenter plan: the kernel's block fast path merged (15x cheaper on the Mac, identical entries) --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index d6ff00b36..0c731fa66 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. The device's reading with it is next. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From ce2a3fe2799cd61ef306c12d8d372560abbae311 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:41:45 +0200 Subject: [PATCH 22/72] [Docs] XR RealityKit presenter plan: the device frame back at 9.8 ms with the kernel's fast path --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 0c731fa66..05c93b828 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. The device's reading with it is next. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 9.8 ms (max 18) at 256 × 240 with the band off**, from 16: the old footprint pass cost about 6 ms on the device. The frame is under its 11.1 ms again before the band (2 ms) and MSAA (3 ms), which together reach the edge of the budget; the rate map's own cost on the device is still unread. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From c5f07cb3802a5e0d95daec08106a7bd3231fb0ed Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:43:08 +0200 Subject: [PATCH 23/72] [Docs] XR RealityKit presenter plan: the device frame is 3.8 ms at 256x240 with the fast path (the earlier 9.8 was the 512 grid) --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 05c93b828..d1e8162bd 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 9.8 ms (max 18) at 256 × 240 with the band off**, from 16: the old footprint pass cost about 6 ms on the device. The frame is under its 11.1 ms again before the band (2 ms) and MSAA (3 ms), which together reach the edge of the budget; the rate map's own cost on the device is still unread. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. With the band (2 ms) and MSAA (3 ms) the frame should sit near 9 ms of the 11.1; the rate map's own cost on the device is still unread. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From f168dd34f576b29eda7e2632636245e2e9a01577 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:45:24 +0200 Subject: [PATCH 24/72] [Docs] XR RealityKit presenter plan: the device frame at 3.6 ms with the band and MSAA on --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index d1e8162bd..c5faa0ca9 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. With the band (2 ms) and MSAA (3 ms) the frame should sit near 9 ms of the 11.1; the rate map's own cost on the device is still unread. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. Everything judged today fits in a third of the frame; the rate map's reading on the device is the one left. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From d6e12ce55e7e2e03fcedfdebed99fc7ea024661d Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:48:21 +0200 Subject: [PATCH 25/72] [Docs] XR RealityKit presenter plan: the device frame at 3.8 ms with everything on; what is left --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index c5faa0ca9..e14f153fb 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. Everything judged today fits in a third of the frame; the rate map's reading on the device is the one left. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 29edcb6ae83b78458dcb265d483ddad28600d837 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:50:00 +0200 Subject: [PATCH 26/72] [Docs] XR RealityKit presenter plan: the headset section's frame times are Debug-build numbers --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index e14f153fb..281a6d0e6 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -200,7 +200,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - Simulator: at 256 × 240 the sphere loses its one-cell bites (vertices whose taps missed the edge) and the slab's staircase is unchanged; at 512 × 480 the staircase halves as before. Numbers unchanged (replace 7.9–8.2 ms, frame 2.9–3.4 ms, GPU 0.3–0.5 ms at either grid). - The dashed hairline above the sphere at 512 × 480 is NOT a sampling gap, so it stays: the eye's textures and grid dumped from the simulator show no texel with colour and no depth, every texel in some footprint, and a software rasterization of that grid from the simulator's viewpoint draws no hairline. In RealityKit the line is one pixel row (about a quarter covered) along the grid row above the sphere's top, carrying the near-side mask and fully opaque sphere texels: it is there with the eye at the camera (no parallax), it goes with a far shell of 50 m or less or a cut epsilon of 0.01 or more (which brings fins back), and it moves to other cell edges with 481 rows. That is RealityKit's interpolation of a stretched triangle's attributes between a vertex at 1.6 m and one at 100 m, which the footprint cannot touch; the knob is the far shell (30 or 50 m give a clean 512 grid in the simulator), a decision for the headset stage together with the relative tear threshold. The default grid stays 256 × 240 until then. Screenshots, crops, the usda-swap and config-run scripts and the analysis scripts are with the hand-over note of this session (`xr-presenter-footprint`). -### The headset run (2026-10-18, Javier's Apple Vision Pro, visionOS 27.0.1; both apps installed and launched from the Mac with `devicectl`, the spike's log read through its console with `OS_ACTIVITY_DT_MODE=YES`) +### The headset run (2026-10-18, Javier's Apple Vision Pro, visionOS 27.0.1; every frame-time number in this section comes from DEBUG builds of the example and the engine package, which compare with each other but not with a release build, whose render time differs greatly: release readings to follow; both apps installed and launched from the Mac with `devicectl`, the spike's log read through its console with `OS_ACTIVITY_DT_MODE=YES`) - **D2 settled: `replace(using:)` stays.** The spike at full size (1920 × 1824 per eye, grids 256 × 240, the replace route): 90.0 fps, 90.0 RealityKit updates/s, 0 dropped tickets, the ticket's age at pickup 0.03 ms, **the main thread inside the replace calls 0.3 ms per frame (textures 0.10, grids 0.19; max 0.6)**, encode 0.35 ms, GPU 1.6 ms (max spikes 3–11 ms), the lead 22.2 ms. The simulator's 8 ms per frame was the simulator's own cost. The IOSurface and shared-handle routes are not needed; they stay in the spike as switches. - The eye material loaded on the device (`Eye material loaded; parameters: colorTexture, eye`); the system log also prints a harmless `MaterialX node def file not found` line from ShaderGraph at load. From f77149ea53264f34ef2bf7dcc1c234d22eb2ba15 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 07:57:41 +0200 Subject: [PATCH 27/72] [Docs] XR RealityKit presenter plan: the Release build's first device reading, unexplained against Debug --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 281a6d0e6..8b4fcc1a6 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), to be separated with the band off and FXAA on the Release build. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 7c270a42bcc3e55910a696b4d3b3f3f1c2e54a28 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 08:00:22 +0200 Subject: [PATCH 28/72] [Docs] XR RealityKit presenter plan: the band and MSAA cost more on the Release build than on Debug; open --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 8b4fcc1a6..628326c48 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), to be separated with the band off and FXAA on the Release build. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. Open; the next step is a GPU frame capture on the device for each build with the same settings, which shows what each frame draws and where the milliseconds sit. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From f68d75196fecbe29196af8204aa06b3187cc7505 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 08:03:57 +0200 Subject: [PATCH 29/72] [Docs] XR RealityKit presenter plan: the Release build's band-off FXAA reading --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 628326c48..94fa37508 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. Open; the next step is a GPU frame capture on the device for each build with the same settings, which shows what each frame draws and where the milliseconds sit. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). Open; the next step is a GPU frame capture on the device for each build with the same settings, which shows what each frame draws and where the milliseconds sit (in its own session). Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 87cba6b2b1b8df2043f376d7f41f6b0ec0ed9465 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 08:35:30 +0200 Subject: [PATCH 30/72] [Docs] XR RealityKit presenter plan: the Release/Debug gap was the device's state, not the build (probe apps: 2.9 vs 3.6 ms) --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 94fa37508..c32e1cc51 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). Open; the next step is a GPU frame capture on the device for each build with the same settings, which shows what each frame draws and where the milliseconds sit (in its own session). Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs; the full report follows. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From c4ef407a42911eaf553962a9376c8b26f15d725c Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 08:39:46 +0200 Subject: [PATCH 31/72] [Docs] XR RealityKit presenter plan: the numbers line carries the engine's GPU time and the thermal state (merged) --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index c32e1cc51..ed6fe29bd 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs; the full report follows. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs. The same session added the engine's own GPU time and the device's thermal state to the numbers line beside the ticket's span (engine `df216eb5`, example `5f0dc4f`, both fast-forwarded into the branches on 2026-10-19), so a reading names a throttled device when it is one. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 8ab3f5b40835753dbd27aeb4a9f7b1d608789831 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 08:48:18 +0200 Subject: [PATCH 32/72] [Docs] XR RealityKit presenter plan: the 512 grid's price on a nominal device with the new numbers line --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index ed6fe29bd..9f7736b13 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs. The same session added the engine's own GPU time and the device's thermal state to the numbers line beside the ticket's span (engine `df216eb5`, example `5f0dc4f`, both fast-forwarded into the branches on 2026-10-19), so a reading names a throttled device when it is one. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs. The same session added the engine's own GPU time and the device's thermal state to the numbers line beside the ticket's span (engine `df216eb5`, example `5f0dc4f`, both fast-forwarded into the branches on 2026-10-19), so a reading names a throttled device when it is one. First reading with it, the device nominal: **9.2 ms of span, 9.4 ms of the engine's own GPU, on the 512 grid with the map on** (the map's grid at four-texel cells, four times the vertices): the 512 grid's own price, as Debug's 9.8 ms was; the kernel's remaining scan work scales with the vertex count. The 512 grid showed no visible gain on the headset, so 256 × 240 stays the setting. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From ffeb5fcd3a4ad2bf3b2891b0ee199d9c8a8e784d Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 08:50:11 +0200 Subject: [PATCH 33/72] [Docs] XR RealityKit presenter plan: the closing frame reading, 4.4 ms of span and 4.6 ms of engine GPU on a nominal device in Release --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 9f7736b13..ba1707c16 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -223,7 +223,7 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The view-offset reading (2026-10-18, 22:15): "not much difference"** with both viewpoints 3 cm up for the next space: the picture stays in place (the reprojection holds across a deliberate pose error) and the remaining rim on the cube tops does not move or grow, so it is not a rim vertex at a wrong depth. Left: per-cell residue (the 512 grid halves it) or something outside the presenter's data path (RealityKit's rasterization of a one-to-two-pixel sliver, the compositor's resample). - **The 512 grid reading (2026-10-18, 22:25): the rim on the cube tops is unchanged** on 512 × 480 with the 50 m shell. So it is unchanged by FXAA against MSAA, by the view offset and by the cell size, and only reduced by the projective uv: by the panel's own readout it lies outside the presenter's data path: the engine's rendering of a grazing one-to-two-texel sliver at a near-horizontal edge (its coverage steps along the edge), magnified by the resampling onto the display, and RealityKit's rasterization of the thin band. To be verified on the Mac by rendering the same grazing geometry into the eye texture and measuring the edge itself. - **A reopened space drew nothing (2026-10-18, 23:00), found and fixed on the Mac.** The presenter view closed a session by stopping the presenter (the loop ends with the mailbox) and then resetting the engine's world; the engine defers destroying the entities to the start of a frame and runs the reset's completion then (the active camera cleared, the loop stopped), so with the old loop gone that frame was the next session's first, which lost its camera and drew nothing (`renderXR` returns without a camera; the eye textures keep RealityKit's uninitialised content). Why the earlier reopens showed the scene is unverified (the engine's camera sync may have recovered it; the dark reopen came "as soon as" the space was reopened). The fix: `XRRealityKitPresenter.stop(after:)` hands the shutdown a completion, posts the ticket of one more frame so the old loop runs the destroys and the completion itself, and ends the exchange when the completion fires (or after a second); the view's `close()` uses it with `shutdownUntoldEngineXR(xr, completion:)`. A Mac test closes a session that way and opens another: the same picture, 83 dB; with the old order the second session's eye is garbage, 1.9 dB. Also read on the device before this: **GPU 29 ms average, 34 max, with the rate map on and MSAA** (grid possibly 512), against the 11.1 ms frame and 10 ms before both; the shares are not yet separated. -- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs. The same session added the engine's own GPU time and the device's thermal state to the numbers line beside the ticket's span (engine `df216eb5`, example `5f0dc4f`, both fast-forwarded into the branches on 2026-10-19), so a reading names a throttled device when it is one. First reading with it, the device nominal: **9.2 ms of span, 9.4 ms of the engine's own GPU, on the 512 grid with the map on** (the map's grid at four-texel cells, four times the vertices): the 512 grid's own price, as Debug's 9.8 ms was; the kernel's remaining scan work scales with the vertex count. The 512 grid showed no visible gain on the headset, so 256 × 240 stays the setting. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. +- **The frame's cost on the device (2026-10-18, 23:55), from the window's numbers:** with no rate map, the 256 × 240 grid, the 100 m shell and FXAA: **GPU 27.4 ms (max 36.4), 65 frames per second, the main thread's replace calls 13.4 ms (max 20.3)**, against the morning's 10 ms, 90 and 0.05 ms. MSAA adds about 3 ms (29.4 against 26 with the map on). So the cost came in with the day's engine changes, and the edge band is the suspect: a second blended entity per eye over the whole grid, over a core alpha-tested at 0.9 (no early depth rejection); the band's toggle applies at once and separates it. **Band off: 16 ms (max 27)**, so the band costs about 11 ms on the device, a second blended pass over the whole grid per eye; the remaining 6 ms over the morning are unexplained (the core's draw at the display's resolution, the every-texel footprint, the projective graph's divide). Both landed in their own sessions and were merged into the presenter branch on 2026-10-19 (merge `235427b4`, pushed): **the band over the rim cells only** (`c111c76c`: the band on a mesh of its own per eye, a fourth kernel pass keeping the cells that touch a rim or tear vertex, 19 % of the cells on the kernel test scene and 6.8 % around the three cubes, the rest degenerate; 31 tests, the picture identical to the whole-grid band under MSAA) and **the look pass grading and tone-mapping the colour behind a texel's coverage** (`8554445e`, rebuilt kernels: the rim over the face from 1.65 / 1.46 / 1.20 under ACES and up to 2.72 under AgX to 0.99–1.00; a new test, 108 render tests and the PSNR references pass, the opaque path unchanged). **Device reading (2026-10-19): GPU 16 ms without the band, 18 ms with it**: the band costs 2 ms now, against 11. The 16 ms baseline against the morning's 10 (the stale build's old 4 × 4-tap kernel) is the open item: the every-texel footprint pass (about 140 reads per vertex, 61k vertices per eye), the core's draw at the display's resolution with the projective graph, or RealityKit's own work in the measured span; the 512 grid reading (four times the vertices) separates the kernel's share. **512 × 480, band off: GPU 30 ms (max 38)**, so four times the vertices add 14 ms and the kernel's share at 256 × 240 is about 4–5 ms: the every-texel footprint pass, about 240 reads per vertex (the cell of 7.5 texels plus half a cell and the margin each side) for 61k vertices per eye, at 90 Hz. The remaining 11 ms are the engine's own render of two 1920 × 1824 eyes (9–10 ms this morning, the same on the compositor path) and RealityKit's draw inside the measured span. The kernel's fix, next: a block-minimum pass over the depth (8 × 8 blocks), and a fast path in pass 1 for the vertex whose own centre texel lies within the tear threshold of the footprint's nearest depth read from the blocks (a superset of the footprint, so the test is exact when it passes): the chosen texel is then the centre texel, no rim, about ten reads; only rim vertices scan the footprint. Identical results by construction. **Landed (2026-10-19, its own session, `83029e81`, fast-forwarded into the presenter branch and pushed):** a block pass writes each 8 × 8 block's nearest depth, pass 1 settles a vertex from the blocks when its centre texel lies within the threshold of the covering blocks' nearest, and the vertices that need the scan are gathered into a dense dispatch; byte-identical entries, 11 kernel tests and 34 presenter tests; the unprojection at the headset's size on the Mac from 0.689 ms to 0.045 ms median. **Device reading: GPU 3.8 ms (max 16) at 256 × 240 and 9.8 ms (max 18) at 512 × 480, the band off**, from 16 and 30: the old footprint pass cost about 12 ms of the morning's frame at 256 (the morning's 10 ms were mostly it), and the engine's render of this small scene is a few milliseconds. **With the band on and MSAA: 3.6 ms (max 16)**: the band's and MSAA's earlier shares (2 and 3 ms) were the saturated GPU's contention under the old kernel, not their own cost. **With the rate map on as well: 3.8 ms (max 12).** A RELEASE build of the same (2026-10-19, 01:45; the settings line confirming 256 × 240, the map, the band and MSAA) reads **9.5 ms (max 18)**, which the optimisation level cannot explain on its own; unexplained at the time of writing (the device's temperature after hours of use, the engine's Release-only paths, or the measurement itself), On the Release build, the band off and FXAA each drop the GPU clearly, so the band and MSAA cost more in Release than in Debug, which the optimisation level cannot do by itself: either a Debug-only gate leaves them without their full work, or the two builds' measured GPU spans count different things. A further Release reading: the map on (the grid then the map's 241 × 229), the band off, FXAA: **4.8 ms (max 12.1)**, so on Release the band and MSAA together cost about 4.7 ms and on Debug about nothing, with similar base frames (3.8 against 4.8). **Resolved the next morning (2026-10-19) by its own session**, with two probe apps under their own bundle ids: the same settings read **Debug 2.9 ms and Release 3.6 ms**, the engine copy byte-identical to the branch tip, so the build was not the cause; the evening's 9.5 ms, and the 4.7 ms the band and MSAA seemed to cost on Release, were the device's state after hours of use. Readings on the device compare only when taken minutes apart on a cool headset; Instruments cannot attach over Wi-Fi (the Developer Strap would give a system GPU timeline), so the evidence is the per-app GPU captures and numbers logs. The same session added the engine's own GPU time and the device's thermal state to the numbers line beside the ticket's span (engine `df216eb5`, example `5f0dc4f`, both fast-forwarded into the branches on 2026-10-19), so a reading names a throttled device when it is one. First reading with it, the device nominal: **9.2 ms of span, 9.4 ms of the engine's own GPU, on the 512 grid with the map on** (the map's grid at four-texel cells, four times the vertices): the 512 grid's own price, as Debug's 9.8 ms was; the kernel's remaining scan work scales with the vertex count. The 512 grid showed no visible gain on the headset, so 256 × 240 stays the setting. **The closing reading (2026-10-19, Release build, device nominal, 256 × 240, the rate map, the band and MSAA on): 4.4 ms of span, 4.6 ms of the engine's own GPU**, of the 11.1 ms frame: the presenter with everything judged in these two days takes about four milliseconds, measured by the engine's own clock on a cool headset. Everything judged today, on together, takes a third of the 11.1 ms frame on the device: the kernel's fast path, the rim-cell band with MSAA coverage over a 0.9 core, the projective uv, the equi-angular rate map, the settings and the look pass on un-premultiplied colour. What is left for the picture: the engine's one-texel staircase along near-horizontal edges reaching the display through the 0.9 cut (lower the cut or let the band carry the partial texels at full weight, then TAA), the halo of lost background around silhouettes when a scene draws its own background (the two-layer grid), a relative tear threshold for far oblique surfaces; for the branch: the fork PRs' descriptions with before-and-after pictures, and issue #1371. - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). From 95e2bd189cb4d7b4e873e7556b9faf0e21d5a8b2 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:13:55 +0200 Subject: [PATCH 34/72] [Docs] XRRealityKitPresenter plan: the cut and band measurement (nothing beats the shipped default) and the adaptive lead with the swim probe recorded --- docs/proposals/XRRealityKitPresenter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index ba1707c16..8338653a4 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -227,3 +227,5 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Is the rim in the engine's texture? Measured on the Mac (2026-10-18, a 5-agent panel rendering the example scene through the presenter's path at 1920 × 1824, the eyes 3.5 cm above the cube tops to match the headset's sliver of 1.9–2.2 texels).** Partly. (1) Read as a hard edge (no AA, FXAA whose alpha stays near binary at 12–21/255, or MSAA cut at a threshold), the top faces' far edge wanders **0.26–0.31 texel rms about a line, one full texel peak to peak, with period 1/slope** (24–26 texels at 0.040 rows per column): the quantisation constant 1/√12, independent of the slope. (2) Read through its MSAA coverage, the same edge is straight to 0.07 texel rms with 6-texel periods: a correctly filtered MSAA texture cannot make the rim; **the presenter's 0.9 cut discards most of the coverage, so the staircase reaches the display nearly unfiltered**, bilinear magnification keeping the step. (3) Under MSAA the partly covered rim texels are **1.27–1.30× brighter than the face** at quarter and half coverage: not shading extrapolated at the pixel centre (it survives nothing of a tone-map switch) but **the look pass applying ACES to premultiplied colour**, T(a·c) > a·T(c); the brightness pulses at the staircase period. Fix: tone-map un-premultiplied colour in LookShader.metal (divide by alpha before the grade, re-multiply after, guard zero), a few lines, correct for everything drawn over passthrough with partial alpha. (4) The headset's 0.34–0.40 texel rms exceeds the texture's largest possible staircase (0.29), so part of it is outside the eye texture (the grid, the band, RealityKit's raster, or the capture's own quantisation, 0.29 px if read at integer pixels); the period family fits a 3° head roll, unmeasured. Levers ranked: the tone-map fix (verified); lowering the core's cut or letting the band carry the partial texels at full weight (cheap, decides between two simulations: MSAA could come out at 0.14–0.17 texel against 0.26–0.31 binary); more centre density (the map, done); TAA (strongest against the staircase, most expensive: history, reprojection with the XR pose, ghosting); MSAA 8× low value; centroid interpolation no gain. - **The end-to-end test, reviewed and hardened (2026-10-18, 20:35).** An 11-agent review confirmed the mathematics, the kernel and the layout (an offset-camera experiment through the repository's graph matched a ray-cast reference to 0.26 texel mean with the projective uv against 38 texels with the affine one) and faulted the test four times: its content guard was dead (RealityRenderer's output has alpha 1 everywhere), nothing checked the engine's picture, a stall became a skip, and the affine graph passed it (64.6 dB against a 30 dB threshold). Now: colour-only guards on both pictures, a finite ratio, a failing timeout, the cubes turned 35° so their faces run across cells at changing depth, four-texel cells, and a 70 dB threshold: **projective 90 dB (identical at 8 bits), affine 58.4 dB on the same scene.** On the way, the test showed the absolute tear threshold at work: at this scale (39 mm per texel at the cubes) a turned cube's grazing side face changes its distance to the eye by 0.52 m between neighbouring vertices, so the 0.5 m threshold tore it and RealityKit lost a one-cell strip of the face (34.9 dB); the test's threshold is 1 m. The headset's 2 mm texels never reach that on such a face, but a relative threshold (a fraction of the distance, or a slope test) stays on the list for far oblique surfaces. - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). +- **The cut and the band's weight, measured on the Mac (2026-10-19, 11:13, a side session; branch `feature/xr_presenter_opacity_cut` `e622d22d` on `df216eb5`, pushed, no PR: a test and a note in the `.usda`, no production change).** `XRPresenterEdgeStraightnessTests` renders the example scene through the presenter's chain (RealityRenderer, the left eye 35 mm above the middle cube's top, the head rolled atan(0.04) so the top face's far edge crosses one row every 25 columns, 4 picture pixels per texel, over a black and a white room to recover each pixel's opacity) and fits lines to five edge stretches; deviations in eye texels. The texture's own coverage edge: 0.074 rms (the quarter steps of 4× MSAA); read as a hard edge 0.29. **Shipped (cut 0.9, band at coverage): 0.066 rms, at the coverage floor, bias 0, no halo**; the rim-cell band equals a whole-grid band to the last digit. The cut without the band 0.22; cuts 0.5 / 0.25 / 0.1 with the band 0.09 / 0.13 / 0.22 and the object grown over the room by 0.2 / 0.44 / 0.61 texel; the band at double weight 0.12; the band at full weight 0.30 with a 0.9-texel halo. The rim's colour behind its opacity is the face's (0.98) in every blended variant. **So the staircase-through-the-cut explanation does not hold**: from the engine's own eye the shipped chain reproduces the coverage-filtered edge exactly, and the cut and the band stay as they are. What remains on the Mac is the 0.07 texel of 4× MSAA's quarter steps, invisible at rest and marching along the edge under head motion; 8× MSAA is unavailable (`supportsTextureSampleCount(8)` is false on the M4 Max, to be confirmed on the headset's GPU), so that lever is temporal (TAA) or density. The causes the Mac does not exercise, now the candidates for the headset's 0.34–0.40 texel: the view not being the engine's eye (the lead and the pose, which the swim probe below reads), the compositor's resample and display, and RealityKit's blending over passthrough in an immersive space. +- **The adaptive lead and the swim probe, built (2026-10-19, 11:13, a side session; engine `feature/xr_presenter_adaptive_lead` `77fdab4b` on `df216eb5`, example `feature/realitykit_presenter_example_lead` `e8f9e7d` on `5f0dc4f`, both pushed, no PR, awaiting the merge and the headset).** `XRPresenterLead`: the lead is the time from a ticket's posting to the instant its frame is posed for. Fixed, as before: the ticket's age at pickup plus `leadSeconds` (22.2 ms). Adaptive (`adaptiveLead`, off by default): the age, plus an exponential average (weight 1/8 on the latest frame, drawn frames only, each sample clamped to 0.1 s so a stall is forgotten in a few frames) of the frame's time from pickup to its command buffer completing on the GPU, plus `displayLatencySeconds` for what follows the completion (RealityKit noticing the replace, its frame, the compositor's), the whole clamped to 0.1 s; until a frame has completed the fixed lead stands in; the history is kept whether or not the adaptive lead is on. The constant's default is 16 ms: the fixed 22.2 ms less the headset's Release reading (age 0.03, CPU 2.8, GPU 3.6 ms), so the two leads agree at today's numbers (a test pins this within 0.3 ms). The statistics carry the lead in force and the time to completion (average and max), shown on the numbers line. New on `UntoldEngineXR`: `devicePose(forPresentationAt:)`, the pose the frame being updated is posed by. The example's `SwimProbe`: a yellow cross locked to the room 1.5 m ahead of where the head looked when it came on and a cyan dot locked to the head the same distance ahead, placed every frame from the frame's own predicted pose, drawn by the engine (emissive, hidden through the render component's visibility flag while off); rows Adaptive lead, Display latency and Swim probe in the panel, remembered, and `--adaptive-lead`, `--fixed-lead`, `--display-latency `, `--swim-probe`. **How to read it on the headset**: turn the head at a steady pace; the cross must stay in the room, the dot in view; a dot that lags the turn and catches up when the head stops says the lead is short (raise the display latency), one that runs ahead says it is long (lower it); the lead in force reads back as age + completion + the constant; toggling the adaptive lead off compares the fixed lead at the same pose. Left: the same marker in a Compositor Services example for the plan's comparison (ProceduralGeometry pins upstream develop, which lacks the pose accessor; four lines once its engine is pinned at the branch). 45 tests in the presenter filter. From 4390ceb2f3db106b850de641508b5257e3d655ba Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:18:37 +0200 Subject: [PATCH 35/72] [Docs] XRRealityKitPresenter plan: the cut test and the adaptive lead merged into the presenter branches --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 8338653a4..c7cf7b333 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -229,3 +229,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). - **The cut and the band's weight, measured on the Mac (2026-10-19, 11:13, a side session; branch `feature/xr_presenter_opacity_cut` `e622d22d` on `df216eb5`, pushed, no PR: a test and a note in the `.usda`, no production change).** `XRPresenterEdgeStraightnessTests` renders the example scene through the presenter's chain (RealityRenderer, the left eye 35 mm above the middle cube's top, the head rolled atan(0.04) so the top face's far edge crosses one row every 25 columns, 4 picture pixels per texel, over a black and a white room to recover each pixel's opacity) and fits lines to five edge stretches; deviations in eye texels. The texture's own coverage edge: 0.074 rms (the quarter steps of 4× MSAA); read as a hard edge 0.29. **Shipped (cut 0.9, band at coverage): 0.066 rms, at the coverage floor, bias 0, no halo**; the rim-cell band equals a whole-grid band to the last digit. The cut without the band 0.22; cuts 0.5 / 0.25 / 0.1 with the band 0.09 / 0.13 / 0.22 and the object grown over the room by 0.2 / 0.44 / 0.61 texel; the band at double weight 0.12; the band at full weight 0.30 with a 0.9-texel halo. The rim's colour behind its opacity is the face's (0.98) in every blended variant. **So the staircase-through-the-cut explanation does not hold**: from the engine's own eye the shipped chain reproduces the coverage-filtered edge exactly, and the cut and the band stay as they are. What remains on the Mac is the 0.07 texel of 4× MSAA's quarter steps, invisible at rest and marching along the edge under head motion; 8× MSAA is unavailable (`supportsTextureSampleCount(8)` is false on the M4 Max, to be confirmed on the headset's GPU), so that lever is temporal (TAA) or density. The causes the Mac does not exercise, now the candidates for the headset's 0.34–0.40 texel: the view not being the engine's eye (the lead and the pose, which the swim probe below reads), the compositor's resample and display, and RealityKit's blending over passthrough in an immersive space. - **The adaptive lead and the swim probe, built (2026-10-19, 11:13, a side session; engine `feature/xr_presenter_adaptive_lead` `77fdab4b` on `df216eb5`, example `feature/realitykit_presenter_example_lead` `e8f9e7d` on `5f0dc4f`, both pushed, no PR, awaiting the merge and the headset).** `XRPresenterLead`: the lead is the time from a ticket's posting to the instant its frame is posed for. Fixed, as before: the ticket's age at pickup plus `leadSeconds` (22.2 ms). Adaptive (`adaptiveLead`, off by default): the age, plus an exponential average (weight 1/8 on the latest frame, drawn frames only, each sample clamped to 0.1 s so a stall is forgotten in a few frames) of the frame's time from pickup to its command buffer completing on the GPU, plus `displayLatencySeconds` for what follows the completion (RealityKit noticing the replace, its frame, the compositor's), the whole clamped to 0.1 s; until a frame has completed the fixed lead stands in; the history is kept whether or not the adaptive lead is on. The constant's default is 16 ms: the fixed 22.2 ms less the headset's Release reading (age 0.03, CPU 2.8, GPU 3.6 ms), so the two leads agree at today's numbers (a test pins this within 0.3 ms). The statistics carry the lead in force and the time to completion (average and max), shown on the numbers line. New on `UntoldEngineXR`: `devicePose(forPresentationAt:)`, the pose the frame being updated is posed by. The example's `SwimProbe`: a yellow cross locked to the room 1.5 m ahead of where the head looked when it came on and a cyan dot locked to the head the same distance ahead, placed every frame from the frame's own predicted pose, drawn by the engine (emissive, hidden through the render component's visibility flag while off); rows Adaptive lead, Display latency and Swim probe in the panel, remembered, and `--adaptive-lead`, `--fixed-lead`, `--display-latency `, `--swim-probe`. **How to read it on the headset**: turn the head at a steady pace; the cross must stay in the room, the dot in view; a dot that lags the turn and catches up when the head stops says the lead is short (raise the display latency), one that runs ahead says it is long (lower it); the lead in force reads back as age + completion + the constant; toggling the adaptive lead off compares the fixed lead at the same pose. Left: the same marker in a Compositor Services example for the plan's comparison (ProceduralGeometry pins upstream develop, which lacks the pose accessor; four lines once its engine is pinned at the branch). 45 tests in the presenter filter. +- **Both landed in the presenter branches (2026-10-19, 11:18).** Engine `feature/xr_realitykit_presenter` fast-forwarded to the straightness test (`e622d22d`) with the adaptive lead cherry-picked on top (`71c5115c`, from the side branch's `77fdab4b`; no file overlaps), example `feature/realitykit_presenter_example` fast-forwarded to the swim probe (`e8f9e7d`); both pushed. The Release example built from the merged engine; the install waits for the headset, which the Mac saw last at 8:45 (devicectl: unavailable). Next reading on the device: the swim probe with the fixed and the adaptive lead at one pose, and whether the rim on the cube tops moves with the lead. From 2a0417a6e3eec9a6faf6518192892694b384c656 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:21:29 +0200 Subject: [PATCH 36/72] [Docs] XRRealityKitPresenter plan: the merged Release example installed on the headset --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index c7cf7b333..caeb19d9f 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -229,4 +229,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Item 1 landed (2026-10-18, 21:00): the presenter's own equi-angular rasterization rate map**, built in its own session on `feature/xr_presenter_rate_map` and fast-forwarded into the presenter branch (tip `944436da`): `XRPresenterFoveation.swift`, `XRRealityKitPresenterConfiguration.foveation` (`.none` / `.equiAngular`, `foveatedCellTexels` 8), the kernel through `rasterization_rate_map_decoder`, the grid uniform in physical texels (a vertex every 8), `XREye.foveation` carrying one frame for both eyes; on the Mac the 1920 × 1824 budget gives a 2618 × 2429 screen, 64 samples per axis, granularity 32, grid 241 × 229, the centre 1.36 × 1.33 denser, the same pixel count. A second picture test renders the foveated grids at the map's screen size against the engine's uniform frame of that size: 39.8 dB (a wrong texel-to-ray mapping would score about 21). 26 tests pass. The example (`feature/realitykit_presenter_example`, tip `3d1b42f`) has the toggle "Equi-angular rate map" in the next-space row and `--foveation equiangular|none`; the settings line names the map in force. Lesson of the day: the headset is one device with one bundle id, so only the session driving it installs, and side sessions push branches for it to merge. **Headset verdict (2026-10-18, 22:10): "a little bit sharper" at the centre of view with the map on against off**, the same scene and pose, judged from the window's toggle with the space reopened. The map stays; its GPU cost on the device is to be read from the numbers line (expected unchanged: the same pixel count). - **The cut and the band's weight, measured on the Mac (2026-10-19, 11:13, a side session; branch `feature/xr_presenter_opacity_cut` `e622d22d` on `df216eb5`, pushed, no PR: a test and a note in the `.usda`, no production change).** `XRPresenterEdgeStraightnessTests` renders the example scene through the presenter's chain (RealityRenderer, the left eye 35 mm above the middle cube's top, the head rolled atan(0.04) so the top face's far edge crosses one row every 25 columns, 4 picture pixels per texel, over a black and a white room to recover each pixel's opacity) and fits lines to five edge stretches; deviations in eye texels. The texture's own coverage edge: 0.074 rms (the quarter steps of 4× MSAA); read as a hard edge 0.29. **Shipped (cut 0.9, band at coverage): 0.066 rms, at the coverage floor, bias 0, no halo**; the rim-cell band equals a whole-grid band to the last digit. The cut without the band 0.22; cuts 0.5 / 0.25 / 0.1 with the band 0.09 / 0.13 / 0.22 and the object grown over the room by 0.2 / 0.44 / 0.61 texel; the band at double weight 0.12; the band at full weight 0.30 with a 0.9-texel halo. The rim's colour behind its opacity is the face's (0.98) in every blended variant. **So the staircase-through-the-cut explanation does not hold**: from the engine's own eye the shipped chain reproduces the coverage-filtered edge exactly, and the cut and the band stay as they are. What remains on the Mac is the 0.07 texel of 4× MSAA's quarter steps, invisible at rest and marching along the edge under head motion; 8× MSAA is unavailable (`supportsTextureSampleCount(8)` is false on the M4 Max, to be confirmed on the headset's GPU), so that lever is temporal (TAA) or density. The causes the Mac does not exercise, now the candidates for the headset's 0.34–0.40 texel: the view not being the engine's eye (the lead and the pose, which the swim probe below reads), the compositor's resample and display, and RealityKit's blending over passthrough in an immersive space. - **The adaptive lead and the swim probe, built (2026-10-19, 11:13, a side session; engine `feature/xr_presenter_adaptive_lead` `77fdab4b` on `df216eb5`, example `feature/realitykit_presenter_example_lead` `e8f9e7d` on `5f0dc4f`, both pushed, no PR, awaiting the merge and the headset).** `XRPresenterLead`: the lead is the time from a ticket's posting to the instant its frame is posed for. Fixed, as before: the ticket's age at pickup plus `leadSeconds` (22.2 ms). Adaptive (`adaptiveLead`, off by default): the age, plus an exponential average (weight 1/8 on the latest frame, drawn frames only, each sample clamped to 0.1 s so a stall is forgotten in a few frames) of the frame's time from pickup to its command buffer completing on the GPU, plus `displayLatencySeconds` for what follows the completion (RealityKit noticing the replace, its frame, the compositor's), the whole clamped to 0.1 s; until a frame has completed the fixed lead stands in; the history is kept whether or not the adaptive lead is on. The constant's default is 16 ms: the fixed 22.2 ms less the headset's Release reading (age 0.03, CPU 2.8, GPU 3.6 ms), so the two leads agree at today's numbers (a test pins this within 0.3 ms). The statistics carry the lead in force and the time to completion (average and max), shown on the numbers line. New on `UntoldEngineXR`: `devicePose(forPresentationAt:)`, the pose the frame being updated is posed by. The example's `SwimProbe`: a yellow cross locked to the room 1.5 m ahead of where the head looked when it came on and a cyan dot locked to the head the same distance ahead, placed every frame from the frame's own predicted pose, drawn by the engine (emissive, hidden through the render component's visibility flag while off); rows Adaptive lead, Display latency and Swim probe in the panel, remembered, and `--adaptive-lead`, `--fixed-lead`, `--display-latency `, `--swim-probe`. **How to read it on the headset**: turn the head at a steady pace; the cross must stay in the room, the dot in view; a dot that lags the turn and catches up when the head stops says the lead is short (raise the display latency), one that runs ahead says it is long (lower it); the lead in force reads back as age + completion + the constant; toggling the adaptive lead off compares the fixed lead at the same pose. Left: the same marker in a Compositor Services example for the plan's comparison (ProceduralGeometry pins upstream develop, which lacks the pose accessor; four lines once its engine is pinned at the branch). 45 tests in the presenter filter. -- **Both landed in the presenter branches (2026-10-19, 11:18).** Engine `feature/xr_realitykit_presenter` fast-forwarded to the straightness test (`e622d22d`) with the adaptive lead cherry-picked on top (`71c5115c`, from the side branch's `77fdab4b`; no file overlaps), example `feature/realitykit_presenter_example` fast-forwarded to the swim probe (`e8f9e7d`); both pushed. The Release example built from the merged engine; the install waits for the headset, which the Mac saw last at 8:45 (devicectl: unavailable). Next reading on the device: the swim probe with the fixed and the adaptive lead at one pose, and whether the rim on the cube tops moves with the lead. +- **Both landed in the presenter branches (2026-10-19, 11:18).** Engine `feature/xr_realitykit_presenter` fast-forwarded to the straightness test (`e622d22d`) with the adaptive lead cherry-picked on top (`71c5115c`, from the side branch's `77fdab4b`; no file overlaps), example `feature/realitykit_presenter_example` fast-forwarded to the swim probe (`e8f9e7d`); both pushed. The Release example built from the merged engine; installed and launched on the headset at 11:20 once it woke (unavailable to devicectl since 8:45). Next reading on the device: the swim probe with the fixed and the adaptive lead at one pose, and whether the rim on the cube tops moves with the lead. From 339efa7095e769d339ab9cc6ca8a2db01575480a Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:28:12 +0200 Subject: [PATCH 37/72] [Docs] XRRealityKitPresenter plan: the swim reading on the headset (no lag under either lead; a two-step jump on quick turns) --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index caeb19d9f..2fbdd4a35 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -230,3 +230,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The cut and the band's weight, measured on the Mac (2026-10-19, 11:13, a side session; branch `feature/xr_presenter_opacity_cut` `e622d22d` on `df216eb5`, pushed, no PR: a test and a note in the `.usda`, no production change).** `XRPresenterEdgeStraightnessTests` renders the example scene through the presenter's chain (RealityRenderer, the left eye 35 mm above the middle cube's top, the head rolled atan(0.04) so the top face's far edge crosses one row every 25 columns, 4 picture pixels per texel, over a black and a white room to recover each pixel's opacity) and fits lines to five edge stretches; deviations in eye texels. The texture's own coverage edge: 0.074 rms (the quarter steps of 4× MSAA); read as a hard edge 0.29. **Shipped (cut 0.9, band at coverage): 0.066 rms, at the coverage floor, bias 0, no halo**; the rim-cell band equals a whole-grid band to the last digit. The cut without the band 0.22; cuts 0.5 / 0.25 / 0.1 with the band 0.09 / 0.13 / 0.22 and the object grown over the room by 0.2 / 0.44 / 0.61 texel; the band at double weight 0.12; the band at full weight 0.30 with a 0.9-texel halo. The rim's colour behind its opacity is the face's (0.98) in every blended variant. **So the staircase-through-the-cut explanation does not hold**: from the engine's own eye the shipped chain reproduces the coverage-filtered edge exactly, and the cut and the band stay as they are. What remains on the Mac is the 0.07 texel of 4× MSAA's quarter steps, invisible at rest and marching along the edge under head motion; 8× MSAA is unavailable (`supportsTextureSampleCount(8)` is false on the M4 Max, to be confirmed on the headset's GPU), so that lever is temporal (TAA) or density. The causes the Mac does not exercise, now the candidates for the headset's 0.34–0.40 texel: the view not being the engine's eye (the lead and the pose, which the swim probe below reads), the compositor's resample and display, and RealityKit's blending over passthrough in an immersive space. - **The adaptive lead and the swim probe, built (2026-10-19, 11:13, a side session; engine `feature/xr_presenter_adaptive_lead` `77fdab4b` on `df216eb5`, example `feature/realitykit_presenter_example_lead` `e8f9e7d` on `5f0dc4f`, both pushed, no PR, awaiting the merge and the headset).** `XRPresenterLead`: the lead is the time from a ticket's posting to the instant its frame is posed for. Fixed, as before: the ticket's age at pickup plus `leadSeconds` (22.2 ms). Adaptive (`adaptiveLead`, off by default): the age, plus an exponential average (weight 1/8 on the latest frame, drawn frames only, each sample clamped to 0.1 s so a stall is forgotten in a few frames) of the frame's time from pickup to its command buffer completing on the GPU, plus `displayLatencySeconds` for what follows the completion (RealityKit noticing the replace, its frame, the compositor's), the whole clamped to 0.1 s; until a frame has completed the fixed lead stands in; the history is kept whether or not the adaptive lead is on. The constant's default is 16 ms: the fixed 22.2 ms less the headset's Release reading (age 0.03, CPU 2.8, GPU 3.6 ms), so the two leads agree at today's numbers (a test pins this within 0.3 ms). The statistics carry the lead in force and the time to completion (average and max), shown on the numbers line. New on `UntoldEngineXR`: `devicePose(forPresentationAt:)`, the pose the frame being updated is posed by. The example's `SwimProbe`: a yellow cross locked to the room 1.5 m ahead of where the head looked when it came on and a cyan dot locked to the head the same distance ahead, placed every frame from the frame's own predicted pose, drawn by the engine (emissive, hidden through the render component's visibility flag while off); rows Adaptive lead, Display latency and Swim probe in the panel, remembered, and `--adaptive-lead`, `--fixed-lead`, `--display-latency `, `--swim-probe`. **How to read it on the headset**: turn the head at a steady pace; the cross must stay in the room, the dot in view; a dot that lags the turn and catches up when the head stops says the lead is short (raise the display latency), one that runs ahead says it is long (lower it); the lead in force reads back as age + completion + the constant; toggling the adaptive lead off compares the fixed lead at the same pose. Left: the same marker in a Compositor Services example for the plan's comparison (ProceduralGeometry pins upstream develop, which lacks the pose accessor; four lines once its engine is pinned at the branch). 45 tests in the presenter filter. - **Both landed in the presenter branches (2026-10-19, 11:18).** Engine `feature/xr_realitykit_presenter` fast-forwarded to the straightness test (`e622d22d`) with the adaptive lead cherry-picked on top (`71c5115c`, from the side branch's `77fdab4b`; no file overlaps), example `feature/realitykit_presenter_example` fast-forwarded to the swim probe (`e8f9e7d`); both pushed. The Release example built from the merged engine; installed and launched on the headset at 11:20 once it woke (unavailable to devicectl since 8:45). Next reading on the device: the swim probe with the fixed and the adaptive lead at one pose, and whether the rim on the cube tops moves with the lead. +- **The swim reading (2026-10-19, 11:28, headset, Release, the merged build).** With the probe on, the cross stays in the room and the dot follows the head with no visible lag, under the fixed lead and under the adaptive one alike; moving the display-latency constant changes nothing the eye can see. So the fixed 22 ms lead is right to the eye's resolution at steady turns, and the adaptive lead stays off by default: it has nothing to correct. What the reading did show: on a quick head turn the dot jumps by about two steps, a doubled position rather than a lag, which is the signature of an engine frame shown for two refreshes (a late replace or a dropped ticket under the faster head motion) and not of a wrong lead; the cross and the cubes, reprojected by RealityKit from the grid, would not show it. To be read from the numbers line during quick turns: frames per second against updates, dropped tickets, the replace maximum and the time-to-completion maximum. From 7abd262b542f5e15d99ad09a469f9cc102b991f3 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:36:45 +0200 Subject: [PATCH 38/72] [Docs] XRRealityKitPresenter plan: the headset at 100 Hz and the quick-turn jump as a late frame, from the numbers line --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 2fbdd4a35..662454cd2 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -231,3 +231,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The adaptive lead and the swim probe, built (2026-10-19, 11:13, a side session; engine `feature/xr_presenter_adaptive_lead` `77fdab4b` on `df216eb5`, example `feature/realitykit_presenter_example_lead` `e8f9e7d` on `5f0dc4f`, both pushed, no PR, awaiting the merge and the headset).** `XRPresenterLead`: the lead is the time from a ticket's posting to the instant its frame is posed for. Fixed, as before: the ticket's age at pickup plus `leadSeconds` (22.2 ms). Adaptive (`adaptiveLead`, off by default): the age, plus an exponential average (weight 1/8 on the latest frame, drawn frames only, each sample clamped to 0.1 s so a stall is forgotten in a few frames) of the frame's time from pickup to its command buffer completing on the GPU, plus `displayLatencySeconds` for what follows the completion (RealityKit noticing the replace, its frame, the compositor's), the whole clamped to 0.1 s; until a frame has completed the fixed lead stands in; the history is kept whether or not the adaptive lead is on. The constant's default is 16 ms: the fixed 22.2 ms less the headset's Release reading (age 0.03, CPU 2.8, GPU 3.6 ms), so the two leads agree at today's numbers (a test pins this within 0.3 ms). The statistics carry the lead in force and the time to completion (average and max), shown on the numbers line. New on `UntoldEngineXR`: `devicePose(forPresentationAt:)`, the pose the frame being updated is posed by. The example's `SwimProbe`: a yellow cross locked to the room 1.5 m ahead of where the head looked when it came on and a cyan dot locked to the head the same distance ahead, placed every frame from the frame's own predicted pose, drawn by the engine (emissive, hidden through the render component's visibility flag while off); rows Adaptive lead, Display latency and Swim probe in the panel, remembered, and `--adaptive-lead`, `--fixed-lead`, `--display-latency `, `--swim-probe`. **How to read it on the headset**: turn the head at a steady pace; the cross must stay in the room, the dot in view; a dot that lags the turn and catches up when the head stops says the lead is short (raise the display latency), one that runs ahead says it is long (lower it); the lead in force reads back as age + completion + the constant; toggling the adaptive lead off compares the fixed lead at the same pose. Left: the same marker in a Compositor Services example for the plan's comparison (ProceduralGeometry pins upstream develop, which lacks the pose accessor; four lines once its engine is pinned at the branch). 45 tests in the presenter filter. - **Both landed in the presenter branches (2026-10-19, 11:18).** Engine `feature/xr_realitykit_presenter` fast-forwarded to the straightness test (`e622d22d`) with the adaptive lead cherry-picked on top (`71c5115c`, from the side branch's `77fdab4b`; no file overlaps), example `feature/realitykit_presenter_example` fast-forwarded to the swim probe (`e8f9e7d`); both pushed. The Release example built from the merged engine; installed and launched on the headset at 11:20 once it woke (unavailable to devicectl since 8:45). Next reading on the device: the swim probe with the fixed and the adaptive lead at one pose, and whether the rim on the cube tops moves with the lead. - **The swim reading (2026-10-19, 11:28, headset, Release, the merged build).** With the probe on, the cross stays in the room and the dot follows the head with no visible lag, under the fixed lead and under the adaptive one alike; moving the display-latency constant changes nothing the eye can see. So the fixed 22 ms lead is right to the eye's resolution at steady turns, and the adaptive lead stays off by default: it has nothing to correct. What the reading did show: on a quick head turn the dot jumps by about two steps, a doubled position rather than a lag, which is the signature of an engine frame shown for two refreshes (a late replace or a dropped ticket under the faster head motion) and not of a wrong lead; the cross and the cubes, reprojected by RealityKit from the grid, would not show it. To be read from the numbers line during quick turns: frames per second against updates, dropped tickets, the replace maximum and the time-to-completion maximum. +- **The jump explained by the numbers line during quick turns (2026-10-19, 11:36): 100 updates/s, 100 frames/s, 0 dropped, replace 0.30 ms, to completion 6.6 ms average and 16.6 ms maximum.** Two findings. (1) **The headset ran at 100 Hz**, not 90: visionOS raises the refresh to 100 Hz under 50 Hz mains lighting (flicker avoidance; 96 Hz for 24 fps video), so the refresh is 10 ms here and the frame's budget 10 ms, not 11.1; the fixed lead of 22.2 ms is then 2.2 refreshes, which the reading found right, and every "of 11.1 ms" in this section is "of 10 ms" when the room is lit at 50 Hz; the presenter's constants should come from the measured update rate rather than assume 90 Hz. (2) **The two-step jump is a frame completing late**: every ticket is taken and drawn (frames equal updates, nothing dropped, the replace costs nothing), but a frame whose pickup-to-completion runs to 16.6 ms against the 6.6 ms average misses the refresh it was posed for and lands one refresh late, so RealityKit shows the previous texture once more, the head moves on, and the head-locked dot, baked into the texture at its frame's pose, catches up two steps at once when the next frame lands on time; the cross and the cubes are untouched because RealityKit keeps reprojecting the grid at the actual head pose. A late frame would do the same to head-locked content on the Compositor path, which reprojects the frame as a whole. Not a lead error: the adaptive lead averages and cannot absorb a per-frame spike. What to read next: the GPU span maximum during the turns (if it reaches the completion maximum the shared GPU is the spike, the compositor's and ARKit's work growing with head motion; if it stays near 5 ms the spike is the CPU frame or the queue wait before the GPU starts, which the statistics do not yet separate: the CPU frame has no maximum on the line and the commit-to-GPU-start wait is not recorded). Mitigations, once the stage is known: the engine's frame starting earlier in the refresh, a second set of textures and meshes so a late frame never holds the one on show, less work in the frame on head motion (culling and shadow updates), or a head-locked layer drawn by RealityKit itself for HUD content. World-locked content is unaffected either way. From 5c151afd41f14282c0fe7aaa0e99955fba632d66 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:42:50 +0200 Subject: [PATCH 39/72] [Docs] XRRealityKitPresenter plan: the quick-turn spike is the shared GPU (span max 13.9 ms) --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 662454cd2..179d4fd9b 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -232,3 +232,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Both landed in the presenter branches (2026-10-19, 11:18).** Engine `feature/xr_realitykit_presenter` fast-forwarded to the straightness test (`e622d22d`) with the adaptive lead cherry-picked on top (`71c5115c`, from the side branch's `77fdab4b`; no file overlaps), example `feature/realitykit_presenter_example` fast-forwarded to the swim probe (`e8f9e7d`); both pushed. The Release example built from the merged engine; installed and launched on the headset at 11:20 once it woke (unavailable to devicectl since 8:45). Next reading on the device: the swim probe with the fixed and the adaptive lead at one pose, and whether the rim on the cube tops moves with the lead. - **The swim reading (2026-10-19, 11:28, headset, Release, the merged build).** With the probe on, the cross stays in the room and the dot follows the head with no visible lag, under the fixed lead and under the adaptive one alike; moving the display-latency constant changes nothing the eye can see. So the fixed 22 ms lead is right to the eye's resolution at steady turns, and the adaptive lead stays off by default: it has nothing to correct. What the reading did show: on a quick head turn the dot jumps by about two steps, a doubled position rather than a lag, which is the signature of an engine frame shown for two refreshes (a late replace or a dropped ticket under the faster head motion) and not of a wrong lead; the cross and the cubes, reprojected by RealityKit from the grid, would not show it. To be read from the numbers line during quick turns: frames per second against updates, dropped tickets, the replace maximum and the time-to-completion maximum. - **The jump explained by the numbers line during quick turns (2026-10-19, 11:36): 100 updates/s, 100 frames/s, 0 dropped, replace 0.30 ms, to completion 6.6 ms average and 16.6 ms maximum.** Two findings. (1) **The headset ran at 100 Hz**, not 90: visionOS raises the refresh to 100 Hz under 50 Hz mains lighting (flicker avoidance; 96 Hz for 24 fps video), so the refresh is 10 ms here and the frame's budget 10 ms, not 11.1; the fixed lead of 22.2 ms is then 2.2 refreshes, which the reading found right, and every "of 11.1 ms" in this section is "of 10 ms" when the room is lit at 50 Hz; the presenter's constants should come from the measured update rate rather than assume 90 Hz. (2) **The two-step jump is a frame completing late**: every ticket is taken and drawn (frames equal updates, nothing dropped, the replace costs nothing), but a frame whose pickup-to-completion runs to 16.6 ms against the 6.6 ms average misses the refresh it was posed for and lands one refresh late, so RealityKit shows the previous texture once more, the head moves on, and the head-locked dot, baked into the texture at its frame's pose, catches up two steps at once when the next frame lands on time; the cross and the cubes are untouched because RealityKit keeps reprojecting the grid at the actual head pose. A late frame would do the same to head-locked content on the Compositor path, which reprojects the frame as a whole. Not a lead error: the adaptive lead averages and cannot absorb a per-frame spike. What to read next: the GPU span maximum during the turns (if it reaches the completion maximum the shared GPU is the spike, the compositor's and ARKit's work growing with head motion; if it stays near 5 ms the spike is the CPU frame or the queue wait before the GPU starts, which the statistics do not yet separate: the CPU frame has no maximum on the line and the commit-to-GPU-start wait is not recorded). Mitigations, once the stage is known: the engine's frame starting earlier in the refresh, a second set of textures and meshes so a late frame never holds the one on show, less work in the frame on head motion (culling and shadow updates), or a head-locked layer drawn by RealityKit itself for HUD content. World-locked content is unaffected either way. +- **The spike is the GPU (2026-10-19, 11:42): GPU span 3.9 ms average, 13.9 ms maximum during the same quick turns**, against the completion's 6.6 / 16.6. The span is from the ticket's command buffer starting on the GPU to its ending, and on the headset it holds whatever else the shared GPU ran in between: RealityKit's own render of the same frame in this process and the compositor's work, which grow with head motion. The engine's own work is 3.6–4.6 ms whatever the head does (the engine-GPU figure), so a 13.9 ms span is 10 ms of waiting inside the span, not more drawing; with the 2.8 ms CPU frame it accounts for the 16.6 ms completion, leaving nothing for a queue wait before the GPU starts. Hence the late frames: a 10 ms refresh at 100 Hz, 2.8 ms of CPU, 4 ms of GPU, and bursts of 10 ms of other work on the GPU in front of ours. The Compositor path gives the immersive app's drawing the compositor's priority and deadline; here our command buffer is ordinary work beside RealityKit's. The split instrumentation (its own session, in progress) will count the late frames per second, which decides whether this is a quality item at all for world-locked games (the grid reprojects them regardless) or only for head-locked content, which a game can give to RealityKit to draw. From 233225a3837f38a2b5ca721a34e05f2ea62a121a Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 11:58:53 +0200 Subject: [PATCH 40/72] [Docs] XRRealityKitPresenter plan: the frames' stages and the late count on the numbers line, merged and installed --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 179d4fd9b..a030f4b9f 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -233,3 +233,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The swim reading (2026-10-19, 11:28, headset, Release, the merged build).** With the probe on, the cross stays in the room and the dot follows the head with no visible lag, under the fixed lead and under the adaptive one alike; moving the display-latency constant changes nothing the eye can see. So the fixed 22 ms lead is right to the eye's resolution at steady turns, and the adaptive lead stays off by default: it has nothing to correct. What the reading did show: on a quick head turn the dot jumps by about two steps, a doubled position rather than a lag, which is the signature of an engine frame shown for two refreshes (a late replace or a dropped ticket under the faster head motion) and not of a wrong lead; the cross and the cubes, reprojected by RealityKit from the grid, would not show it. To be read from the numbers line during quick turns: frames per second against updates, dropped tickets, the replace maximum and the time-to-completion maximum. - **The jump explained by the numbers line during quick turns (2026-10-19, 11:36): 100 updates/s, 100 frames/s, 0 dropped, replace 0.30 ms, to completion 6.6 ms average and 16.6 ms maximum.** Two findings. (1) **The headset ran at 100 Hz**, not 90: visionOS raises the refresh to 100 Hz under 50 Hz mains lighting (flicker avoidance; 96 Hz for 24 fps video), so the refresh is 10 ms here and the frame's budget 10 ms, not 11.1; the fixed lead of 22.2 ms is then 2.2 refreshes, which the reading found right, and every "of 11.1 ms" in this section is "of 10 ms" when the room is lit at 50 Hz; the presenter's constants should come from the measured update rate rather than assume 90 Hz. (2) **The two-step jump is a frame completing late**: every ticket is taken and drawn (frames equal updates, nothing dropped, the replace costs nothing), but a frame whose pickup-to-completion runs to 16.6 ms against the 6.6 ms average misses the refresh it was posed for and lands one refresh late, so RealityKit shows the previous texture once more, the head moves on, and the head-locked dot, baked into the texture at its frame's pose, catches up two steps at once when the next frame lands on time; the cross and the cubes are untouched because RealityKit keeps reprojecting the grid at the actual head pose. A late frame would do the same to head-locked content on the Compositor path, which reprojects the frame as a whole. Not a lead error: the adaptive lead averages and cannot absorb a per-frame spike. What to read next: the GPU span maximum during the turns (if it reaches the completion maximum the shared GPU is the spike, the compositor's and ARKit's work growing with head motion; if it stays near 5 ms the spike is the CPU frame or the queue wait before the GPU starts, which the statistics do not yet separate: the CPU frame has no maximum on the line and the commit-to-GPU-start wait is not recorded). Mitigations, once the stage is known: the engine's frame starting earlier in the refresh, a second set of textures and meshes so a late frame never holds the one on show, less work in the frame on head motion (culling and shadow updates), or a head-locked layer drawn by RealityKit itself for HUD content. World-locked content is unaffected either way. - **The spike is the GPU (2026-10-19, 11:42): GPU span 3.9 ms average, 13.9 ms maximum during the same quick turns**, against the completion's 6.6 / 16.6. The span is from the ticket's command buffer starting on the GPU to its ending, and on the headset it holds whatever else the shared GPU ran in between: RealityKit's own render of the same frame in this process and the compositor's work, which grow with head motion. The engine's own work is 3.6–4.6 ms whatever the head does (the engine-GPU figure), so a 13.9 ms span is 10 ms of waiting inside the span, not more drawing; with the 2.8 ms CPU frame it accounts for the 16.6 ms completion, leaving nothing for a queue wait before the GPU starts. Hence the late frames: a 10 ms refresh at 100 Hz, 2.8 ms of CPU, 4 ms of GPU, and bursts of 10 ms of other work on the GPU in front of ours. The Compositor path gives the immersive app's drawing the compositor's priority and deadline; here our command buffer is ordinary work beside RealityKit's. The split instrumentation (its own session, in progress) will count the late frames per second, which decides whether this is a quality item at all for world-locked games (the grid reprojects them regardless) or only for head-locked content, which a game can give to RealityKit to draw. +- **The frames' stages on the numbers line (2026-10-19, 11:58; engine `feature/xr_realitykit_presenter` fast-forwarded to `5c1c6cdf`, example `feature/realitykit_presenter_example` to `ed50e54`, both pushed; Release built and installed on the headset at 11:56).** Built in its own session. `XRFrame` gains `willCommit(_:)` (a no-op by default; the loop calls it right before its single commit, the Compositor path untouched), where the RealityKit frame stamps the commit instant and adds its completed handler; `XRPresenterFrameStages` splits a frame into the CPU frame (pickup to commit), the wait (commit to the GPU's start: the driver's submission and the queue, where the compositor's and RealityKit's own work go first) and the GPU span, summing to the completion. The statistics keep each stage's average and maximum, the refresh period measured from the update rate (100 updates/s reads 10.0 ms, 90 reads 11.1; nothing assumes 90 Hz, except the fixed lead's default of 2/90 s and the lead's comments, left as they are), the count of frames whose completion exceeded that refresh, and the worst frame's split. The example's line: "100 updates/s (refresh 10.0 ms) · 100 frames/s · 0 dropped · N late · … · cpu (max) · wait (max) · gpu (max) · completion (max) · worst 16.6 = cpu 3.1 + wait 9.2 + gpu 4.3 · engine GPU · lead (max) · thermal"; the replace maximum left the line. 58 tests (the presenter filter plus the loop's, which now expect `willCommit` between present and endDrawing). To read on the headset during quick turns: the late count per second (how often a frame lands a refresh late) and the worst frame's split (a big wait = the GPU was busy with others' work when the buffer was committed; a big gpu = the span itself; a big cpu = the engine's frame). From a3c61ff09e61bff4c5ff2b1da6576346b4c2d874 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 12:02:28 +0200 Subject: [PATCH 41/72] [Docs] XRRealityKitPresenter plan: the worst frame's split on the headset (gpu 20 of 24.8, wait 0.1) --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index a030f4b9f..5f90ce7f9 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -234,3 +234,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The jump explained by the numbers line during quick turns (2026-10-19, 11:36): 100 updates/s, 100 frames/s, 0 dropped, replace 0.30 ms, to completion 6.6 ms average and 16.6 ms maximum.** Two findings. (1) **The headset ran at 100 Hz**, not 90: visionOS raises the refresh to 100 Hz under 50 Hz mains lighting (flicker avoidance; 96 Hz for 24 fps video), so the refresh is 10 ms here and the frame's budget 10 ms, not 11.1; the fixed lead of 22.2 ms is then 2.2 refreshes, which the reading found right, and every "of 11.1 ms" in this section is "of 10 ms" when the room is lit at 50 Hz; the presenter's constants should come from the measured update rate rather than assume 90 Hz. (2) **The two-step jump is a frame completing late**: every ticket is taken and drawn (frames equal updates, nothing dropped, the replace costs nothing), but a frame whose pickup-to-completion runs to 16.6 ms against the 6.6 ms average misses the refresh it was posed for and lands one refresh late, so RealityKit shows the previous texture once more, the head moves on, and the head-locked dot, baked into the texture at its frame's pose, catches up two steps at once when the next frame lands on time; the cross and the cubes are untouched because RealityKit keeps reprojecting the grid at the actual head pose. A late frame would do the same to head-locked content on the Compositor path, which reprojects the frame as a whole. Not a lead error: the adaptive lead averages and cannot absorb a per-frame spike. What to read next: the GPU span maximum during the turns (if it reaches the completion maximum the shared GPU is the spike, the compositor's and ARKit's work growing with head motion; if it stays near 5 ms the spike is the CPU frame or the queue wait before the GPU starts, which the statistics do not yet separate: the CPU frame has no maximum on the line and the commit-to-GPU-start wait is not recorded). Mitigations, once the stage is known: the engine's frame starting earlier in the refresh, a second set of textures and meshes so a late frame never holds the one on show, less work in the frame on head motion (culling and shadow updates), or a head-locked layer drawn by RealityKit itself for HUD content. World-locked content is unaffected either way. - **The spike is the GPU (2026-10-19, 11:42): GPU span 3.9 ms average, 13.9 ms maximum during the same quick turns**, against the completion's 6.6 / 16.6. The span is from the ticket's command buffer starting on the GPU to its ending, and on the headset it holds whatever else the shared GPU ran in between: RealityKit's own render of the same frame in this process and the compositor's work, which grow with head motion. The engine's own work is 3.6–4.6 ms whatever the head does (the engine-GPU figure), so a 13.9 ms span is 10 ms of waiting inside the span, not more drawing; with the 2.8 ms CPU frame it accounts for the 16.6 ms completion, leaving nothing for a queue wait before the GPU starts. Hence the late frames: a 10 ms refresh at 100 Hz, 2.8 ms of CPU, 4 ms of GPU, and bursts of 10 ms of other work on the GPU in front of ours. The Compositor path gives the immersive app's drawing the compositor's priority and deadline; here our command buffer is ordinary work beside RealityKit's. The split instrumentation (its own session, in progress) will count the late frames per second, which decides whether this is a quality item at all for world-locked games (the grid reprojects them regardless) or only for head-locked content, which a game can give to RealityKit to draw. - **The frames' stages on the numbers line (2026-10-19, 11:58; engine `feature/xr_realitykit_presenter` fast-forwarded to `5c1c6cdf`, example `feature/realitykit_presenter_example` to `ed50e54`, both pushed; Release built and installed on the headset at 11:56).** Built in its own session. `XRFrame` gains `willCommit(_:)` (a no-op by default; the loop calls it right before its single commit, the Compositor path untouched), where the RealityKit frame stamps the commit instant and adds its completed handler; `XRPresenterFrameStages` splits a frame into the CPU frame (pickup to commit), the wait (commit to the GPU's start: the driver's submission and the queue, where the compositor's and RealityKit's own work go first) and the GPU span, summing to the completion. The statistics keep each stage's average and maximum, the refresh period measured from the update rate (100 updates/s reads 10.0 ms, 90 reads 11.1; nothing assumes 90 Hz, except the fixed lead's default of 2/90 s and the lead's comments, left as they are), the count of frames whose completion exceeded that refresh, and the worst frame's split. The example's line: "100 updates/s (refresh 10.0 ms) · 100 frames/s · 0 dropped · N late · … · cpu (max) · wait (max) · gpu (max) · completion (max) · worst 16.6 = cpu 3.1 + wait 9.2 + gpu 4.3 · engine GPU · lead (max) · thermal"; the replace maximum left the line. 58 tests (the presenter filter plus the loop's, which now expect `willCommit` between present and endDrawing). To read on the headset during quick turns: the late count per second (how often a frame lands a refresh late) and the worst frame's split (a big wait = the GPU was busy with others' work when the buffer was committed; a big gpu = the span itself; a big cpu = the engine's frame). +- **The worst frame split on the headset (2026-10-19, 12:02, Release, quick head turns): worst 24.8 = cpu 4 + wait 0.1 + gpu 20, the wait always 0.1 ms.** So the queue is not it: the GPU takes the ticket's command buffer at once and then needs 20 ms to get through the engine's 4 ms of work, which means the buffer runs time-sliced with other GPU work for the whole of two refreshes. The engine's own passes are the same whatever the head does (a slab, cubes, a sphere), so the other 16 ms belong to the GPU's other clients during a quick turn: the compositor's and RealityKit's per-refresh work twice over, and whatever grows with head motion, ARKit's tracking and world sensing being the candidate (new areas of the room entering the view). On this path the engine's command buffer is ordinary work from the app's process beside RealityKit's own render; the Compositor path runs the app's render at the compositor's priority. Which of the two explanations holds, the platform's load under head motion or this path's priority, is decided by the same reading on the Compositor path: the engine's GPU frame in a Compositor Services example during the same quick turns, which needs that example to print the engine's frame numbers. Until then the late frames stay a head-locked-content item only: world-locked content is reprojected by RealityKit from the grid and does not show them. From dfc4e6d2bb3265a1606dad5508b59464eecbca37 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 12:08:41 +0200 Subject: [PATCH 42/72] [Docs] XRRealityKitPresenter plan: 1 to 5 late frames per second on quick turns, head-locked content only --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 5f90ce7f9..b0954e28d 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -235,3 +235,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The spike is the GPU (2026-10-19, 11:42): GPU span 3.9 ms average, 13.9 ms maximum during the same quick turns**, against the completion's 6.6 / 16.6. The span is from the ticket's command buffer starting on the GPU to its ending, and on the headset it holds whatever else the shared GPU ran in between: RealityKit's own render of the same frame in this process and the compositor's work, which grow with head motion. The engine's own work is 3.6–4.6 ms whatever the head does (the engine-GPU figure), so a 13.9 ms span is 10 ms of waiting inside the span, not more drawing; with the 2.8 ms CPU frame it accounts for the 16.6 ms completion, leaving nothing for a queue wait before the GPU starts. Hence the late frames: a 10 ms refresh at 100 Hz, 2.8 ms of CPU, 4 ms of GPU, and bursts of 10 ms of other work on the GPU in front of ours. The Compositor path gives the immersive app's drawing the compositor's priority and deadline; here our command buffer is ordinary work beside RealityKit's. The split instrumentation (its own session, in progress) will count the late frames per second, which decides whether this is a quality item at all for world-locked games (the grid reprojects them regardless) or only for head-locked content, which a game can give to RealityKit to draw. - **The frames' stages on the numbers line (2026-10-19, 11:58; engine `feature/xr_realitykit_presenter` fast-forwarded to `5c1c6cdf`, example `feature/realitykit_presenter_example` to `ed50e54`, both pushed; Release built and installed on the headset at 11:56).** Built in its own session. `XRFrame` gains `willCommit(_:)` (a no-op by default; the loop calls it right before its single commit, the Compositor path untouched), where the RealityKit frame stamps the commit instant and adds its completed handler; `XRPresenterFrameStages` splits a frame into the CPU frame (pickup to commit), the wait (commit to the GPU's start: the driver's submission and the queue, where the compositor's and RealityKit's own work go first) and the GPU span, summing to the completion. The statistics keep each stage's average and maximum, the refresh period measured from the update rate (100 updates/s reads 10.0 ms, 90 reads 11.1; nothing assumes 90 Hz, except the fixed lead's default of 2/90 s and the lead's comments, left as they are), the count of frames whose completion exceeded that refresh, and the worst frame's split. The example's line: "100 updates/s (refresh 10.0 ms) · 100 frames/s · 0 dropped · N late · … · cpu (max) · wait (max) · gpu (max) · completion (max) · worst 16.6 = cpu 3.1 + wait 9.2 + gpu 4.3 · engine GPU · lead (max) · thermal"; the replace maximum left the line. 58 tests (the presenter filter plus the loop's, which now expect `willCommit` between present and endDrawing). To read on the headset during quick turns: the late count per second (how often a frame lands a refresh late) and the worst frame's split (a big wait = the GPU was busy with others' work when the buffer was committed; a big gpu = the span itself; a big cpu = the engine's frame). - **The worst frame split on the headset (2026-10-19, 12:02, Release, quick head turns): worst 24.8 = cpu 4 + wait 0.1 + gpu 20, the wait always 0.1 ms.** So the queue is not it: the GPU takes the ticket's command buffer at once and then needs 20 ms to get through the engine's 4 ms of work, which means the buffer runs time-sliced with other GPU work for the whole of two refreshes. The engine's own passes are the same whatever the head does (a slab, cubes, a sphere), so the other 16 ms belong to the GPU's other clients during a quick turn: the compositor's and RealityKit's per-refresh work twice over, and whatever grows with head motion, ARKit's tracking and world sensing being the candidate (new areas of the room entering the view). On this path the engine's command buffer is ordinary work from the app's process beside RealityKit's own render; the Compositor path runs the app's render at the compositor's priority. Which of the two explanations holds, the platform's load under head motion or this path's priority, is decided by the same reading on the Compositor path: the engine's GPU frame in a Compositor Services example during the same quick turns, which needs that example to print the engine's frame numbers. Until then the late frames stay a head-locked-content item only: world-locked content is reprojected by RealityKit from the grid and does not show them. +- **How often (2026-10-19, 12:08): 1 to 2 late frames per second during quick head turns, up to 5 with the fastest, at 100 frames per second.** So 1 to 5 % of the frames land a refresh late while the head turns fast, each a one-refresh repeat of the previous texture that only head-locked content shows (the probe's dot); world-locked content, reprojected by RealityKit from the grid, shows nothing. The size of the item for a game: a HUD drawn by the engine stutters now and then on fast turns, a HUD drawn by RealityKit does not; the scene itself is unaffected. Whether the Compositor path repeats frames the same way under the same turns is the comparison in progress (its own session: the engine's stats monitor and ProceduralGeometry's stats window gain the GPU span maximum, the late count and the worst frame's split with the presenter's definitions). From f05dd75668dc20e14f9d271b583e4c122ab8d8b2 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 12:37:15 +0200 Subject: [PATCH 43/72] [Docs] XRRealityKitPresenter plan: the Compositor path instrumented for the late-frame comparison, built and installed --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index b0954e28d..63402e337 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -236,3 +236,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The frames' stages on the numbers line (2026-10-19, 11:58; engine `feature/xr_realitykit_presenter` fast-forwarded to `5c1c6cdf`, example `feature/realitykit_presenter_example` to `ed50e54`, both pushed; Release built and installed on the headset at 11:56).** Built in its own session. `XRFrame` gains `willCommit(_:)` (a no-op by default; the loop calls it right before its single commit, the Compositor path untouched), where the RealityKit frame stamps the commit instant and adds its completed handler; `XRPresenterFrameStages` splits a frame into the CPU frame (pickup to commit), the wait (commit to the GPU's start: the driver's submission and the queue, where the compositor's and RealityKit's own work go first) and the GPU span, summing to the completion. The statistics keep each stage's average and maximum, the refresh period measured from the update rate (100 updates/s reads 10.0 ms, 90 reads 11.1; nothing assumes 90 Hz, except the fixed lead's default of 2/90 s and the lead's comments, left as they are), the count of frames whose completion exceeded that refresh, and the worst frame's split. The example's line: "100 updates/s (refresh 10.0 ms) · 100 frames/s · 0 dropped · N late · … · cpu (max) · wait (max) · gpu (max) · completion (max) · worst 16.6 = cpu 3.1 + wait 9.2 + gpu 4.3 · engine GPU · lead (max) · thermal"; the replace maximum left the line. 58 tests (the presenter filter plus the loop's, which now expect `willCommit` between present and endDrawing). To read on the headset during quick turns: the late count per second (how often a frame lands a refresh late) and the worst frame's split (a big wait = the GPU was busy with others' work when the buffer was committed; a big gpu = the span itself; a big cpu = the engine's frame). - **The worst frame split on the headset (2026-10-19, 12:02, Release, quick head turns): worst 24.8 = cpu 4 + wait 0.1 + gpu 20, the wait always 0.1 ms.** So the queue is not it: the GPU takes the ticket's command buffer at once and then needs 20 ms to get through the engine's 4 ms of work, which means the buffer runs time-sliced with other GPU work for the whole of two refreshes. The engine's own passes are the same whatever the head does (a slab, cubes, a sphere), so the other 16 ms belong to the GPU's other clients during a quick turn: the compositor's and RealityKit's per-refresh work twice over, and whatever grows with head motion, ARKit's tracking and world sensing being the candidate (new areas of the room entering the view). On this path the engine's command buffer is ordinary work from the app's process beside RealityKit's own render; the Compositor path runs the app's render at the compositor's priority. Which of the two explanations holds, the platform's load under head motion or this path's priority, is decided by the same reading on the Compositor path: the engine's GPU frame in a Compositor Services example during the same quick turns, which needs that example to print the engine's frame numbers. Until then the late frames stay a head-locked-content item only: world-locked content is reprojected by RealityKit from the grid and does not show them. - **How often (2026-10-19, 12:08): 1 to 2 late frames per second during quick head turns, up to 5 with the fastest, at 100 frames per second.** So 1 to 5 % of the frames land a refresh late while the head turns fast, each a one-refresh repeat of the previous texture that only head-locked content shows (the probe's dot); world-locked content, reprojected by RealityKit from the grid, shows nothing. The size of the item for a game: a HUD drawn by the engine stutters now and then on fast turns, a HUD drawn by RealityKit does not; the scene itself is unaffected. Whether the Compositor path repeats frames the same way under the same turns is the comparison in progress (its own session: the engine's stats monitor and ProceduralGeometry's stats window gain the GPU span maximum, the late count and the worst frame's split with the presenter's definitions). +- **The Compositor path instrumented for the comparison (2026-10-19, 12:37; engine `feature/xr_realitykit_presenter` fast-forwarded to `2eb7414f`, pushed; arcade branch `feature/procedural_geometry_frame_stages` `d91bb7f` on main, not merged; ProceduralGeometry built in Release with `SWIFT_ACTIVE_COMPILATION_CONDITIONS='$(inherited) ENGINE_STATS_ENABLED'` against the merged engine, signed as com.miolabs.proceduralgeometry, installed beside the presenter example at 12:33).** Built in its own session. The engine's stats monitor splits the XR loop's frames with the presenter's definitions (`EngineFrameStages`, `EngineFrameStageStats`, `formatEngineFrameStages`, all under `ENGINE_STATS_ENABLED`; the loop stamps the commit before `commit()` and adds a second completed handler on the display's buffer): cpu = the loop's frame begin to the commit less the compositor's pacing sleep for the optimal input time (the RealityKit path sleeps nothing, so "late" means the same on both), wait = commit to the GPU's start, gpu = the raw span, late = completion over the refresh measured from the frames begun, plus a count of frames past the compositor's rendering deadline. ProceduralGeometry's Engine Stats window shows the line "gpu span max X ms · late N of F (refresh R ms) · worst A = cpu B + wait C + gpu D" above the compact text and takes `--open-space --stats-window --log-stats`; its committed project did not build from a clean checkout (the extension's pin moved to 187eaaf; the committed Release app target is at -Onone, set to -O in the local spec for the build). 79 tests. The reading: the same quick turns in both apps; a span stretched to ~20 ms with a small wait on the Compositor path too says the platform's GPU load under head motion; a span at its rest value with late 0 says the RealityKit path's buffer runs at app priority behind RealityKit's own render. From b46e5ccab3fc5a8ad56657e47ad7bde1dc6ce22a Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 12:47:37 +0200 Subject: [PATCH 44/72] [Docs] XRRealityKitPresenter plan: the Compositor path's launch crash on the headset (main-actor hook) found and fixed --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 63402e337..1cbb7be29 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -237,3 +237,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The worst frame split on the headset (2026-10-19, 12:02, Release, quick head turns): worst 24.8 = cpu 4 + wait 0.1 + gpu 20, the wait always 0.1 ms.** So the queue is not it: the GPU takes the ticket's command buffer at once and then needs 20 ms to get through the engine's 4 ms of work, which means the buffer runs time-sliced with other GPU work for the whole of two refreshes. The engine's own passes are the same whatever the head does (a slab, cubes, a sphere), so the other 16 ms belong to the GPU's other clients during a quick turn: the compositor's and RealityKit's per-refresh work twice over, and whatever grows with head motion, ARKit's tracking and world sensing being the candidate (new areas of the room entering the view). On this path the engine's command buffer is ordinary work from the app's process beside RealityKit's own render; the Compositor path runs the app's render at the compositor's priority. Which of the two explanations holds, the platform's load under head motion or this path's priority, is decided by the same reading on the Compositor path: the engine's GPU frame in a Compositor Services example during the same quick turns, which needs that example to print the engine's frame numbers. Until then the late frames stay a head-locked-content item only: world-locked content is reprojected by RealityKit from the grid and does not show them. - **How often (2026-10-19, 12:08): 1 to 2 late frames per second during quick head turns, up to 5 with the fastest, at 100 frames per second.** So 1 to 5 % of the frames land a refresh late while the head turns fast, each a one-refresh repeat of the previous texture that only head-locked content shows (the probe's dot); world-locked content, reprojected by RealityKit from the grid, shows nothing. The size of the item for a game: a HUD drawn by the engine stutters now and then on fast turns, a HUD drawn by RealityKit does not; the scene itself is unaffected. Whether the Compositor path repeats frames the same way under the same turns is the comparison in progress (its own session: the engine's stats monitor and ProceduralGeometry's stats window gain the GPU span maximum, the late count and the worst frame's split with the presenter's definitions). - **The Compositor path instrumented for the comparison (2026-10-19, 12:37; engine `feature/xr_realitykit_presenter` fast-forwarded to `2eb7414f`, pushed; arcade branch `feature/procedural_geometry_frame_stages` `d91bb7f` on main, not merged; ProceduralGeometry built in Release with `SWIFT_ACTIVE_COMPILATION_CONDITIONS='$(inherited) ENGINE_STATS_ENABLED'` against the merged engine, signed as com.miolabs.proceduralgeometry, installed beside the presenter example at 12:33).** Built in its own session. The engine's stats monitor splits the XR loop's frames with the presenter's definitions (`EngineFrameStages`, `EngineFrameStageStats`, `formatEngineFrameStages`, all under `ENGINE_STATS_ENABLED`; the loop stamps the commit before `commit()` and adds a second completed handler on the display's buffer): cpu = the loop's frame begin to the commit less the compositor's pacing sleep for the optimal input time (the RealityKit path sleeps nothing, so "late" means the same on both), wait = commit to the GPU's start, gpu = the raw span, late = completion over the refresh measured from the frames begun, plus a count of frames past the compositor's rendering deadline. ProceduralGeometry's Engine Stats window shows the line "gpu span max X ms · late N of F (refresh R ms) · worst A = cpu B + wait C + gpu D" above the compact text and takes `--open-space --stats-window --log-stats`; its committed project did not build from a clean checkout (the extension's pin moved to 187eaaf; the committed Release app target is at -Onone, set to -O in the local spec for the build). 79 tests. The reading: the same quick turns in both apps; a span stretched to ~20 ms with a small wait on the Compositor path too says the platform's GPU load under head motion; a span at its rest value with late 0 says the RealityKit path's buffer runs at app priority behind RealityKit's own render. +- **The Compositor path crashed at launch on the headset, found and fixed (2026-10-19, 12:47; engine `0b946f03` on the presenter branch, pushed).** The first ProceduralGeometry build on the seam died within a second of launching, three crash logs in a minute (`EXC_BREAKPOINT` in `dispatch_assert_queue` from `_swift_task_checkIsolatedSwift`, faulting in `closure #1 in CompositorFrameSource.init` called by `XRRuntime.printXRAnchorDiagnostics` from the session event monitor's task). The seam (stage 1) had given the runtime a hook, `presenterStateDescription: () -> String`, which the Compositor source fills in from its `@MainActor` init with the layer's state; a non-Sendable closure formed in a main-actor context inherits the actor, so the first diagnostics print off the main actor tripped the Swift 6 runtime isolation check. The seam's Compositor path had been built and tested on the Mac and in the simulator but never run on the headset until this comparison, where the session monitor's first "anchor missing" print fires at once; the RealityKit path never sets the hook. Fix: the hook is `@Sendable` and the source answers through itself (`CompositorFrameSource` is `@unchecked Sendable`) with `layerStateDescription`, read as the loop reads the state from its thread; `XRRuntimeHookTests` pins the type under `#if os(visionOS)`. Lesson for the seam branch: every Compositor Services app on `feature/xr_frame_source` and the presenter branch was unlaunchable on a device until this commit, so the seam's headset check is now done, by accident, and the upstream PR of the seam must carry this fix. ProceduralGeometry rebuilt with it runs on the headset (alive at 8 and 20 s), the Mac filter passes 79 tests. From 1b9b95e6775f0bb9e4fad60e0fb881517ea99d65 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 12:59:54 +0200 Subject: [PATCH 45/72] [Docs] XRRealityKitPresenter plan: the second Compositor crash (frames let go) and the first stages reading --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 1cbb7be29..a527ffd40 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -238,3 +238,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **How often (2026-10-19, 12:08): 1 to 2 late frames per second during quick head turns, up to 5 with the fastest, at 100 frames per second.** So 1 to 5 % of the frames land a refresh late while the head turns fast, each a one-refresh repeat of the previous texture that only head-locked content shows (the probe's dot); world-locked content, reprojected by RealityKit from the grid, shows nothing. The size of the item for a game: a HUD drawn by the engine stutters now and then on fast turns, a HUD drawn by RealityKit does not; the scene itself is unaffected. Whether the Compositor path repeats frames the same way under the same turns is the comparison in progress (its own session: the engine's stats monitor and ProceduralGeometry's stats window gain the GPU span maximum, the late count and the worst frame's split with the presenter's definitions). - **The Compositor path instrumented for the comparison (2026-10-19, 12:37; engine `feature/xr_realitykit_presenter` fast-forwarded to `2eb7414f`, pushed; arcade branch `feature/procedural_geometry_frame_stages` `d91bb7f` on main, not merged; ProceduralGeometry built in Release with `SWIFT_ACTIVE_COMPILATION_CONDITIONS='$(inherited) ENGINE_STATS_ENABLED'` against the merged engine, signed as com.miolabs.proceduralgeometry, installed beside the presenter example at 12:33).** Built in its own session. The engine's stats monitor splits the XR loop's frames with the presenter's definitions (`EngineFrameStages`, `EngineFrameStageStats`, `formatEngineFrameStages`, all under `ENGINE_STATS_ENABLED`; the loop stamps the commit before `commit()` and adds a second completed handler on the display's buffer): cpu = the loop's frame begin to the commit less the compositor's pacing sleep for the optimal input time (the RealityKit path sleeps nothing, so "late" means the same on both), wait = commit to the GPU's start, gpu = the raw span, late = completion over the refresh measured from the frames begun, plus a count of frames past the compositor's rendering deadline. ProceduralGeometry's Engine Stats window shows the line "gpu span max X ms · late N of F (refresh R ms) · worst A = cpu B + wait C + gpu D" above the compact text and takes `--open-space --stats-window --log-stats`; its committed project did not build from a clean checkout (the extension's pin moved to 187eaaf; the committed Release app target is at -Onone, set to -O in the local spec for the build). 79 tests. The reading: the same quick turns in both apps; a span stretched to ~20 ms with a small wait on the Compositor path too says the platform's GPU load under head motion; a span at its rest value with late 0 says the RealityKit path's buffer runs at app priority behind RealityKit's own render. - **The Compositor path crashed at launch on the headset, found and fixed (2026-10-19, 12:47; engine `0b946f03` on the presenter branch, pushed).** The first ProceduralGeometry build on the seam died within a second of launching, three crash logs in a minute (`EXC_BREAKPOINT` in `dispatch_assert_queue` from `_swift_task_checkIsolatedSwift`, faulting in `closure #1 in CompositorFrameSource.init` called by `XRRuntime.printXRAnchorDiagnostics` from the session event monitor's task). The seam (stage 1) had given the runtime a hook, `presenterStateDescription: () -> String`, which the Compositor source fills in from its `@MainActor` init with the layer's state; a non-Sendable closure formed in a main-actor context inherits the actor, so the first diagnostics print off the main actor tripped the Swift 6 runtime isolation check. The seam's Compositor path had been built and tested on the Mac and in the simulator but never run on the headset until this comparison, where the session monitor's first "anchor missing" print fires at once; the RealityKit path never sets the hook. Fix: the hook is `@Sendable` and the source answers through itself (`CompositorFrameSource` is `@unchecked Sendable`) with `layerStateDescription`, read as the loop reads the state from its thread; `XRRuntimeHookTests` pins the type under `#if os(visionOS)`. Lesson for the seam branch: every Compositor Services app on `feature/xr_frame_source` and the presenter branch was unlaunchable on a device until this commit, so the seam's headset check is now done, by accident, and the upstream PR of the seam must carry this fix. ProceduralGeometry rebuilt with it runs on the headset (alive at 8 and 20 s), the Mac filter passes 79 tests. +- **A second Compositor-path crash on the seam, and the first reading (2026-10-19, 12:59).** With the isolation fix in, ProceduralGeometry ran for three minutes and then aborted on the render thread inside `queryNextFrame` with the compositor's own "BUG IN CLIENT" (crash log 12:49:01, `cp_layer_renderer_query_next_frame` from `CompositorFrameSource.takeFrame()`), which is the compositor's refusal when frames are taken and never submitted (three of them abort the app, as the source's own pacing comment records). The seam's frame lifecycle matches upstream's path by path on the Mac (query, predictTiming, update, the optimal-input wait, submission, drawables, endSubmission), so the trigger is in what the seam reworked or added: the capture drawable (a screenshot or recording adds a second drawable per frame, and the user was taking the window's screenshot at the time), the half-rate pacing sleep before the query (new on the seam), or the loop's start and stop across a close and reopen of the space. Under review. Meanwhile the first stages line, read before the crash: GPU span max 8.6 ms against 6.9 ms of GPU execution on average, worst 18.8 = cpu 10.0 + wait 0.5 + gpu 8.2, and "late 100 of 100": the last is an artifact fixed as `5e5cb57d` (the CPU stage held the loop's wait for the drawables, which the compositor hands out at its phase of the refresh; both waits are the pacing now), the first two are the comparison's first data: the Compositor path's span stretched under 2 ms where the presenter's stretched 16, and no frame passed the compositor's deadline. Whether that line was read during quick turns is to be confirmed; the corrected build is installed and waits for the headset to be worn (the launch hangs with the device asleep). From e219d58964eeb4478c8265199e33e6b10023d7a0 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 13:22:56 +0200 Subject: [PATCH 46/72] [Docs] XRRealityKitPresenter plan: the frames let go on the Compositor path, reviewed and closed --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index a527ffd40..4859d9853 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -239,3 +239,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The Compositor path instrumented for the comparison (2026-10-19, 12:37; engine `feature/xr_realitykit_presenter` fast-forwarded to `2eb7414f`, pushed; arcade branch `feature/procedural_geometry_frame_stages` `d91bb7f` on main, not merged; ProceduralGeometry built in Release with `SWIFT_ACTIVE_COMPILATION_CONDITIONS='$(inherited) ENGINE_STATS_ENABLED'` against the merged engine, signed as com.miolabs.proceduralgeometry, installed beside the presenter example at 12:33).** Built in its own session. The engine's stats monitor splits the XR loop's frames with the presenter's definitions (`EngineFrameStages`, `EngineFrameStageStats`, `formatEngineFrameStages`, all under `ENGINE_STATS_ENABLED`; the loop stamps the commit before `commit()` and adds a second completed handler on the display's buffer): cpu = the loop's frame begin to the commit less the compositor's pacing sleep for the optimal input time (the RealityKit path sleeps nothing, so "late" means the same on both), wait = commit to the GPU's start, gpu = the raw span, late = completion over the refresh measured from the frames begun, plus a count of frames past the compositor's rendering deadline. ProceduralGeometry's Engine Stats window shows the line "gpu span max X ms · late N of F (refresh R ms) · worst A = cpu B + wait C + gpu D" above the compact text and takes `--open-space --stats-window --log-stats`; its committed project did not build from a clean checkout (the extension's pin moved to 187eaaf; the committed Release app target is at -Onone, set to -O in the local spec for the build). 79 tests. The reading: the same quick turns in both apps; a span stretched to ~20 ms with a small wait on the Compositor path too says the platform's GPU load under head motion; a span at its rest value with late 0 says the RealityKit path's buffer runs at app priority behind RealityKit's own render. - **The Compositor path crashed at launch on the headset, found and fixed (2026-10-19, 12:47; engine `0b946f03` on the presenter branch, pushed).** The first ProceduralGeometry build on the seam died within a second of launching, three crash logs in a minute (`EXC_BREAKPOINT` in `dispatch_assert_queue` from `_swift_task_checkIsolatedSwift`, faulting in `closure #1 in CompositorFrameSource.init` called by `XRRuntime.printXRAnchorDiagnostics` from the session event monitor's task). The seam (stage 1) had given the runtime a hook, `presenterStateDescription: () -> String`, which the Compositor source fills in from its `@MainActor` init with the layer's state; a non-Sendable closure formed in a main-actor context inherits the actor, so the first diagnostics print off the main actor tripped the Swift 6 runtime isolation check. The seam's Compositor path had been built and tested on the Mac and in the simulator but never run on the headset until this comparison, where the session monitor's first "anchor missing" print fires at once; the RealityKit path never sets the hook. Fix: the hook is `@Sendable` and the source answers through itself (`CompositorFrameSource` is `@unchecked Sendable`) with `layerStateDescription`, read as the loop reads the state from its thread; `XRRuntimeHookTests` pins the type under `#if os(visionOS)`. Lesson for the seam branch: every Compositor Services app on `feature/xr_frame_source` and the presenter branch was unlaunchable on a device until this commit, so the seam's headset check is now done, by accident, and the upstream PR of the seam must carry this fix. ProceduralGeometry rebuilt with it runs on the headset (alive at 8 and 20 s), the Mac filter passes 79 tests. - **A second Compositor-path crash on the seam, and the first reading (2026-10-19, 12:59).** With the isolation fix in, ProceduralGeometry ran for three minutes and then aborted on the render thread inside `queryNextFrame` with the compositor's own "BUG IN CLIENT" (crash log 12:49:01, `cp_layer_renderer_query_next_frame` from `CompositorFrameSource.takeFrame()`), which is the compositor's refusal when frames are taken and never submitted (three of them abort the app, as the source's own pacing comment records). The seam's frame lifecycle matches upstream's path by path on the Mac (query, predictTiming, update, the optimal-input wait, submission, drawables, endSubmission), so the trigger is in what the seam reworked or added: the capture drawable (a screenshot or recording adds a second drawable per frame, and the user was taking the window's screenshot at the time), the half-rate pacing sleep before the query (new on the seam), or the loop's start and stop across a close and reopen of the space. Under review. Meanwhile the first stages line, read before the crash: GPU span max 8.6 ms against 6.9 ms of GPU execution on average, worst 18.8 = cpu 10.0 + wait 0.5 + gpu 8.2, and "late 100 of 100": the last is an artifact fixed as `5e5cb57d` (the CPU stage held the loop's wait for the drawables, which the compositor hands out at its phase of the refresh; both waits are the pacing now), the first two are the comparison's first data: the Compositor path's span stretched under 2 ms where the presenter's stretched 16, and no frame passed the compositor's deadline. Whether that line was read during quick turns is to be confirmed; the corrected build is installed and waits for the headset to be worn (the launch hangs with the device asleep). +- **The frames let go, reviewed and closed (2026-10-19, 13:22; engine `9415dcf3` on the presenter branch, pushed, 81 tests).** A 41-agent review (three lenses over the seam's loop, source and runtime against upstream's old loop, two skeptics per finding) confirmed 18 paths where a frame taken from the compositor can go unsubmitted, five of them the seam's own: a frame handed out after `stop()` landed while the source waited (the seam checked the loop once before the wait, upstream checked again right before the query); a stop() during a half-rate pacing sleep (same); a frame whose drawables were all skipped, a capture-only frame, which the loop treated as cancelled and returned with the submission begun and nothing presented (the capture drawable of a screenshot is exactly the drawable that is skipped, and ProceduralGeometry died while one was being taken). The rest are upstream's too: a stop() mid-frame dropping the frame in hand; a drawing refused because the layer paused; a target whose command buffer could not be made, marked handed out and never presented; `predictTiming` nil dropping a frame (Apple's sample too). ProceduralGeometry calls stop() never, so its three frames went by the capture path or by pauses mid-frame; the fix covers all of them: `XRFrame.release()` (nothing by default; the Compositor frame runs the phases a let-go frame missed on a running layer, queries its drawables if it had not, presents every drawable not yet presented cleared and ends the submission; a frame the compositor cancelled is left alone, a paused or invalidated layer reclaims its frames), drawables marked at presentation rather than at hand-out, the loop letting a frame go through `release()` on every abandon path with a line in the log ("XR: a frame let go (reason)", the first ten then one in a hundred) and a count on the stages line ("N let go"), the source re-reading the layer's state after its pacing sleep and saying when a frame comes without timing. Left as upstream has them: `endSubmission` skipped when the layer is not running at the end of a frame, and the frame without timing dropped (counted now). Three loop tests cover the let-go paths with fake frames (handed out after the stop, drawing refused, no targets). The seam's upstream PR must carry this commit and the isolation fix. From e130855acb052759f66d67aa6cc2ed75b3be22ce Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 13:28:19 +0200 Subject: [PATCH 47/72] [Docs] XRRealityKitPresenter plan: the Compositor path's first honest stages line --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 4859d9853..68dcc2fef 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -240,3 +240,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The Compositor path crashed at launch on the headset, found and fixed (2026-10-19, 12:47; engine `0b946f03` on the presenter branch, pushed).** The first ProceduralGeometry build on the seam died within a second of launching, three crash logs in a minute (`EXC_BREAKPOINT` in `dispatch_assert_queue` from `_swift_task_checkIsolatedSwift`, faulting in `closure #1 in CompositorFrameSource.init` called by `XRRuntime.printXRAnchorDiagnostics` from the session event monitor's task). The seam (stage 1) had given the runtime a hook, `presenterStateDescription: () -> String`, which the Compositor source fills in from its `@MainActor` init with the layer's state; a non-Sendable closure formed in a main-actor context inherits the actor, so the first diagnostics print off the main actor tripped the Swift 6 runtime isolation check. The seam's Compositor path had been built and tested on the Mac and in the simulator but never run on the headset until this comparison, where the session monitor's first "anchor missing" print fires at once; the RealityKit path never sets the hook. Fix: the hook is `@Sendable` and the source answers through itself (`CompositorFrameSource` is `@unchecked Sendable`) with `layerStateDescription`, read as the loop reads the state from its thread; `XRRuntimeHookTests` pins the type under `#if os(visionOS)`. Lesson for the seam branch: every Compositor Services app on `feature/xr_frame_source` and the presenter branch was unlaunchable on a device until this commit, so the seam's headset check is now done, by accident, and the upstream PR of the seam must carry this fix. ProceduralGeometry rebuilt with it runs on the headset (alive at 8 and 20 s), the Mac filter passes 79 tests. - **A second Compositor-path crash on the seam, and the first reading (2026-10-19, 12:59).** With the isolation fix in, ProceduralGeometry ran for three minutes and then aborted on the render thread inside `queryNextFrame` with the compositor's own "BUG IN CLIENT" (crash log 12:49:01, `cp_layer_renderer_query_next_frame` from `CompositorFrameSource.takeFrame()`), which is the compositor's refusal when frames are taken and never submitted (three of them abort the app, as the source's own pacing comment records). The seam's frame lifecycle matches upstream's path by path on the Mac (query, predictTiming, update, the optimal-input wait, submission, drawables, endSubmission), so the trigger is in what the seam reworked or added: the capture drawable (a screenshot or recording adds a second drawable per frame, and the user was taking the window's screenshot at the time), the half-rate pacing sleep before the query (new on the seam), or the loop's start and stop across a close and reopen of the space. Under review. Meanwhile the first stages line, read before the crash: GPU span max 8.6 ms against 6.9 ms of GPU execution on average, worst 18.8 = cpu 10.0 + wait 0.5 + gpu 8.2, and "late 100 of 100": the last is an artifact fixed as `5e5cb57d` (the CPU stage held the loop's wait for the drawables, which the compositor hands out at its phase of the refresh; both waits are the pacing now), the first two are the comparison's first data: the Compositor path's span stretched under 2 ms where the presenter's stretched 16, and no frame passed the compositor's deadline. Whether that line was read during quick turns is to be confirmed; the corrected build is installed and waits for the headset to be worn (the launch hangs with the device asleep). - **The frames let go, reviewed and closed (2026-10-19, 13:22; engine `9415dcf3` on the presenter branch, pushed, 81 tests).** A 41-agent review (three lenses over the seam's loop, source and runtime against upstream's old loop, two skeptics per finding) confirmed 18 paths where a frame taken from the compositor can go unsubmitted, five of them the seam's own: a frame handed out after `stop()` landed while the source waited (the seam checked the loop once before the wait, upstream checked again right before the query); a stop() during a half-rate pacing sleep (same); a frame whose drawables were all skipped, a capture-only frame, which the loop treated as cancelled and returned with the submission begun and nothing presented (the capture drawable of a screenshot is exactly the drawable that is skipped, and ProceduralGeometry died while one was being taken). The rest are upstream's too: a stop() mid-frame dropping the frame in hand; a drawing refused because the layer paused; a target whose command buffer could not be made, marked handed out and never presented; `predictTiming` nil dropping a frame (Apple's sample too). ProceduralGeometry calls stop() never, so its three frames went by the capture path or by pauses mid-frame; the fix covers all of them: `XRFrame.release()` (nothing by default; the Compositor frame runs the phases a let-go frame missed on a running layer, queries its drawables if it had not, presents every drawable not yet presented cleared and ends the submission; a frame the compositor cancelled is left alone, a paused or invalidated layer reclaims its frames), drawables marked at presentation rather than at hand-out, the loop letting a frame go through `release()` on every abandon path with a line in the log ("XR: a frame let go (reason)", the first ten then one in a hundred) and a count on the stages line ("N let go"), the source re-reading the layer's state after its pacing sleep and saying when a frame comes without timing. Left as upstream has them: `endSubmission` skipped when the layer is not running at the end of a frame, and the frame without timing dropped (counted now). Three loop tests cover the let-go paths with fake frames (handed out after the stop, drawing refused, no targets). The seam's upstream PR must carry this commit and the isolation fix. +- **The Compositor path's first honest stages line (2026-10-19, 13:28, ProceduralGeometry Release with stats, the release-path build, the conditions to be confirmed): gpu span max 12.5 ms · late 20 of 101 (refresh 10.0 ms) · worst 15.1 = cpu 1.5 + wait 1.1 + gpu 12.5; GPU execution 7.84 ms average, GPU cadence 86 frames per second, no frame past the compositor's deadline, nothing let go.** With the drawable wait out of the CPU stage the frame's own work reads 1.5 ms, as the presenter's 2.8 did. This scene costs the GPU twice the presenter example's (7.8 against 4 ms), so its margin to the 10 ms refresh is 2 ms and a fifth of the frames overrun it; none overran the compositor's rendering deadline, which lies later than the refresh (the compositor predicts a presentation two or more refreshes ahead), so "late against the refresh" is not the compositor's own lateness and the presenter's count is the stricter of the two readings. The span's stretch, the number that compares: 12.5 against 7.8 on this path (1.6×, 4.7 ms), 20 against 4 on the RealityKit path (5×, 16 ms) at its quick-turn reading. If this line was read during quick turns, the compositor keeps the GPU for the app's render where the presenter's buffer, ordinary work beside RealityKit's own, is time-sliced; if at rest, the quick-turn line is still to be read. The build survived the screenshot that produced it; the cumulative let-go count is being added to the line so a let-go frame from any moment since the launch shows. From 38bebfd779f2cd40a1ea1d54535242ad17e06080 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 13:49:04 +0200 Subject: [PATCH 48/72] [Docs] XRRealityKitPresenter plan: the pause mid-frame as the trigger, fixed; the Compositor comparison so far --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 68dcc2fef..07b9f7d5d 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -241,3 +241,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **A second Compositor-path crash on the seam, and the first reading (2026-10-19, 12:59).** With the isolation fix in, ProceduralGeometry ran for three minutes and then aborted on the render thread inside `queryNextFrame` with the compositor's own "BUG IN CLIENT" (crash log 12:49:01, `cp_layer_renderer_query_next_frame` from `CompositorFrameSource.takeFrame()`), which is the compositor's refusal when frames are taken and never submitted (three of them abort the app, as the source's own pacing comment records). The seam's frame lifecycle matches upstream's path by path on the Mac (query, predictTiming, update, the optimal-input wait, submission, drawables, endSubmission), so the trigger is in what the seam reworked or added: the capture drawable (a screenshot or recording adds a second drawable per frame, and the user was taking the window's screenshot at the time), the half-rate pacing sleep before the query (new on the seam), or the loop's start and stop across a close and reopen of the space. Under review. Meanwhile the first stages line, read before the crash: GPU span max 8.6 ms against 6.9 ms of GPU execution on average, worst 18.8 = cpu 10.0 + wait 0.5 + gpu 8.2, and "late 100 of 100": the last is an artifact fixed as `5e5cb57d` (the CPU stage held the loop's wait for the drawables, which the compositor hands out at its phase of the refresh; both waits are the pacing now), the first two are the comparison's first data: the Compositor path's span stretched under 2 ms where the presenter's stretched 16, and no frame passed the compositor's deadline. Whether that line was read during quick turns is to be confirmed; the corrected build is installed and waits for the headset to be worn (the launch hangs with the device asleep). - **The frames let go, reviewed and closed (2026-10-19, 13:22; engine `9415dcf3` on the presenter branch, pushed, 81 tests).** A 41-agent review (three lenses over the seam's loop, source and runtime against upstream's old loop, two skeptics per finding) confirmed 18 paths where a frame taken from the compositor can go unsubmitted, five of them the seam's own: a frame handed out after `stop()` landed while the source waited (the seam checked the loop once before the wait, upstream checked again right before the query); a stop() during a half-rate pacing sleep (same); a frame whose drawables were all skipped, a capture-only frame, which the loop treated as cancelled and returned with the submission begun and nothing presented (the capture drawable of a screenshot is exactly the drawable that is skipped, and ProceduralGeometry died while one was being taken). The rest are upstream's too: a stop() mid-frame dropping the frame in hand; a drawing refused because the layer paused; a target whose command buffer could not be made, marked handed out and never presented; `predictTiming` nil dropping a frame (Apple's sample too). ProceduralGeometry calls stop() never, so its three frames went by the capture path or by pauses mid-frame; the fix covers all of them: `XRFrame.release()` (nothing by default; the Compositor frame runs the phases a let-go frame missed on a running layer, queries its drawables if it had not, presents every drawable not yet presented cleared and ends the submission; a frame the compositor cancelled is left alone, a paused or invalidated layer reclaims its frames), drawables marked at presentation rather than at hand-out, the loop letting a frame go through `release()` on every abandon path with a line in the log ("XR: a frame let go (reason)", the first ten then one in a hundred) and a count on the stages line ("N let go"), the source re-reading the layer's state after its pacing sleep and saying when a frame comes without timing. Left as upstream has them: `endSubmission` skipped when the layer is not running at the end of a frame, and the frame without timing dropped (counted now). Three loop tests cover the let-go paths with fake frames (handed out after the stop, drawing refused, no targets). The seam's upstream PR must carry this commit and the isolation fix. - **The Compositor path's first honest stages line (2026-10-19, 13:28, ProceduralGeometry Release with stats, the release-path build, the conditions to be confirmed): gpu span max 12.5 ms · late 20 of 101 (refresh 10.0 ms) · worst 15.1 = cpu 1.5 + wait 1.1 + gpu 12.5; GPU execution 7.84 ms average, GPU cadence 86 frames per second, no frame past the compositor's deadline, nothing let go.** With the drawable wait out of the CPU stage the frame's own work reads 1.5 ms, as the presenter's 2.8 did. This scene costs the GPU twice the presenter example's (7.8 against 4 ms), so its margin to the 10 ms refresh is 2 ms and a fifth of the frames overrun it; none overran the compositor's rendering deadline, which lies later than the refresh (the compositor predicts a presentation two or more refreshes ahead), so "late against the refresh" is not the compositor's own lateness and the presenter's count is the stricter of the two readings. The span's stretch, the number that compares: 12.5 against 7.8 on this path (1.6×, 4.7 ms), 20 against 4 on the RealityKit path (5×, 16 ms) at its quick-turn reading. If this line was read during quick turns, the compositor keeps the GPU for the app's render where the presenter's buffer, ordinary work beside RealityKit's own, is time-sliced; if at rest, the quick-turn line is still to be read. The build survived the screenshot that produced it; the cumulative let-go count is being added to the line so a let-go frame from any moment since the launch shows. +- **The trigger found: a pause mid-frame (2026-10-19, 13:49; engine `9f3977eb` on the presenter branch, pushed, 81 tests).** The build with the release path survived its screenshot and died at a press of the Digital Crown, with the same assertion (crash log 13:26:27, the first `queryNextFrame` after the resume). The crown pauses the layer; the frame in hand was then dropped by the three guards on the layer's state (`beginDrawing` refusing on a layer not running, `endDrawing` skipping `endSubmission`, `release()` leaving a paused layer's frame alone on the assumption that the compositor reclaims it), and the compositor still counted it. Apple's sample checks the layer's state only at the top of its loop: a frame taken goes through `startSubmission` and `endSubmission` regardless, and the end is skipped only when the drawable query gives nothing. The guards are gone (a layer that is gone gives no drawables, which is the cancelled frame's end); upstream's loop carries the same guards and the same hole, so the seam's upstream PR fixes it there too. The Compositor comparison so far, conditions unconfirmed: span 12.5–12.7 ms max against 7.2–7.8 average (1.6×, no frame past the compositor's deadline, 20–32 of 101 over the refresh with this heavier scene) against the presenter's 20 against 4 (5×): the presenter's buffer, ordinary work beside RealityKit's own render, is the one time-sliced. To be confirmed by the user's conditions; then the Compositor detour closes. Housekeeping: the stage 1 seam has now had its headset check, by way of ProceduralGeometry (three bugs: the main-actor hook, the frames let go, the pause mid-frame). From c1ef740fb4e7b3674389025c59376564915f3574 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 14:02:10 +0200 Subject: [PATCH 49/72] [Docs] XRRealityKitPresenter plan: the crown press survived on the headset; the Compositor detour closed --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 07b9f7d5d..0ad13d452 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -242,3 +242,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The frames let go, reviewed and closed (2026-10-19, 13:22; engine `9415dcf3` on the presenter branch, pushed, 81 tests).** A 41-agent review (three lenses over the seam's loop, source and runtime against upstream's old loop, two skeptics per finding) confirmed 18 paths where a frame taken from the compositor can go unsubmitted, five of them the seam's own: a frame handed out after `stop()` landed while the source waited (the seam checked the loop once before the wait, upstream checked again right before the query); a stop() during a half-rate pacing sleep (same); a frame whose drawables were all skipped, a capture-only frame, which the loop treated as cancelled and returned with the submission begun and nothing presented (the capture drawable of a screenshot is exactly the drawable that is skipped, and ProceduralGeometry died while one was being taken). The rest are upstream's too: a stop() mid-frame dropping the frame in hand; a drawing refused because the layer paused; a target whose command buffer could not be made, marked handed out and never presented; `predictTiming` nil dropping a frame (Apple's sample too). ProceduralGeometry calls stop() never, so its three frames went by the capture path or by pauses mid-frame; the fix covers all of them: `XRFrame.release()` (nothing by default; the Compositor frame runs the phases a let-go frame missed on a running layer, queries its drawables if it had not, presents every drawable not yet presented cleared and ends the submission; a frame the compositor cancelled is left alone, a paused or invalidated layer reclaims its frames), drawables marked at presentation rather than at hand-out, the loop letting a frame go through `release()` on every abandon path with a line in the log ("XR: a frame let go (reason)", the first ten then one in a hundred) and a count on the stages line ("N let go"), the source re-reading the layer's state after its pacing sleep and saying when a frame comes without timing. Left as upstream has them: `endSubmission` skipped when the layer is not running at the end of a frame, and the frame without timing dropped (counted now). Three loop tests cover the let-go paths with fake frames (handed out after the stop, drawing refused, no targets). The seam's upstream PR must carry this commit and the isolation fix. - **The Compositor path's first honest stages line (2026-10-19, 13:28, ProceduralGeometry Release with stats, the release-path build, the conditions to be confirmed): gpu span max 12.5 ms · late 20 of 101 (refresh 10.0 ms) · worst 15.1 = cpu 1.5 + wait 1.1 + gpu 12.5; GPU execution 7.84 ms average, GPU cadence 86 frames per second, no frame past the compositor's deadline, nothing let go.** With the drawable wait out of the CPU stage the frame's own work reads 1.5 ms, as the presenter's 2.8 did. This scene costs the GPU twice the presenter example's (7.8 against 4 ms), so its margin to the 10 ms refresh is 2 ms and a fifth of the frames overrun it; none overran the compositor's rendering deadline, which lies later than the refresh (the compositor predicts a presentation two or more refreshes ahead), so "late against the refresh" is not the compositor's own lateness and the presenter's count is the stricter of the two readings. The span's stretch, the number that compares: 12.5 against 7.8 on this path (1.6×, 4.7 ms), 20 against 4 on the RealityKit path (5×, 16 ms) at its quick-turn reading. If this line was read during quick turns, the compositor keeps the GPU for the app's render where the presenter's buffer, ordinary work beside RealityKit's own, is time-sliced; if at rest, the quick-turn line is still to be read. The build survived the screenshot that produced it; the cumulative let-go count is being added to the line so a let-go frame from any moment since the launch shows. - **The trigger found: a pause mid-frame (2026-10-19, 13:49; engine `9f3977eb` on the presenter branch, pushed, 81 tests).** The build with the release path survived its screenshot and died at a press of the Digital Crown, with the same assertion (crash log 13:26:27, the first `queryNextFrame` after the resume). The crown pauses the layer; the frame in hand was then dropped by the three guards on the layer's state (`beginDrawing` refusing on a layer not running, `endDrawing` skipping `endSubmission`, `release()` leaving a paused layer's frame alone on the assumption that the compositor reclaims it), and the compositor still counted it. Apple's sample checks the layer's state only at the top of its loop: a frame taken goes through `startSubmission` and `endSubmission` regardless, and the end is skipped only when the drawable query gives nothing. The guards are gone (a layer that is gone gives no drawables, which is the cancelled frame's end); upstream's loop carries the same guards and the same hole, so the seam's upstream PR fixes it there too. The Compositor comparison so far, conditions unconfirmed: span 12.5–12.7 ms max against 7.2–7.8 average (1.6×, no frame past the compositor's deadline, 20–32 of 101 over the refresh with this heavier scene) against the presenter's 20 against 4 (5×): the presenter's buffer, ordinary work beside RealityKit's own render, is the one time-sliced. To be confirmed by the user's conditions; then the Compositor detour closes. Housekeeping: the stage 1 seam has now had its headset check, by way of ProceduralGeometry (three bugs: the main-actor hook, the frames let go, the pause mid-frame). +- **Verified on the headset (2026-10-19, 14:02): with `9f3977eb` ProceduralGeometry survives the crown press that killed it at 13:26** (a double press pauses and resumes the layer: the frame in hand now goes through its submission; no crash log since 13:26, the process alive). A single press closes the immersive space, and this demo makes its engine once, bound to the first layer, so a reopened space draws nothing: the demo's own limitation, the same on upstream, not the engine's (the RealityKit example reopens its space; the Compositor path's reopen with a new layer is a template question for stage 5). The Compositor detour closes here; the seam's headset check is done. From b4dee0d801abf7e0c315f1fefa423ab2c4e9f2e4 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 14:03:22 +0200 Subject: [PATCH 50/72] [Docs] XRRealityKitPresenter plan: the comparison's verdict, the late frames are the RealityKit route's --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 0ad13d452..1724272bc 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -243,3 +243,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The Compositor path's first honest stages line (2026-10-19, 13:28, ProceduralGeometry Release with stats, the release-path build, the conditions to be confirmed): gpu span max 12.5 ms · late 20 of 101 (refresh 10.0 ms) · worst 15.1 = cpu 1.5 + wait 1.1 + gpu 12.5; GPU execution 7.84 ms average, GPU cadence 86 frames per second, no frame past the compositor's deadline, nothing let go.** With the drawable wait out of the CPU stage the frame's own work reads 1.5 ms, as the presenter's 2.8 did. This scene costs the GPU twice the presenter example's (7.8 against 4 ms), so its margin to the 10 ms refresh is 2 ms and a fifth of the frames overrun it; none overran the compositor's rendering deadline, which lies later than the refresh (the compositor predicts a presentation two or more refreshes ahead), so "late against the refresh" is not the compositor's own lateness and the presenter's count is the stricter of the two readings. The span's stretch, the number that compares: 12.5 against 7.8 on this path (1.6×, 4.7 ms), 20 against 4 on the RealityKit path (5×, 16 ms) at its quick-turn reading. If this line was read during quick turns, the compositor keeps the GPU for the app's render where the presenter's buffer, ordinary work beside RealityKit's own, is time-sliced; if at rest, the quick-turn line is still to be read. The build survived the screenshot that produced it; the cumulative let-go count is being added to the line so a let-go frame from any moment since the launch shows. - **The trigger found: a pause mid-frame (2026-10-19, 13:49; engine `9f3977eb` on the presenter branch, pushed, 81 tests).** The build with the release path survived its screenshot and died at a press of the Digital Crown, with the same assertion (crash log 13:26:27, the first `queryNextFrame` after the resume). The crown pauses the layer; the frame in hand was then dropped by the three guards on the layer's state (`beginDrawing` refusing on a layer not running, `endDrawing` skipping `endSubmission`, `release()` leaving a paused layer's frame alone on the assumption that the compositor reclaims it), and the compositor still counted it. Apple's sample checks the layer's state only at the top of its loop: a frame taken goes through `startSubmission` and `endSubmission` regardless, and the end is skipped only when the drawable query gives nothing. The guards are gone (a layer that is gone gives no drawables, which is the cancelled frame's end); upstream's loop carries the same guards and the same hole, so the seam's upstream PR fixes it there too. The Compositor comparison so far, conditions unconfirmed: span 12.5–12.7 ms max against 7.2–7.8 average (1.6×, no frame past the compositor's deadline, 20–32 of 101 over the refresh with this heavier scene) against the presenter's 20 against 4 (5×): the presenter's buffer, ordinary work beside RealityKit's own render, is the one time-sliced. To be confirmed by the user's conditions; then the Compositor detour closes. Housekeeping: the stage 1 seam has now had its headset check, by way of ProceduralGeometry (three bugs: the main-actor hook, the frames let go, the pause mid-frame). - **Verified on the headset (2026-10-19, 14:02): with `9f3977eb` ProceduralGeometry survives the crown press that killed it at 13:26** (a double press pauses and resumes the layer: the frame in hand now goes through its submission; no crash log since 13:26, the process alive). A single press closes the immersive space, and this demo makes its engine once, bound to the first layer, so a reopened space draws nothing: the demo's own limitation, the same on upstream, not the engine's (the RealityKit example reopens its space; the Compositor path's reopen with a new layer is a template question for stage 5). The Compositor detour closes here; the seam's headset check is done. +- **The comparison's verdict (2026-10-19, 14:03): the presenter's late frames are the RealityKit route's, not the device's.** The user confirms both ProceduralGeometry lines were read while turning the head quickly. Under the same motion the Compositor path's GPU span stretched from 7.2–7.8 ms of execution to 12.5–12.7 ms at most (1.6×, with no frame past the compositor's rendering deadline), while the presenter's stretched from 4 to 20 ms (5×): the compositor keeps its app's render on schedule, and the presenter's command buffer, ordinary work from the app's process beside RealityKit's own render of the same frame, is the one time-sliced. So the 1 to 5 late frames in a hundred on quick turns are a cost of presenting through RealityKit, visible only on head-locked content drawn by the engine (the scene is reprojected from the grid). What could reduce it, none yet chosen: a second refresh of pipeline depth (the engine's frame started a refresh earlier, which absorbs 10 ms of stretch at the price of one refresh of lead, which the swim probe would judge), less GPU work per frame (a shorter buffer is time-sliced less), head-locked content drawn by RealityKit, and a request to Apple for a priority for a LowLevelTexture producer's GPU work. Nothing on this is needed for world-locked games. The Compositor detour closes with the three seam fixes (`0b946f03`, `9415dcf3` with `90994806`, `9f3977eb`) and the stages instrumentation on both paths. From 90d3cc8a0da127cc0cfc6188734359fe72437210 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:08:35 +0200 Subject: [PATCH 51/72] [Docs] XRRealityKitPresenter plan: the pipeline two refreshes deep, built and merged --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 1724272bc..db067bb1f 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -244,3 +244,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The trigger found: a pause mid-frame (2026-10-19, 13:49; engine `9f3977eb` on the presenter branch, pushed, 81 tests).** The build with the release path survived its screenshot and died at a press of the Digital Crown, with the same assertion (crash log 13:26:27, the first `queryNextFrame` after the resume). The crown pauses the layer; the frame in hand was then dropped by the three guards on the layer's state (`beginDrawing` refusing on a layer not running, `endDrawing` skipping `endSubmission`, `release()` leaving a paused layer's frame alone on the assumption that the compositor reclaims it), and the compositor still counted it. Apple's sample checks the layer's state only at the top of its loop: a frame taken goes through `startSubmission` and `endSubmission` regardless, and the end is skipped only when the drawable query gives nothing. The guards are gone (a layer that is gone gives no drawables, which is the cancelled frame's end); upstream's loop carries the same guards and the same hole, so the seam's upstream PR fixes it there too. The Compositor comparison so far, conditions unconfirmed: span 12.5–12.7 ms max against 7.2–7.8 average (1.6×, no frame past the compositor's deadline, 20–32 of 101 over the refresh with this heavier scene) against the presenter's 20 against 4 (5×): the presenter's buffer, ordinary work beside RealityKit's own render, is the one time-sliced. To be confirmed by the user's conditions; then the Compositor detour closes. Housekeeping: the stage 1 seam has now had its headset check, by way of ProceduralGeometry (three bugs: the main-actor hook, the frames let go, the pause mid-frame). - **Verified on the headset (2026-10-19, 14:02): with `9f3977eb` ProceduralGeometry survives the crown press that killed it at 13:26** (a double press pauses and resumes the layer: the frame in hand now goes through its submission; no crash log since 13:26, the process alive). A single press closes the immersive space, and this demo makes its engine once, bound to the first layer, so a reopened space draws nothing: the demo's own limitation, the same on upstream, not the engine's (the RealityKit example reopens its space; the Compositor path's reopen with a new layer is a template question for stage 5). The Compositor detour closes here; the seam's headset check is done. - **The comparison's verdict (2026-10-19, 14:03): the presenter's late frames are the RealityKit route's, not the device's.** The user confirms both ProceduralGeometry lines were read while turning the head quickly. Under the same motion the Compositor path's GPU span stretched from 7.2–7.8 ms of execution to 12.5–12.7 ms at most (1.6×, with no frame past the compositor's rendering deadline), while the presenter's stretched from 4 to 20 ms (5×): the compositor keeps its app's render on schedule, and the presenter's command buffer, ordinary work from the app's process beside RealityKit's own render of the same frame, is the one time-sliced. So the 1 to 5 late frames in a hundred on quick turns are a cost of presenting through RealityKit, visible only on head-locked content drawn by the engine (the scene is reprojected from the grid). What could reduce it, none yet chosen: a second refresh of pipeline depth (the engine's frame started a refresh earlier, which absorbs 10 ms of stretch at the price of one refresh of lead, which the swim probe would judge), less GPU work per frame (a shorter buffer is time-sliced less), head-locked content drawn by RealityKit, and a request to Apple for a priority for a LowLevelTexture producer's GPU work. Nothing on this is needed for world-locked games. The Compositor detour closes with the three seam fixes (`0b946f03`, `9415dcf3` with `90994806`, `9f3977eb`) and the stages instrumentation on both paths. +- **The pipeline two refreshes deep, built and merged (2026-10-19, 15:08; engine `feature/xr_realitykit_presenter` fast-forwarded to `fe14609f`, example `feature/realitykit_presenter_example` to `32dea7e`, both pushed; 88 tests).** Built in its own session, off by default (`XRRealityKitPresenterConfiguration.pipelineDepth`, 1 or 2, chosen for the next space opened; `XRPresenterSettings.pipelineDepth` within the sets made). Two sets of textures, grids, band meshes and entities, alternated by update; each ticket's replace calls on a presentation buffer of its own, which the presenter commits at the next update once the engine has committed the frame's buffer (ordered behind it on the queue, so RealityKit takes the frame drawn); at each update the set shown is the one whose ticket of two updates ago completed, the other hidden, a frame later than its plan held back and skipped. A single set could not hold a frame for a later render: RealityKit shows a replaced texture at the first render after its buffer completes. Each ticket carries the refresh period measured from the updates (`XRPresenterRefresh`, an averaged interval, never assumed), and the engine's thread adds one refresh to the fixed or the adaptive lead. The late count's plan is the depth: the line reads "N late (plan 2 refreshes)" and the lead about 32 ms at 100 Hz. The example: "Two-refresh pipeline" in the next-space row, `--pipeline-depth 2|1`. Simulator: 0 late at depth 2, the scene and probe whole. Two things reasoned, not measured, for the headset: a frame one render early when its presentation completes in the sub-millisecond before an update while RealityKit's pickup precedes its update event; a frame two renders early when the engine's whole frame fits inside RealityKit's update-to-pickup gap. The headset reading: the late count during quick turns with the toggle off and on (near zero expected on), and the swim probe's dot with the lead one refresh longer (it must still hold still; a lag wants the fixed lead or the display latency moved, not the depth). From bb0d53d35a3a85676f508d27bf665535047b2ebe Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:26:46 +0200 Subject: [PATCH 52/72] [Docs] XRRealityKitPresenter plan: the two-refresh pipeline removes the late frames and the jump on the headset --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index db067bb1f..253d4e7c5 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -245,3 +245,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Verified on the headset (2026-10-19, 14:02): with `9f3977eb` ProceduralGeometry survives the crown press that killed it at 13:26** (a double press pauses and resumes the layer: the frame in hand now goes through its submission; no crash log since 13:26, the process alive). A single press closes the immersive space, and this demo makes its engine once, bound to the first layer, so a reopened space draws nothing: the demo's own limitation, the same on upstream, not the engine's (the RealityKit example reopens its space; the Compositor path's reopen with a new layer is a template question for stage 5). The Compositor detour closes here; the seam's headset check is done. - **The comparison's verdict (2026-10-19, 14:03): the presenter's late frames are the RealityKit route's, not the device's.** The user confirms both ProceduralGeometry lines were read while turning the head quickly. Under the same motion the Compositor path's GPU span stretched from 7.2–7.8 ms of execution to 12.5–12.7 ms at most (1.6×, with no frame past the compositor's rendering deadline), while the presenter's stretched from 4 to 20 ms (5×): the compositor keeps its app's render on schedule, and the presenter's command buffer, ordinary work from the app's process beside RealityKit's own render of the same frame, is the one time-sliced. So the 1 to 5 late frames in a hundred on quick turns are a cost of presenting through RealityKit, visible only on head-locked content drawn by the engine (the scene is reprojected from the grid). What could reduce it, none yet chosen: a second refresh of pipeline depth (the engine's frame started a refresh earlier, which absorbs 10 ms of stretch at the price of one refresh of lead, which the swim probe would judge), less GPU work per frame (a shorter buffer is time-sliced less), head-locked content drawn by RealityKit, and a request to Apple for a priority for a LowLevelTexture producer's GPU work. Nothing on this is needed for world-locked games. The Compositor detour closes with the three seam fixes (`0b946f03`, `9415dcf3` with `90994806`, `9f3977eb`) and the stages instrumentation on both paths. - **The pipeline two refreshes deep, built and merged (2026-10-19, 15:08; engine `feature/xr_realitykit_presenter` fast-forwarded to `fe14609f`, example `feature/realitykit_presenter_example` to `32dea7e`, both pushed; 88 tests).** Built in its own session, off by default (`XRRealityKitPresenterConfiguration.pipelineDepth`, 1 or 2, chosen for the next space opened; `XRPresenterSettings.pipelineDepth` within the sets made). Two sets of textures, grids, band meshes and entities, alternated by update; each ticket's replace calls on a presentation buffer of its own, which the presenter commits at the next update once the engine has committed the frame's buffer (ordered behind it on the queue, so RealityKit takes the frame drawn); at each update the set shown is the one whose ticket of two updates ago completed, the other hidden, a frame later than its plan held back and skipped. A single set could not hold a frame for a later render: RealityKit shows a replaced texture at the first render after its buffer completes. Each ticket carries the refresh period measured from the updates (`XRPresenterRefresh`, an averaged interval, never assumed), and the engine's thread adds one refresh to the fixed or the adaptive lead. The late count's plan is the depth: the line reads "N late (plan 2 refreshes)" and the lead about 32 ms at 100 Hz. The example: "Two-refresh pipeline" in the next-space row, `--pipeline-depth 2|1`. Simulator: 0 late at depth 2, the scene and probe whole. Two things reasoned, not measured, for the headset: a frame one render early when its presentation completes in the sub-millisecond before an update while RealityKit's pickup precedes its update event; a frame two renders early when the engine's whole frame fits inside RealityKit's update-to-pickup gap. The headset reading: the late count during quick turns with the toggle off and on (near zero expected on), and the swim probe's dot with the lead one refresh longer (it must still hold still; a lag wants the fixed lead or the display latency moved, not the depth). +- **Headset verdict on the pipeline depth (2026-10-19, 15:26, Release, engine `fe14609f`): with the two-refresh pipeline on, no late frames on the line and no jump of the head-locked dot during quick turns**, against 1 to 5 in a hundred with it off. The RealityKit route's time-slicing of the engine's buffer is absorbed by the second refresh, as designed; neither of the two unmeasured edge cases (a frame a render early) showed as a step. What it costs, one refresh of lead (about 10 ms at 100 Hz), is to be judged by the dot's steadiness with the setting on before it becomes the default. From 72e3900e8c60375312c46dd01f68553855248c80 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:28:35 +0200 Subject: [PATCH 53/72] [Docs] XRRealityKitPresenter plan: the dot holds still under the two-refresh pipeline; the lead item closes --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 253d4e7c5..c5faa3216 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -246,3 +246,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The comparison's verdict (2026-10-19, 14:03): the presenter's late frames are the RealityKit route's, not the device's.** The user confirms both ProceduralGeometry lines were read while turning the head quickly. Under the same motion the Compositor path's GPU span stretched from 7.2–7.8 ms of execution to 12.5–12.7 ms at most (1.6×, with no frame past the compositor's rendering deadline), while the presenter's stretched from 4 to 20 ms (5×): the compositor keeps its app's render on schedule, and the presenter's command buffer, ordinary work from the app's process beside RealityKit's own render of the same frame, is the one time-sliced. So the 1 to 5 late frames in a hundred on quick turns are a cost of presenting through RealityKit, visible only on head-locked content drawn by the engine (the scene is reprojected from the grid). What could reduce it, none yet chosen: a second refresh of pipeline depth (the engine's frame started a refresh earlier, which absorbs 10 ms of stretch at the price of one refresh of lead, which the swim probe would judge), less GPU work per frame (a shorter buffer is time-sliced less), head-locked content drawn by RealityKit, and a request to Apple for a priority for a LowLevelTexture producer's GPU work. Nothing on this is needed for world-locked games. The Compositor detour closes with the three seam fixes (`0b946f03`, `9415dcf3` with `90994806`, `9f3977eb`) and the stages instrumentation on both paths. - **The pipeline two refreshes deep, built and merged (2026-10-19, 15:08; engine `feature/xr_realitykit_presenter` fast-forwarded to `fe14609f`, example `feature/realitykit_presenter_example` to `32dea7e`, both pushed; 88 tests).** Built in its own session, off by default (`XRRealityKitPresenterConfiguration.pipelineDepth`, 1 or 2, chosen for the next space opened; `XRPresenterSettings.pipelineDepth` within the sets made). Two sets of textures, grids, band meshes and entities, alternated by update; each ticket's replace calls on a presentation buffer of its own, which the presenter commits at the next update once the engine has committed the frame's buffer (ordered behind it on the queue, so RealityKit takes the frame drawn); at each update the set shown is the one whose ticket of two updates ago completed, the other hidden, a frame later than its plan held back and skipped. A single set could not hold a frame for a later render: RealityKit shows a replaced texture at the first render after its buffer completes. Each ticket carries the refresh period measured from the updates (`XRPresenterRefresh`, an averaged interval, never assumed), and the engine's thread adds one refresh to the fixed or the adaptive lead. The late count's plan is the depth: the line reads "N late (plan 2 refreshes)" and the lead about 32 ms at 100 Hz. The example: "Two-refresh pipeline" in the next-space row, `--pipeline-depth 2|1`. Simulator: 0 late at depth 2, the scene and probe whole. Two things reasoned, not measured, for the headset: a frame one render early when its presentation completes in the sub-millisecond before an update while RealityKit's pickup precedes its update event; a frame two renders early when the engine's whole frame fits inside RealityKit's update-to-pickup gap. The headset reading: the late count during quick turns with the toggle off and on (near zero expected on), and the swim probe's dot with the lead one refresh longer (it must still hold still; a lag wants the fixed lead or the display latency moved, not the depth). - **Headset verdict on the pipeline depth (2026-10-19, 15:26, Release, engine `fe14609f`): with the two-refresh pipeline on, no late frames on the line and no jump of the head-locked dot during quick turns**, against 1 to 5 in a hundred with it off. The RealityKit route's time-slicing of the engine's buffer is absorbed by the second refresh, as designed; neither of the two unmeasured edge cases (a frame a render early) showed as a step. What it costs, one refresh of lead (about 10 ms at 100 Hz), is to be judged by the dot's steadiness with the setting on before it becomes the default. +- **And the lead's cost is invisible (2026-10-19, 15:28): with the two-refresh pipeline on and the swim probe on, the dot holds still during turns.** So the second refresh costs the eye nothing at the 22 ms fixed lead plus one refresh: the stage 3 lead item closes with the depth, and the late frames of the RealityKit route are gone for one refresh of latency and a second set of textures and grids. The only open decision is whether depth 2 becomes the presenter's default (the configuration's `pipelineDepth`, the example's remembered toggle, the README); the evidence says yes. From 90016c7b69f6e73a4686333b140ba22ef8ed6b4f Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:34:19 +0200 Subject: [PATCH 54/72] [Docs] XRRealityKitPresenter plan: two refreshes deep by default; the sphere's steadiness to be compared --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index c5faa3216..d2423ec7c 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -247,3 +247,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The pipeline two refreshes deep, built and merged (2026-10-19, 15:08; engine `feature/xr_realitykit_presenter` fast-forwarded to `fe14609f`, example `feature/realitykit_presenter_example` to `32dea7e`, both pushed; 88 tests).** Built in its own session, off by default (`XRRealityKitPresenterConfiguration.pipelineDepth`, 1 or 2, chosen for the next space opened; `XRPresenterSettings.pipelineDepth` within the sets made). Two sets of textures, grids, band meshes and entities, alternated by update; each ticket's replace calls on a presentation buffer of its own, which the presenter commits at the next update once the engine has committed the frame's buffer (ordered behind it on the queue, so RealityKit takes the frame drawn); at each update the set shown is the one whose ticket of two updates ago completed, the other hidden, a frame later than its plan held back and skipped. A single set could not hold a frame for a later render: RealityKit shows a replaced texture at the first render after its buffer completes. Each ticket carries the refresh period measured from the updates (`XRPresenterRefresh`, an averaged interval, never assumed), and the engine's thread adds one refresh to the fixed or the adaptive lead. The late count's plan is the depth: the line reads "N late (plan 2 refreshes)" and the lead about 32 ms at 100 Hz. The example: "Two-refresh pipeline" in the next-space row, `--pipeline-depth 2|1`. Simulator: 0 late at depth 2, the scene and probe whole. Two things reasoned, not measured, for the headset: a frame one render early when its presentation completes in the sub-millisecond before an update while RealityKit's pickup precedes its update event; a frame two renders early when the engine's whole frame fits inside RealityKit's update-to-pickup gap. The headset reading: the late count during quick turns with the toggle off and on (near zero expected on), and the swim probe's dot with the lead one refresh longer (it must still hold still; a lag wants the fixed lead or the display latency moved, not the depth). - **Headset verdict on the pipeline depth (2026-10-19, 15:26, Release, engine `fe14609f`): with the two-refresh pipeline on, no late frames on the line and no jump of the head-locked dot during quick turns**, against 1 to 5 in a hundred with it off. The RealityKit route's time-slicing of the engine's buffer is absorbed by the second refresh, as designed; neither of the two unmeasured edge cases (a frame a render early) showed as a step. What it costs, one refresh of lead (about 10 ms at 100 Hz), is to be judged by the dot's steadiness with the setting on before it becomes the default. - **And the lead's cost is invisible (2026-10-19, 15:28): with the two-refresh pipeline on and the swim probe on, the dot holds still during turns.** So the second refresh costs the eye nothing at the 22 ms fixed lead plus one refresh: the stage 3 lead item closes with the depth, and the late frames of the RealityKit route are gone for one refresh of latency and a second set of textures and grids. The only open decision is whether depth 2 becomes the presenter's default (the configuration's `pipelineDepth`, the example's remembered toggle, the README); the evidence says yes. +- **Two refreshes deep by default (2026-10-19, 15:34, on the user's say-so):** `XRRealityKitPresenterConfiguration.pipelineDepth` is 2, the example's toggle starts on (`--pipeline-depth 1` for the pipeline as it was), the README says so; the four tests that count one set's entities or read the picture at the first render pin depth 1 (88 tests). One observation to settle before the default is final: the user sees the head-locked dot hold still better than the world-locked sphere. A world-locked silhouette is rebuilt each frame from the pose predicted for the frame, and the second refresh predicts one refresh further ahead, so a larger prediction error at a quick turn would show first on silhouettes (the stretched edge cells breathing), not on the dot; whether the sphere is steadier with the pipeline off is the reading that decides. If it is, the default is a trade between the late frames of head-locked content and the steadiness of world-locked silhouettes under fast turns. From 76671e27866a144473ac50e4a70f2a316286461c Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:37:03 +0200 Subject: [PATCH 55/72] [Docs] XRRealityKitPresenter plan: the default stands; where the presenter stands at the end of the day --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index d2423ec7c..fe8fca266 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -248,3 +248,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Headset verdict on the pipeline depth (2026-10-19, 15:26, Release, engine `fe14609f`): with the two-refresh pipeline on, no late frames on the line and no jump of the head-locked dot during quick turns**, against 1 to 5 in a hundred with it off. The RealityKit route's time-slicing of the engine's buffer is absorbed by the second refresh, as designed; neither of the two unmeasured edge cases (a frame a render early) showed as a step. What it costs, one refresh of lead (about 10 ms at 100 Hz), is to be judged by the dot's steadiness with the setting on before it becomes the default. - **And the lead's cost is invisible (2026-10-19, 15:28): with the two-refresh pipeline on and the swim probe on, the dot holds still during turns.** So the second refresh costs the eye nothing at the 22 ms fixed lead plus one refresh: the stage 3 lead item closes with the depth, and the late frames of the RealityKit route are gone for one refresh of latency and a second set of textures and grids. The only open decision is whether depth 2 becomes the presenter's default (the configuration's `pipelineDepth`, the example's remembered toggle, the README); the evidence says yes. - **Two refreshes deep by default (2026-10-19, 15:34, on the user's say-so):** `XRRealityKitPresenterConfiguration.pipelineDepth` is 2, the example's toggle starts on (`--pipeline-depth 1` for the pipeline as it was), the README says so; the four tests that count one set's entities or read the picture at the first render pin depth 1 (88 tests). One observation to settle before the default is final: the user sees the head-locked dot hold still better than the world-locked sphere. A world-locked silhouette is rebuilt each frame from the pose predicted for the frame, and the second refresh predicts one refresh further ahead, so a larger prediction error at a quick turn would show first on silhouettes (the stretched edge cells breathing), not on the dot; whether the sphere is steadier with the pipeline off is the reading that decides. If it is, the default is a trade between the late frames of head-locked content and the steadiness of world-locked silhouettes under fast turns. +- **The default stands (2026-10-19, 15:37): the sphere looks the same with the two-refresh pipeline on and off; only the dot differs.** The longer lead costs world-locked content nothing visible. What the eye reads on the sphere against the dot is the grid's own approximation, a curved silhouette drawn as planar cells of eight texels, exact on the slab and approximate on the sphere under head motion: the plan's two-layer grid item (and finer cells near silhouettes), unchanged by the pipeline. **Where the presenter stands at the end of 2026-10-19:** engine `feature/xr_realitykit_presenter` at `ac8cdde7` (the day's commits: the straightness test, the adaptive lead, the frame stages on both paths, the three seam fixes for the Compositor path, the two-refresh pipeline as the default), example `feature/realitykit_presenter_example` at `2284024`, both pushed; the arcade branch `feature/procedural_geometry_frame_stages` (`27e5640`, the Engine Stats line and launch options for ProceduralGeometry, plus the extension pin it needed to build) stays apart for a fork PR of its own; the headset holds a presenter example built from `fe14609f` with the toggle remembered on, and ProceduralGeometry built from `9f3977eb`. Closed today: the lead (fixed 22 ms right, adaptive kept off), the late frames (the RealityKit route's, absorbed by the second refresh), the cut and the band (at the coverage floor), the seam's headset check (three bugs found by way of ProceduralGeometry). Left in the plan: the picture items (the two-layer grid for the halo and the curved silhouettes, a relative tear threshold, TAA), stage 4 (Personas in CoolSaber, a second Vision Pro), stage 5 (template, docs, the editor's preview), and the upstream PRs (the seam first, carrying `0b946f03`, `9415dcf3`, `90994806` and `9f3977eb`; the fork PR descriptions still quote Debug numbers). From 3a0826deddd8cda6b47cb9d6502cc125997de5e1 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:37:40 +0200 Subject: [PATCH 56/72] [Docs] XRRealityKitPresenter plan: the sphere remark was about its edges; the render scale becomes affordable under the deeper pipeline --- docs/proposals/XRRealityKitPresenter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index fe8fca266..833ae55bf 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -248,4 +248,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Headset verdict on the pipeline depth (2026-10-19, 15:26, Release, engine `fe14609f`): with the two-refresh pipeline on, no late frames on the line and no jump of the head-locked dot during quick turns**, against 1 to 5 in a hundred with it off. The RealityKit route's time-slicing of the engine's buffer is absorbed by the second refresh, as designed; neither of the two unmeasured edge cases (a frame a render early) showed as a step. What it costs, one refresh of lead (about 10 ms at 100 Hz), is to be judged by the dot's steadiness with the setting on before it becomes the default. - **And the lead's cost is invisible (2026-10-19, 15:28): with the two-refresh pipeline on and the swim probe on, the dot holds still during turns.** So the second refresh costs the eye nothing at the 22 ms fixed lead plus one refresh: the stage 3 lead item closes with the depth, and the late frames of the RealityKit route are gone for one refresh of latency and a second set of textures and grids. The only open decision is whether depth 2 becomes the presenter's default (the configuration's `pipelineDepth`, the example's remembered toggle, the README); the evidence says yes. - **Two refreshes deep by default (2026-10-19, 15:34, on the user's say-so):** `XRRealityKitPresenterConfiguration.pipelineDepth` is 2, the example's toggle starts on (`--pipeline-depth 1` for the pipeline as it was), the README says so; the four tests that count one set's entities or read the picture at the first render pin depth 1 (88 tests). One observation to settle before the default is final: the user sees the head-locked dot hold still better than the world-locked sphere. A world-locked silhouette is rebuilt each frame from the pose predicted for the frame, and the second refresh predicts one refresh further ahead, so a larger prediction error at a quick turn would show first on silhouettes (the stretched edge cells breathing), not on the dot; whether the sphere is steadier with the pipeline off is the reading that decides. If it is, the default is a trade between the late frames of head-locked content and the steadiness of world-locked silhouettes under fast turns. -- **The default stands (2026-10-19, 15:37): the sphere looks the same with the two-refresh pipeline on and off; only the dot differs.** The longer lead costs world-locked content nothing visible. What the eye reads on the sphere against the dot is the grid's own approximation, a curved silhouette drawn as planar cells of eight texels, exact on the slab and approximate on the sphere under head motion: the plan's two-layer grid item (and finer cells near silhouettes), unchanged by the pipeline. **Where the presenter stands at the end of 2026-10-19:** engine `feature/xr_realitykit_presenter` at `ac8cdde7` (the day's commits: the straightness test, the adaptive lead, the frame stages on both paths, the three seam fixes for the Compositor path, the two-refresh pipeline as the default), example `feature/realitykit_presenter_example` at `2284024`, both pushed; the arcade branch `feature/procedural_geometry_frame_stages` (`27e5640`, the Engine Stats line and launch options for ProceduralGeometry, plus the extension pin it needed to build) stays apart for a fork PR of its own; the headset holds a presenter example built from `fe14609f` with the toggle remembered on, and ProceduralGeometry built from `9f3977eb`. Closed today: the lead (fixed 22 ms right, adaptive kept off), the late frames (the RealityKit route's, absorbed by the second refresh), the cut and the band (at the coverage floor), the seam's headset check (three bugs found by way of ProceduralGeometry). Left in the plan: the picture items (the two-layer grid for the halo and the curved silhouettes, a relative tear threshold, TAA), stage 4 (Personas in CoolSaber, a second Vision Pro), stage 5 (template, docs, the editor's preview), and the upstream PRs (the seam first, carrying `0b946f03`, `9415dcf3`, `90994806` and `9f3977eb`; the fork PR descriptions still quote Debug numbers). +- **The default stands (2026-10-19, 15:37): the sphere looks the same with the two-refresh pipeline on and off; only the dot differs.** The longer lead costs world-locked content nothing visible. The user's point about the sphere is its edges, not its steadiness: the sphere's silhouette looks the same with the pipeline on and off, as expected of a change that only moves when a frame shows, and the edge quality of the morning stays where the rim panel left it (the 4× MSAA quarter steps marching under motion, the grid's planar cells at a curved silhouette, and RealityKit's bilinear magnification of a 12.9 texel-per-degree texture to a 34 pixel-per-degree display). One thing the pipeline changes for the edges: the engine now has two refreshes per frame, so the render scale the plan judged unaffordable at one refresh (a higher texel density is the only lever against the magnification) is affordable: a 1.5× render target is about 2.25× the GPU work, 9 to 10 ms of a 20 ms budget. **Where the presenter stands at the end of 2026-10-19:** engine `feature/xr_realitykit_presenter` at `ac8cdde7` (the day's commits: the straightness test, the adaptive lead, the frame stages on both paths, the three seam fixes for the Compositor path, the two-refresh pipeline as the default), example `feature/realitykit_presenter_example` at `2284024`, both pushed; the arcade branch `feature/procedural_geometry_frame_stages` (`27e5640`, the Engine Stats line and launch options for ProceduralGeometry, plus the extension pin it needed to build) stays apart for a fork PR of its own; the headset holds a presenter example built from `fe14609f` with the toggle remembered on, and ProceduralGeometry built from `9f3977eb`. Closed today: the lead (fixed 22 ms right, adaptive kept off), the late frames (the RealityKit route's, absorbed by the second refresh), the cut and the band (at the coverage floor), the seam's headset check (three bugs found by way of ProceduralGeometry). Left in the plan: the picture items (the two-layer grid for the halo and the curved silhouettes, a relative tear threshold, TAA), stage 4 (Personas in CoolSaber, a second Vision Pro), stage 5 (template, docs, the editor's preview), and the upstream PRs (the seam first, carrying `0b946f03`, `9415dcf3`, `90994806` and `9f3977eb`; the fork PR descriptions still quote Debug numbers). From 370b3e85165d58c821de21f03c41e5deee9681aa Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 15:43:49 +0200 Subject: [PATCH 57/72] [Docs] XRRealityKitPresenter plan: MSAA+TAA leaves the edges as they are on the headset; the density is the lever --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 833ae55bf..59b58d198 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -249,3 +249,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **And the lead's cost is invisible (2026-10-19, 15:28): with the two-refresh pipeline on and the swim probe on, the dot holds still during turns.** So the second refresh costs the eye nothing at the 22 ms fixed lead plus one refresh: the stage 3 lead item closes with the depth, and the late frames of the RealityKit route are gone for one refresh of latency and a second set of textures and grids. The only open decision is whether depth 2 becomes the presenter's default (the configuration's `pipelineDepth`, the example's remembered toggle, the README); the evidence says yes. - **Two refreshes deep by default (2026-10-19, 15:34, on the user's say-so):** `XRRealityKitPresenterConfiguration.pipelineDepth` is 2, the example's toggle starts on (`--pipeline-depth 1` for the pipeline as it was), the README says so; the four tests that count one set's entities or read the picture at the first render pin depth 1 (88 tests). One observation to settle before the default is final: the user sees the head-locked dot hold still better than the world-locked sphere. A world-locked silhouette is rebuilt each frame from the pose predicted for the frame, and the second refresh predicts one refresh further ahead, so a larger prediction error at a quick turn would show first on silhouettes (the stretched edge cells breathing), not on the dot; whether the sphere is steadier with the pipeline off is the reading that decides. If it is, the default is a trade between the late frames of head-locked content and the steadiness of world-locked silhouettes under fast turns. - **The default stands (2026-10-19, 15:37): the sphere looks the same with the two-refresh pipeline on and off; only the dot differs.** The longer lead costs world-locked content nothing visible. The user's point about the sphere is its edges, not its steadiness: the sphere's silhouette looks the same with the pipeline on and off, as expected of a change that only moves when a frame shows, and the edge quality of the morning stays where the rim panel left it (the 4× MSAA quarter steps marching under motion, the grid's planar cells at a curved silhouette, and RealityKit's bilinear magnification of a 12.9 texel-per-degree texture to a 34 pixel-per-degree display). One thing the pipeline changes for the edges: the engine now has two refreshes per frame, so the render scale the plan judged unaffordable at one refresh (a higher texel density is the only lever against the magnification) is affordable: a 1.5× render target is about 2.25× the GPU work, 9 to 10 ms of a 20 ms budget. **Where the presenter stands at the end of 2026-10-19:** engine `feature/xr_realitykit_presenter` at `ac8cdde7` (the day's commits: the straightness test, the adaptive lead, the frame stages on both paths, the three seam fixes for the Compositor path, the two-refresh pipeline as the default), example `feature/realitykit_presenter_example` at `2284024`, both pushed; the arcade branch `feature/procedural_geometry_frame_stages` (`27e5640`, the Engine Stats line and launch options for ProceduralGeometry, plus the extension pin it needed to build) stays apart for a fork PR of its own; the headset holds a presenter example built from `fe14609f` with the toggle remembered on, and ProceduralGeometry built from `9f3977eb`. Closed today: the lead (fixed 22 ms right, adaptive kept off), the late frames (the RealityKit route's, absorbed by the second refresh), the cut and the band (at the coverage floor), the seam's headset check (three bugs found by way of ProceduralGeometry). Left in the plan: the picture items (the two-layer grid for the halo and the curved silhouettes, a relative tear threshold, TAA), stage 4 (Personas in CoolSaber, a second Vision Pro), stage 5 (template, docs, the editor's preview), and the upstream PRs (the seam first, carrying `0b946f03`, `9415dcf3`, `90994806` and `9f3977eb`; the fork PR descriptions still quote Debug numbers). +- **MSAA+TAA on the headset (2026-10-19, 15:43): the edges of the cubes and the sphere look the same.** Not a mode ignored: `TemporalAntiAliasing` keeps a history per eye, `applyJitter()` puts the frame's sub-texel jitter into each eye's projection after `projectionDidChange()` clears the old one, and the screen size comes from the rate map under foveation. What the temporal pass can do is average the engine's edge into a ramp about one texel wide and stop the quarter-step crawl under motion; what it cannot do is add texels, and a one-texel ramp is still shown 2.6 display pixels wide. So the levers left for the edges are the texel density (the render scale, in its own session now that the two-refresh pipeline pays for it) and, for curved silhouettes and the halo, the two-layer grid. TAA's own cost on the device was not read. From efbf0d7f472b11182dd355a541a3f9297ea1b34d Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 16:20:31 +0200 Subject: [PATCH 58/72] [Docs] XRRealityKitPresenter plan: the render scale, built and merged --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 59b58d198..c38d1b23e 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -250,3 +250,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Two refreshes deep by default (2026-10-19, 15:34, on the user's say-so):** `XRRealityKitPresenterConfiguration.pipelineDepth` is 2, the example's toggle starts on (`--pipeline-depth 1` for the pipeline as it was), the README says so; the four tests that count one set's entities or read the picture at the first render pin depth 1 (88 tests). One observation to settle before the default is final: the user sees the head-locked dot hold still better than the world-locked sphere. A world-locked silhouette is rebuilt each frame from the pose predicted for the frame, and the second refresh predicts one refresh further ahead, so a larger prediction error at a quick turn would show first on silhouettes (the stretched edge cells breathing), not on the dot; whether the sphere is steadier with the pipeline off is the reading that decides. If it is, the default is a trade between the late frames of head-locked content and the steadiness of world-locked silhouettes under fast turns. - **The default stands (2026-10-19, 15:37): the sphere looks the same with the two-refresh pipeline on and off; only the dot differs.** The longer lead costs world-locked content nothing visible. The user's point about the sphere is its edges, not its steadiness: the sphere's silhouette looks the same with the pipeline on and off, as expected of a change that only moves when a frame shows, and the edge quality of the morning stays where the rim panel left it (the 4× MSAA quarter steps marching under motion, the grid's planar cells at a curved silhouette, and RealityKit's bilinear magnification of a 12.9 texel-per-degree texture to a 34 pixel-per-degree display). One thing the pipeline changes for the edges: the engine now has two refreshes per frame, so the render scale the plan judged unaffordable at one refresh (a higher texel density is the only lever against the magnification) is affordable: a 1.5× render target is about 2.25× the GPU work, 9 to 10 ms of a 20 ms budget. **Where the presenter stands at the end of 2026-10-19:** engine `feature/xr_realitykit_presenter` at `ac8cdde7` (the day's commits: the straightness test, the adaptive lead, the frame stages on both paths, the three seam fixes for the Compositor path, the two-refresh pipeline as the default), example `feature/realitykit_presenter_example` at `2284024`, both pushed; the arcade branch `feature/procedural_geometry_frame_stages` (`27e5640`, the Engine Stats line and launch options for ProceduralGeometry, plus the extension pin it needed to build) stays apart for a fork PR of its own; the headset holds a presenter example built from `fe14609f` with the toggle remembered on, and ProceduralGeometry built from `9f3977eb`. Closed today: the lead (fixed 22 ms right, adaptive kept off), the late frames (the RealityKit route's, absorbed by the second refresh), the cut and the band (at the coverage floor), the seam's headset check (three bugs found by way of ProceduralGeometry). Left in the plan: the picture items (the two-layer grid for the halo and the curved silhouettes, a relative tear threshold, TAA), stage 4 (Personas in CoolSaber, a second Vision Pro), stage 5 (template, docs, the editor's preview), and the upstream PRs (the seam first, carrying `0b946f03`, `9415dcf3`, `90994806` and `9f3977eb`; the fork PR descriptions still quote Debug numbers). - **MSAA+TAA on the headset (2026-10-19, 15:43): the edges of the cubes and the sphere look the same.** Not a mode ignored: `TemporalAntiAliasing` keeps a history per eye, `applyJitter()` puts the frame's sub-texel jitter into each eye's projection after `projectionDidChange()` clears the old one, and the screen size comes from the rate map under foveation. What the temporal pass can do is average the engine's edge into a ramp about one texel wide and stop the quarter-step crawl under motion; what it cannot do is add texels, and a one-texel ramp is still shown 2.6 display pixels wide. So the levers left for the edges are the texel density (the render scale, in its own session now that the two-refresh pipeline pays for it) and, for curved silhouettes and the halo, the two-layer grid. TAA's own cost on the device was not read. +- **The render scale, built and merged (2026-10-19, 16:20; engine `feature/xr_realitykit_presenter` fast-forwarded to `ca92ed1e`, example `feature/realitykit_presenter_example` to `bf59a7f`, both pushed; 95 tests).** Built in its own session. `XRRealityKitPresenterConfiguration.renderScale` (1 by default, clamped 0.5 to 2, chosen for the next space opened) sizes the eye textures, the depth pair, the kernel's scratch and, through the rate map, the physical budget so the map's screen and the centre density gain the same factor; the grids keep their vertex counts (a uniform grid's cell grows with the texels; a foveated grid's cell is `foveatedCellTexels` times the scale snapped to a divisor of the 32-texel tile: 8 becomes 12, which snaps to 16, the map's grid 181 × 172 for the scaled budget against 241 × 229; 4 would keep the dense grid); scale 1 is byte-identical. `centreTexelsPerDegree` on the configuration and the presenter: 12.9 at scale 1, 19.3 at 1.5, 17.6 and 26.5 through the map, against the display's some 34; the settings line names it. On the Mac (M4 Max, the test scene, two sets) the GPU span goes 2.76 → 5.5 ms for 2.25× the pixels, the eye textures 27 → 62 MB per set. The example: "Render scale 1.5×" in the next-space row, `--render-scale 1.5`. The headset judgement: the cubes' and the sphere's edges with the toggle on, the GPU and the late count on the numbers line under the two-refresh pipeline. From 1af84f14b3981f3e765cea86ca9efda32232d88c Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 16:29:41 +0200 Subject: [PATCH 59/72] [Docs] XRRealityKitPresenter plan: the render scale at 1.5 reads as a small difference; the map with it is the next reading --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index c38d1b23e..4040206c1 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -251,3 +251,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The default stands (2026-10-19, 15:37): the sphere looks the same with the two-refresh pipeline on and off; only the dot differs.** The longer lead costs world-locked content nothing visible. The user's point about the sphere is its edges, not its steadiness: the sphere's silhouette looks the same with the pipeline on and off, as expected of a change that only moves when a frame shows, and the edge quality of the morning stays where the rim panel left it (the 4× MSAA quarter steps marching under motion, the grid's planar cells at a curved silhouette, and RealityKit's bilinear magnification of a 12.9 texel-per-degree texture to a 34 pixel-per-degree display). One thing the pipeline changes for the edges: the engine now has two refreshes per frame, so the render scale the plan judged unaffordable at one refresh (a higher texel density is the only lever against the magnification) is affordable: a 1.5× render target is about 2.25× the GPU work, 9 to 10 ms of a 20 ms budget. **Where the presenter stands at the end of 2026-10-19:** engine `feature/xr_realitykit_presenter` at `ac8cdde7` (the day's commits: the straightness test, the adaptive lead, the frame stages on both paths, the three seam fixes for the Compositor path, the two-refresh pipeline as the default), example `feature/realitykit_presenter_example` at `2284024`, both pushed; the arcade branch `feature/procedural_geometry_frame_stages` (`27e5640`, the Engine Stats line and launch options for ProceduralGeometry, plus the extension pin it needed to build) stays apart for a fork PR of its own; the headset holds a presenter example built from `fe14609f` with the toggle remembered on, and ProceduralGeometry built from `9f3977eb`. Closed today: the lead (fixed 22 ms right, adaptive kept off), the late frames (the RealityKit route's, absorbed by the second refresh), the cut and the band (at the coverage floor), the seam's headset check (three bugs found by way of ProceduralGeometry). Left in the plan: the picture items (the two-layer grid for the halo and the curved silhouettes, a relative tear threshold, TAA), stage 4 (Personas in CoolSaber, a second Vision Pro), stage 5 (template, docs, the editor's preview), and the upstream PRs (the seam first, carrying `0b946f03`, `9415dcf3`, `90994806` and `9f3977eb`; the fork PR descriptions still quote Debug numbers). - **MSAA+TAA on the headset (2026-10-19, 15:43): the edges of the cubes and the sphere look the same.** Not a mode ignored: `TemporalAntiAliasing` keeps a history per eye, `applyJitter()` puts the frame's sub-texel jitter into each eye's projection after `projectionDidChange()` clears the old one, and the screen size comes from the rate map under foveation. What the temporal pass can do is average the engine's edge into a ramp about one texel wide and stop the quarter-step crawl under motion; what it cannot do is add texels, and a one-texel ramp is still shown 2.6 display pixels wide. So the levers left for the edges are the texel density (the render scale, in its own session now that the two-refresh pipeline pays for it) and, for curved silhouettes and the halo, the two-layer grid. TAA's own cost on the device was not read. - **The render scale, built and merged (2026-10-19, 16:20; engine `feature/xr_realitykit_presenter` fast-forwarded to `ca92ed1e`, example `feature/realitykit_presenter_example` to `bf59a7f`, both pushed; 95 tests).** Built in its own session. `XRRealityKitPresenterConfiguration.renderScale` (1 by default, clamped 0.5 to 2, chosen for the next space opened) sizes the eye textures, the depth pair, the kernel's scratch and, through the rate map, the physical budget so the map's screen and the centre density gain the same factor; the grids keep their vertex counts (a uniform grid's cell grows with the texels; a foveated grid's cell is `foveatedCellTexels` times the scale snapped to a divisor of the 32-texel tile: 8 becomes 12, which snaps to 16, the map's grid 181 × 172 for the scaled budget against 241 × 229; 4 would keep the dense grid); scale 1 is byte-identical. `centreTexelsPerDegree` on the configuration and the presenter: 12.9 at scale 1, 19.3 at 1.5, 17.6 and 26.5 through the map, against the display's some 34; the settings line names it. On the Mac (M4 Max, the test scene, two sets) the GPU span goes 2.76 → 5.5 ms for 2.25× the pixels, the eye textures 27 → 62 MB per set. The example: "Render scale 1.5×" in the next-space row, `--render-scale 1.5`. The headset judgement: the cubes' and the sphere's edges with the toggle on, the GPU and the late count on the numbers line under the two-refresh pipeline. +- **Headset reading of the render scale at 1.5 (2026-10-19, 16:29, Release, engine `ca92ed1e`): "maybe a little improvement on the sphere, not on the cubes, and not much of a difference".** As the numbers say: 19.3 texels per degree against the display's 34 shows a texel step 1.8 pixels wide instead of 2.6, a third less, not under a pixel. The two ways to the display's density: the rate map with the scale (26.5 texels per degree at the centre, 1.3 pixels per texel, about 9 ms of GPU on the Mac's ratio), or a scale of 2 (25.8 uniform, 35 through the map, four times the pixels, about 16 ms, at the two-refresh budget's edge). Whether the rate map was on for this reading is to be confirmed; the next reading is both on. If the edges stay at the cubes' with the centre at the display's density, what remains is not the magnification but the chain after the texture: RealityKit's bilinear sample of the texel, the compositor's resample, and the band's blend of the partial texels, each a second resampling of the engine's one-texel ramp. From f458399c678d1a721fd1d3b6b368e995060b4475 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 16:31:16 +0200 Subject: [PATCH 60/72] [Docs] XRRealityKitPresenter plan: the density reading was with the map on; the cubes' fault is not the magnification --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 4040206c1..2bccde2a4 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -252,3 +252,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **MSAA+TAA on the headset (2026-10-19, 15:43): the edges of the cubes and the sphere look the same.** Not a mode ignored: `TemporalAntiAliasing` keeps a history per eye, `applyJitter()` puts the frame's sub-texel jitter into each eye's projection after `projectionDidChange()` clears the old one, and the screen size comes from the rate map under foveation. What the temporal pass can do is average the engine's edge into a ramp about one texel wide and stop the quarter-step crawl under motion; what it cannot do is add texels, and a one-texel ramp is still shown 2.6 display pixels wide. So the levers left for the edges are the texel density (the render scale, in its own session now that the two-refresh pipeline pays for it) and, for curved silhouettes and the halo, the two-layer grid. TAA's own cost on the device was not read. - **The render scale, built and merged (2026-10-19, 16:20; engine `feature/xr_realitykit_presenter` fast-forwarded to `ca92ed1e`, example `feature/realitykit_presenter_example` to `bf59a7f`, both pushed; 95 tests).** Built in its own session. `XRRealityKitPresenterConfiguration.renderScale` (1 by default, clamped 0.5 to 2, chosen for the next space opened) sizes the eye textures, the depth pair, the kernel's scratch and, through the rate map, the physical budget so the map's screen and the centre density gain the same factor; the grids keep their vertex counts (a uniform grid's cell grows with the texels; a foveated grid's cell is `foveatedCellTexels` times the scale snapped to a divisor of the 32-texel tile: 8 becomes 12, which snaps to 16, the map's grid 181 × 172 for the scaled budget against 241 × 229; 4 would keep the dense grid); scale 1 is byte-identical. `centreTexelsPerDegree` on the configuration and the presenter: 12.9 at scale 1, 19.3 at 1.5, 17.6 and 26.5 through the map, against the display's some 34; the settings line names it. On the Mac (M4 Max, the test scene, two sets) the GPU span goes 2.76 → 5.5 ms for 2.25× the pixels, the eye textures 27 → 62 MB per set. The example: "Render scale 1.5×" in the next-space row, `--render-scale 1.5`. The headset judgement: the cubes' and the sphere's edges with the toggle on, the GPU and the late count on the numbers line under the two-refresh pipeline. - **Headset reading of the render scale at 1.5 (2026-10-19, 16:29, Release, engine `ca92ed1e`): "maybe a little improvement on the sphere, not on the cubes, and not much of a difference".** As the numbers say: 19.3 texels per degree against the display's 34 shows a texel step 1.8 pixels wide instead of 2.6, a third less, not under a pixel. The two ways to the display's density: the rate map with the scale (26.5 texels per degree at the centre, 1.3 pixels per texel, about 9 ms of GPU on the Mac's ratio), or a scale of 2 (25.8 uniform, 35 through the map, four times the pixels, about 16 ms, at the two-refresh budget's edge). Whether the rate map was on for this reading is to be confirmed; the next reading is both on. If the edges stay at the cubes' with the centre at the display's density, what remains is not the magnification but the chain after the texture: RealityKit's bilinear sample of the texel, the compositor's resample, and the band's blend of the partial texels, each a second resampling of the engine's one-texel ramp. +- **Correction (2026-10-19, 16:31): the reading above was taken with the rate map AND the scale on, 26.5 texels per degree at the centre, 1.3 display pixels per texel.** So doubling the density changed the sphere a little and the cubes not at all: the cubes' edge fault is not the magnification of the engine's texel steps. It survives at the display's density, which leaves what is after the texture (RealityKit's bilinear sample, the compositor's resample, the band's blend) or what is at the grid's scale and does not move with the render scale (the silhouette cells, the skirt, the tear). The morning's screenshot measured the cube-top edge at 0.66 capture pixels rms of wobble with 22 to 36 pixel periods; the same measurement on a screenshot taken with both on tells the two apart. From 8068946cf278fae6bef4c2095b2a80b16392f642 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 16:37:08 +0200 Subject: [PATCH 61/72] [Docs] XRRealityKitPresenter plan: the cubes' edge wobble is independent of the texel density, measured; the measurement scripts kept with the note --- docs/proposals/XRRealityKitPresenter.md | 1 + .../XRRealityKitPresenter/tools/cube_tops2.py | 45 +++++++++ .../tools/edge_measure.py | 99 +++++++++++++++++++ 3 files changed, 145 insertions(+) create mode 100644 docs/proposals/XRRealityKitPresenter/tools/cube_tops2.py create mode 100644 docs/proposals/XRRealityKitPresenter/tools/edge_measure.py diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 2bccde2a4..4ba82a603 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -253,3 +253,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The render scale, built and merged (2026-10-19, 16:20; engine `feature/xr_realitykit_presenter` fast-forwarded to `ca92ed1e`, example `feature/realitykit_presenter_example` to `bf59a7f`, both pushed; 95 tests).** Built in its own session. `XRRealityKitPresenterConfiguration.renderScale` (1 by default, clamped 0.5 to 2, chosen for the next space opened) sizes the eye textures, the depth pair, the kernel's scratch and, through the rate map, the physical budget so the map's screen and the centre density gain the same factor; the grids keep their vertex counts (a uniform grid's cell grows with the texels; a foveated grid's cell is `foveatedCellTexels` times the scale snapped to a divisor of the 32-texel tile: 8 becomes 12, which snaps to 16, the map's grid 181 × 172 for the scaled budget against 241 × 229; 4 would keep the dense grid); scale 1 is byte-identical. `centreTexelsPerDegree` on the configuration and the presenter: 12.9 at scale 1, 19.3 at 1.5, 17.6 and 26.5 through the map, against the display's some 34; the settings line names it. On the Mac (M4 Max, the test scene, two sets) the GPU span goes 2.76 → 5.5 ms for 2.25× the pixels, the eye textures 27 → 62 MB per set. The example: "Render scale 1.5×" in the next-space row, `--render-scale 1.5`. The headset judgement: the cubes' and the sphere's edges with the toggle on, the GPU and the late count on the numbers line under the two-refresh pipeline. - **Headset reading of the render scale at 1.5 (2026-10-19, 16:29, Release, engine `ca92ed1e`): "maybe a little improvement on the sphere, not on the cubes, and not much of a difference".** As the numbers say: 19.3 texels per degree against the display's 34 shows a texel step 1.8 pixels wide instead of 2.6, a third less, not under a pixel. The two ways to the display's density: the rate map with the scale (26.5 texels per degree at the centre, 1.3 pixels per texel, about 9 ms of GPU on the Mac's ratio), or a scale of 2 (25.8 uniform, 35 through the map, four times the pixels, about 16 ms, at the two-refresh budget's edge). Whether the rate map was on for this reading is to be confirmed; the next reading is both on. If the edges stay at the cubes' with the centre at the display's density, what remains is not the magnification but the chain after the texture: RealityKit's bilinear sample of the texel, the compositor's resample, and the band's blend of the partial texels, each a second resampling of the engine's one-texel ramp. - **Correction (2026-10-19, 16:31): the reading above was taken with the rate map AND the scale on, 26.5 texels per degree at the centre, 1.3 display pixels per texel.** So doubling the density changed the sphere a little and the cubes not at all: the cubes' edge fault is not the magnification of the engine's texel steps. It survives at the display's density, which leaves what is after the texture (RealityKit's bilinear sample, the compositor's resample, the band's blend) or what is at the grid's scale and does not move with the render scale (the silhouette cells, the skirt, the tear). The morning's screenshot measured the cube-top edge at 0.66 capture pixels rms of wobble with 22 to 36 pixel periods; the same measurement on a screenshot taken with both on tells the two apart. +- **Measured: the cubes' edge does not move with the texel density (2026-10-19, 16:37).** Two headset captures of the same framing, this morning's (uniform 1920 × 1824, 12.9 texels per degree) and one now with the rate map and the render scale (26.5 at the centre), measured with the same scripts (`XRRealityKitPresenter/tools/edge_measure.py` for the slab and the sphere, `cube_tops2.py` for the far edge of each cube's top face: the topmost pale-blue pixel per column over the middle 70 % of the cube, a line fit, the residual and its periodogram). Slab's lower edge: 0.32 → 0.30 capture px mean deviation, no step of two pixels, at the floor both times. Cubes' top edges, rms about a line: 0.35 / 0.52 / 0.66 px this morning, 0.51 / 0.44 / 0.41 px now; dominant wobble periods 38 / 29 / 40 px then, 22 / 28 / 28 px now; the amplitude unchanged by a doubling of the density, the sphere's outline 0.9 px from a circle now (the morning's sphere figure from this script was a detection failure; the panel's 0.31 px rms stands). So the cubes' fault is not the engine's texel steps: whatever wobbles the top edge by half a capture pixel with 20 to 40 pixel periods lives after the texture, at the display's scale: the grid's cells (0.6° at the centre, about 10 capture px, so the periods are two to four cells), RealityKit's rasterization of the core's hard cut (no anti-aliasing of an alpha-tested edge unless RealityKit's own pass has it), or its sampling and the compositor's resample. The study that tells them apart is on the Mac: the RealityRenderer harness at display-like magnification (1 texel = 2.6 picture px at scale 1, 1.3 at the map with 1.5), the cube-top edge measured in picture pixels across the cut at 0.9 / 0.97 / 0.99, the band on and off, cells of 8 and 4 texels, the grid 256 and 512: the knob that moves the picture-pixel wobble names the fault. Captures are a weak instrument for this (a 1920 × 1080 recompression of a 3660 × 3200 display), which is why the Mac study comes before another device build. diff --git a/docs/proposals/XRRealityKitPresenter/tools/cube_tops2.py b/docs/proposals/XRRealityKitPresenter/tools/cube_tops2.py new file mode 100644 index 000000000..ea0b7a20d --- /dev/null +++ b/docs/proposals/XRRealityKitPresenter/tools/cube_tops2.py @@ -0,0 +1,45 @@ +"""The cubes as their own pale-blue components above the slab: for each, the topmost pixel per +column over the middle 70 % of its width (the far edge of the top face against the room), a line +fit, the residual rms and peak-to-peak, the steps, the dominant wobble period; and a crop of the +middle cube's top edge, upscaled 4x, saved beside the capture for a look.""" +import sys, cv2, numpy as np +from edge_measure import slab_mask + +def run(path): + img = cv2.imread(path) + mask = slab_mask(img) + n, labels, stats, _ = cv2.connectedComponentsWithStats(mask) + slab = max(range(1, n), key=lambda i: stats[i, cv2.CC_STAT_WIDTH]) + sx, sy, sw, sh, _ = stats[slab] + cubes = [i for i in range(1, n) if i != slab and stats[i, cv2.CC_STAT_AREA] > 4000 + and 80 < stats[i, cv2.CC_STAT_WIDTH] < 400 and stats[i, cv2.CC_STAT_TOP] < sy + sh + and 0.6 < stats[i, cv2.CC_STAT_WIDTH] / stats[i, cv2.CC_STAT_HEIGHT] < 1.8] + cubes.sort(key=lambda i: stats[i, cv2.CC_STAT_LEFT]) + print(f"{path.split('/')[-1]}: slab {sw}x{sh} at ({sx},{sy}); {len(cubes)} cube component(s): " + + ", ".join(f"{stats[i, cv2.CC_STAT_WIDTH]}x{stats[i, cv2.CC_STAT_HEIGHT]} at ({stats[i, cv2.CC_STAT_LEFT]},{stats[i, cv2.CC_STAT_TOP]})" for i in cubes)) + for k, i in enumerate(cubes): + x, y, w, h, _ = stats[i] + comp = labels == i + m = int(w * 0.15) + xs, ys = [], [] + for cx in range(x + m, x + w - m): + col = np.where(comp[:, cx])[0] + if len(col): xs.append(cx); ys.append(col.min()) + xs = np.array(xs); ys = np.array(ys, dtype=float) + if len(xs) < 20: continue + p = np.polyfit(xs, ys, 1); r = ys - np.polyval(p, xs) + steps = np.abs(np.diff(ys)) + f = np.fft.rfft(r - r.mean()); freqs = np.fft.rfftfreq(len(r), 1.0); power = np.abs(f) ** 2 + sel = (freqs > 1 / 120) & (freqs < 1 / 6) + period = 1 / freqs[sel][np.argmax(power[sel])] if sel.any() else float('nan') + print(f" cube {k + 1}: top edge over {len(xs)} columns, slope {p[0]:+.3f}; residual rms {r.std():.2f} px, p-p {r.max() - r.min():.2f} px; " + f"steps >= 2 px: {(steps >= 2).sum()}, largest {int(steps.max())} px; dominant wobble period {period:.0f} px") + if k == 1: + # The middle cube's top edge: a strip 24 px tall around the edge, the middle 70 % of the width, upscaled 4x. + y0 = int(np.polyval(p, xs).min()) - 10; y1 = y0 + 24 + crop = img[max(0, y0):y1, x + m:x + w - m] + crop = cv2.resize(crop, None, fx=4, fy=4, interpolation=cv2.INTER_NEAREST) + out = path.replace('.png', '-cubetop.png'); cv2.imwrite(out, crop); print(f" crop: {out} ({crop.shape[1]}x{crop.shape[0]})") + +for p in sys.argv[1:]: + run(p) diff --git a/docs/proposals/XRRealityKitPresenter/tools/edge_measure.py b/docs/proposals/XRRealityKitPresenter/tools/edge_measure.py new file mode 100644 index 000000000..b362bb923 --- /dev/null +++ b/docs/proposals/XRRealityKitPresenter/tools/edge_measure.py @@ -0,0 +1,99 @@ +"""Measures the silhouettes of the example's slab in a headset screenshot: the lower edge of the +slab (light blue against the room) and the sphere's outline. Prints, for the slab's lower edge, the +boundary's deviation from a fitted straight line (mean and max, in pixels) and the number of steps +(jumps of two pixels or more between neighbouring columns); for the sphere, the deviation of its +outline from a fitted circle.""" +import sys, cv2, numpy as np + +def slab_mask(img): + hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) + # The engine's objects are a pale blue: low saturation, high value, hue around blue. + h, s, v = cv2.split(hsv) + mask = ((h > 90) & (h < 135) & (s > 15) & (s < 190) & (v > 90)).astype(np.uint8) * 255 + mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8)) + return mask + +def lower_edge(mask, x0, x1): + ys = [] + xs = [] + for x in range(x0, x1): + column = np.where(mask[:, x] > 0)[0] + if len(column) == 0: continue + ys.append(column.max()); xs.append(x) + return np.array(xs), np.array(ys) + +def right_edge(mask, y0, y1): + xs = []; ys = [] + for y in range(y0, y1): + row = np.where(mask[y, :] > 0)[0] + if len(row) == 0: continue + xs.append(row.max()); ys.append(y) + return np.array(xs), np.array(ys) + +def measure(path): + img = cv2.imread(path) + mask = slab_mask(img) + n, labels, stats, _ = cv2.connectedComponentsWithStats(mask) + # The slab: the widest component. + best = max(range(1, n), key=lambda i: stats[i, cv2.CC_STAT_WIDTH]) + x, y, w, h, area = stats[best] + slab = (labels == best).astype(np.uint8) * 255 + # The lower edge over the middle 80 % of the slab's width. + xs, ys = lower_edge(slab, x + w // 10, x + w - w // 10) + a, b = np.polyfit(xs, ys, 1) + residual = ys - (a * xs + b) + steps = np.abs(np.diff(ys)) + print(f"{path.split('/')[-1]}: slab {w}x{h} px at ({x},{y}); lower edge over {len(xs)} columns: " + f"deviation from a line mean {np.abs(residual).mean():.2f} px, max {np.abs(residual).max():.2f} px; " + f"steps >= 2 px: {(steps >= 2).sum()}, largest step {steps.max()} px") + # The right edge over the middle 80 % of the slab's height (the edge that descends to the right). + rx, ry = right_edge(slab, y + h // 10, y + h - h // 10) + if len(rx) > 4: + a2, b2 = np.polyfit(ry, rx, 1) + res2 = rx - (a2 * ry + b2) + steps2 = np.abs(np.diff(rx)) + print(f" right edge over {len(rx)} rows: deviation from a line mean {np.abs(res2).mean():.2f} px, max {np.abs(res2).max():.2f} px; " + f"steps >= 2 px: {(steps2 >= 2).sum()}, largest step {steps2.max()} px") + # The sphere: the roundest small component above the slab. + # The sphere: small, about as wide as tall, well above the slab, the roundest of those. + cands = [i for i in range(1, n) if i != best and 400 < stats[i, cv2.CC_STAT_AREA] < 40000 + and 0.7 < stats[i, cv2.CC_STAT_WIDTH] / max(1, stats[i, cv2.CC_STAT_HEIGHT]) < 1.4 + and stats[i, cv2.CC_STAT_TOP] + stats[i, cv2.CC_STAT_HEIGHT] < y - 60] + def circularity(i): + comp = (labels == i).astype(np.uint8) + contours, _ = cv2.findContours(comp, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE) + c = max(contours, key=cv2.contourArea) + per = cv2.arcLength(c, True) + return 4 * np.pi * cv2.contourArea(c) / (per * per) if per > 0 else 0 + if cands: + i = max(cands, key=circularity) + # The blue mask loses the sphere's bright, desaturated top: within a window about the + # candidate, take bright low-saturation pixels too, then the largest component. + sx, sy, sw, sh = stats[i, cv2.CC_STAT_LEFT], stats[i, cv2.CC_STAT_TOP], stats[i, cv2.CC_STAT_WIDTH], stats[i, cv2.CC_STAT_HEIGHT] + pad = int(max(sw, sh) * 0.4) + wx0, wy0 = max(0, sx - pad), max(0, sy - pad) + wx1, wy1 = min(img.shape[1], sx + sw + pad), min(img.shape[0], sy + sh + pad) + hsv = cv2.cvtColor(img[wy0:wy1, wx0:wx1], cv2.COLOR_BGR2HSV) + h_, s_, v_ = cv2.split(hsv) + window = (((h_ > 90) & (h_ < 135) & (v_ > 90)) | ((s_ < 30) & (v_ > 150))).astype(np.uint8) + window = cv2.morphologyEx(window, cv2.MORPH_CLOSE, np.ones((3, 3), np.uint8)) + nw, lw, sw_, _ = cv2.connectedComponentsWithStats(window) + # The component that contains the blue candidate's centre. + ccx, ccy = sx + sw // 2 - wx0, sy + sh // 2 - wy0 + k = lw[min(ccy, lw.shape[0] - 1), min(ccx, lw.shape[1] - 1)] + if k == 0: + k = max(range(1, nw), key=lambda j: sw_[j, cv2.CC_STAT_AREA]) + sphere = np.zeros(mask.shape, np.uint8) + sphere[wy0:wy1, wx0:wx1] = (lw == k).astype(np.uint8) + contours, _ = cv2.findContours(sphere, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE) + c = max(contours, key=cv2.contourArea) + cv2.imwrite(path.rsplit('.', 1)[0] + "-sphere.png", sphere * 255) + (cx, cy), r = cv2.minEnclosingCircle(c) + pts = c.reshape(-1, 2).astype(np.float64) + d = np.sqrt(((pts - [cx, cy]) ** 2).sum(1)) + # A circle fitted by least squares on the contour radius. + print(f" sphere: radius {d.mean():.1f} px, outline deviation from a circle mean {np.abs(d - d.mean()).mean():.2f} px, max {np.abs(d - d.mean()).max():.2f} px") + cv2.imwrite(path.rsplit('.', 1)[0] + "-mask.png", slab) + +for p in sys.argv[1:]: + measure(p) From 69e37456a751f13c150d125a142b67bb8fd5787f Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 17:15:16 +0200 Subject: [PATCH 62/72] [Docs] XRRealityKitPresenter plan: the Mac study finds no presenter knob behind the display-scale wobble; a native control cube decides --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 4ba82a603..ca1cceed3 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -254,3 +254,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Headset reading of the render scale at 1.5 (2026-10-19, 16:29, Release, engine `ca92ed1e`): "maybe a little improvement on the sphere, not on the cubes, and not much of a difference".** As the numbers say: 19.3 texels per degree against the display's 34 shows a texel step 1.8 pixels wide instead of 2.6, a third less, not under a pixel. The two ways to the display's density: the rate map with the scale (26.5 texels per degree at the centre, 1.3 pixels per texel, about 9 ms of GPU on the Mac's ratio), or a scale of 2 (25.8 uniform, 35 through the map, four times the pixels, about 16 ms, at the two-refresh budget's edge). Whether the rate map was on for this reading is to be confirmed; the next reading is both on. If the edges stay at the cubes' with the centre at the display's density, what remains is not the magnification but the chain after the texture: RealityKit's bilinear sample of the texel, the compositor's resample, and the band's blend of the partial texels, each a second resampling of the engine's one-texel ramp. - **Correction (2026-10-19, 16:31): the reading above was taken with the rate map AND the scale on, 26.5 texels per degree at the centre, 1.3 display pixels per texel.** So doubling the density changed the sphere a little and the cubes not at all: the cubes' edge fault is not the magnification of the engine's texel steps. It survives at the display's density, which leaves what is after the texture (RealityKit's bilinear sample, the compositor's resample, the band's blend) or what is at the grid's scale and does not move with the render scale (the silhouette cells, the skirt, the tear). The morning's screenshot measured the cube-top edge at 0.66 capture pixels rms of wobble with 22 to 36 pixel periods; the same measurement on a screenshot taken with both on tells the two apart. - **Measured: the cubes' edge does not move with the texel density (2026-10-19, 16:37).** Two headset captures of the same framing, this morning's (uniform 1920 × 1824, 12.9 texels per degree) and one now with the rate map and the render scale (26.5 at the centre), measured with the same scripts (`XRRealityKitPresenter/tools/edge_measure.py` for the slab and the sphere, `cube_tops2.py` for the far edge of each cube's top face: the topmost pale-blue pixel per column over the middle 70 % of the cube, a line fit, the residual and its periodogram). Slab's lower edge: 0.32 → 0.30 capture px mean deviation, no step of two pixels, at the floor both times. Cubes' top edges, rms about a line: 0.35 / 0.52 / 0.66 px this morning, 0.51 / 0.44 / 0.41 px now; dominant wobble periods 38 / 29 / 40 px then, 22 / 28 / 28 px now; the amplitude unchanged by a doubling of the density, the sphere's outline 0.9 px from a circle now (the morning's sphere figure from this script was a detection failure; the panel's 0.31 px rms stands). So the cubes' fault is not the engine's texel steps: whatever wobbles the top edge by half a capture pixel with 20 to 40 pixel periods lives after the texture, at the display's scale: the grid's cells (0.6° at the centre, about 10 capture px, so the periods are two to four cells), RealityKit's rasterization of the core's hard cut (no anti-aliasing of an alpha-tested edge unless RealityKit's own pass has it), or its sampling and the compositor's resample. The study that tells them apart is on the Mac: the RealityRenderer harness at display-like magnification (1 texel = 2.6 picture px at scale 1, 1.3 at the map with 1.5), the cube-top edge measured in picture pixels across the cut at 0.9 / 0.97 / 0.99, the band on and off, cells of 8 and 4 texels, the grid 256 and 512: the knob that moves the picture-pixel wobble names the fault. Captures are a weak instrument for this (a 1920 × 1080 recompression of a 3660 × 3200 display), which is why the Mac study comes before another device build. +- **The Mac study of the display-scale edge (2026-10-19, 17:15; branch `study/xr_presenter_edge_wobble` `7627fe83` on `ca92ed1e`, pushed, tests only: 99 in the filter, three study tests behind `XR_PRESENTER_WOBBLE_STUDY`, an always-on guard).** `XRPresenterEdgeWobbleTests` draws the straightness scene at 33.5 picture pixels per degree whatever the eye's density (2.6 px per texel for the default eye, 1.26 for the map at scale 1.5) and reads the cubes' top and side edges and the sphere's outline in picture pixels, by a mask as the captures were read, by the one-half crossing and by the coverage; the floor is the engine's eye resampled bilinearly into the same picture. **Top edges by the crossing, px rms**: the floor 0.19; shipped (cut 0.9, band) 0.20; the cut alone at 0.9 / 0.97 / 0.99 without the band 0.60 / 0.72 / 0.77 (exactly a pixel-centre alpha test, and RealityRenderer's own MSAA 4× leaves it there); cuts 0.97 / 0.99 with the band, and no core with a whole-grid band, 0.20; the view 3 mm off the drawing eye 0.19; the uniform 512 × 480 grid 0.44 and skirt 0 0.43 (worse: hard unblended stretches and a notch, the cause open); render scale 1.5 floor / shipped 0.13 / 0.14; the map's cells of 8 0.16 / 0.15, cells of 4 0.15 but the sides 0.30; map with scale 1.5 at cells 16 / 8 0.12 / 0.11 against a floor of 0.11; eight poses a tenth of a texel apart averaged: 0.21 → 0.05 (motion averages the texel staircase out; nothing is locked to cells or to display pixels). **Verdict**: no knob of the presenter moves a display-scale wobble at the shipped configuration; the chain equals the resampled texture to a hundredth of a pixel at every density, the cut's height does not matter with the band, and the raster candidate (the hard cut as a pixel-centre alpha test, 0.6 px) is real but fully covered by the band. The headset's density-independent 0.4 to 0.6 capture px (0.8 to 1.1 display px) is therefore not in the presenter as the Mac can run it: it lives in the device's own RealityKit render and compositor, or in the captures' 1.9× recompression with chroma subsampling read through a colour mask. **The experiment that decides it costs the engine nothing**: a native RealityKit cube beside the engine's in the example's space, captured in the same frame and measured with the same script. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside a wavy engine cube names the presenter's materials under the device's foveated render, and the thing to try then is a core that blends with depth kept (the no-core whole-grid-band row is the same picture on the Mac, at about 11 ms of device GPU as measured earlier). Side finding: cells under 4 texels (the 512 grid, the map's cells of 4) are worse than the shipped cell, with hard unblended stretches and a notch, and a smaller skirt does not mend them; the cause is open. From 93f92c7fc7348b606ce8e884d5e9d4e32d8488ac Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 17:34:43 +0200 Subject: [PATCH 63/72] [Docs] XRRealityKitPresenter plan: the control cube, built and merged; the capture to take --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index ca1cceed3..1af582644 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -255,3 +255,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Correction (2026-10-19, 16:31): the reading above was taken with the rate map AND the scale on, 26.5 texels per degree at the centre, 1.3 display pixels per texel.** So doubling the density changed the sphere a little and the cubes not at all: the cubes' edge fault is not the magnification of the engine's texel steps. It survives at the display's density, which leaves what is after the texture (RealityKit's bilinear sample, the compositor's resample, the band's blend) or what is at the grid's scale and does not move with the render scale (the silhouette cells, the skirt, the tear). The morning's screenshot measured the cube-top edge at 0.66 capture pixels rms of wobble with 22 to 36 pixel periods; the same measurement on a screenshot taken with both on tells the two apart. - **Measured: the cubes' edge does not move with the texel density (2026-10-19, 16:37).** Two headset captures of the same framing, this morning's (uniform 1920 × 1824, 12.9 texels per degree) and one now with the rate map and the render scale (26.5 at the centre), measured with the same scripts (`XRRealityKitPresenter/tools/edge_measure.py` for the slab and the sphere, `cube_tops2.py` for the far edge of each cube's top face: the topmost pale-blue pixel per column over the middle 70 % of the cube, a line fit, the residual and its periodogram). Slab's lower edge: 0.32 → 0.30 capture px mean deviation, no step of two pixels, at the floor both times. Cubes' top edges, rms about a line: 0.35 / 0.52 / 0.66 px this morning, 0.51 / 0.44 / 0.41 px now; dominant wobble periods 38 / 29 / 40 px then, 22 / 28 / 28 px now; the amplitude unchanged by a doubling of the density, the sphere's outline 0.9 px from a circle now (the morning's sphere figure from this script was a detection failure; the panel's 0.31 px rms stands). So the cubes' fault is not the engine's texel steps: whatever wobbles the top edge by half a capture pixel with 20 to 40 pixel periods lives after the texture, at the display's scale: the grid's cells (0.6° at the centre, about 10 capture px, so the periods are two to four cells), RealityKit's rasterization of the core's hard cut (no anti-aliasing of an alpha-tested edge unless RealityKit's own pass has it), or its sampling and the compositor's resample. The study that tells them apart is on the Mac: the RealityRenderer harness at display-like magnification (1 texel = 2.6 picture px at scale 1, 1.3 at the map with 1.5), the cube-top edge measured in picture pixels across the cut at 0.9 / 0.97 / 0.99, the band on and off, cells of 8 and 4 texels, the grid 256 and 512: the knob that moves the picture-pixel wobble names the fault. Captures are a weak instrument for this (a 1920 × 1080 recompression of a 3660 × 3200 display), which is why the Mac study comes before another device build. - **The Mac study of the display-scale edge (2026-10-19, 17:15; branch `study/xr_presenter_edge_wobble` `7627fe83` on `ca92ed1e`, pushed, tests only: 99 in the filter, three study tests behind `XR_PRESENTER_WOBBLE_STUDY`, an always-on guard).** `XRPresenterEdgeWobbleTests` draws the straightness scene at 33.5 picture pixels per degree whatever the eye's density (2.6 px per texel for the default eye, 1.26 for the map at scale 1.5) and reads the cubes' top and side edges and the sphere's outline in picture pixels, by a mask as the captures were read, by the one-half crossing and by the coverage; the floor is the engine's eye resampled bilinearly into the same picture. **Top edges by the crossing, px rms**: the floor 0.19; shipped (cut 0.9, band) 0.20; the cut alone at 0.9 / 0.97 / 0.99 without the band 0.60 / 0.72 / 0.77 (exactly a pixel-centre alpha test, and RealityRenderer's own MSAA 4× leaves it there); cuts 0.97 / 0.99 with the band, and no core with a whole-grid band, 0.20; the view 3 mm off the drawing eye 0.19; the uniform 512 × 480 grid 0.44 and skirt 0 0.43 (worse: hard unblended stretches and a notch, the cause open); render scale 1.5 floor / shipped 0.13 / 0.14; the map's cells of 8 0.16 / 0.15, cells of 4 0.15 but the sides 0.30; map with scale 1.5 at cells 16 / 8 0.12 / 0.11 against a floor of 0.11; eight poses a tenth of a texel apart averaged: 0.21 → 0.05 (motion averages the texel staircase out; nothing is locked to cells or to display pixels). **Verdict**: no knob of the presenter moves a display-scale wobble at the shipped configuration; the chain equals the resampled texture to a hundredth of a pixel at every density, the cut's height does not matter with the band, and the raster candidate (the hard cut as a pixel-centre alpha test, 0.6 px) is real but fully covered by the band. The headset's density-independent 0.4 to 0.6 capture px (0.8 to 1.1 display px) is therefore not in the presenter as the Mac can run it: it lives in the device's own RealityKit render and compositor, or in the captures' 1.9× recompression with chroma subsampling read through a colour mask. **The experiment that decides it costs the engine nothing**: a native RealityKit cube beside the engine's in the example's space, captured in the same frame and measured with the same script. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside a wavy engine cube names the presenter's materials under the device's foveated render, and the thing to try then is a core that blends with depth kept (the no-core whole-grid-band row is the same picture on the Mac, at about 11 ms of device GPU as measured earlier). Side finding: cells under 4 texels (the 512 grid, the map's cells of 4) are worse than the shipped cell, with hard unblended stretches and a notch, and a smaller skirt does not mend them; the cause is open. +- **The control cube, built and merged (2026-10-19, 17:34; example `feature/realitykit_presenter_example` fast-forwarded to `e5b4dc4`, pushed; example only).** Built in its own session: `ControlCube.swift`, a lit pale-blue RealityKit cube (`SimpleMaterial`, roughness 0.4) of the engine cubes' 0.2 m, one step to the right of the engine's three on the slab's plane (x 0.7, the same gap and height), no yaw so its far top edge is one straight line like the middle cube's, a child of the presenter's root beside the grid entities (world-locked like them, nothing the presenter manages); a live, remembered "Control cube" toggle, `--control-cube`, "control cube on" on the settings line. Release built and installed. The capture to take: both cubes in one frame, framed like the morning's, with the rate map and the scale as the user likes; `cube_tops2.py` on it reads the four top edges with one definition. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside wavy engine cubes names the presenter's materials under the device's render, and the next experiment is then a core that blends with depth kept. From 2b67c94fd235b7626d6301263d6fd49a6b57ac67 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 17:40:39 +0200 Subject: [PATCH 64/72] [Docs] XRRealityKitPresenter plan: the control cube clears the device and the captures; the presenter's chain is at fault --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 1af582644..a336087dd 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -256,3 +256,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Measured: the cubes' edge does not move with the texel density (2026-10-19, 16:37).** Two headset captures of the same framing, this morning's (uniform 1920 × 1824, 12.9 texels per degree) and one now with the rate map and the render scale (26.5 at the centre), measured with the same scripts (`XRRealityKitPresenter/tools/edge_measure.py` for the slab and the sphere, `cube_tops2.py` for the far edge of each cube's top face: the topmost pale-blue pixel per column over the middle 70 % of the cube, a line fit, the residual and its periodogram). Slab's lower edge: 0.32 → 0.30 capture px mean deviation, no step of two pixels, at the floor both times. Cubes' top edges, rms about a line: 0.35 / 0.52 / 0.66 px this morning, 0.51 / 0.44 / 0.41 px now; dominant wobble periods 38 / 29 / 40 px then, 22 / 28 / 28 px now; the amplitude unchanged by a doubling of the density, the sphere's outline 0.9 px from a circle now (the morning's sphere figure from this script was a detection failure; the panel's 0.31 px rms stands). So the cubes' fault is not the engine's texel steps: whatever wobbles the top edge by half a capture pixel with 20 to 40 pixel periods lives after the texture, at the display's scale: the grid's cells (0.6° at the centre, about 10 capture px, so the periods are two to four cells), RealityKit's rasterization of the core's hard cut (no anti-aliasing of an alpha-tested edge unless RealityKit's own pass has it), or its sampling and the compositor's resample. The study that tells them apart is on the Mac: the RealityRenderer harness at display-like magnification (1 texel = 2.6 picture px at scale 1, 1.3 at the map with 1.5), the cube-top edge measured in picture pixels across the cut at 0.9 / 0.97 / 0.99, the band on and off, cells of 8 and 4 texels, the grid 256 and 512: the knob that moves the picture-pixel wobble names the fault. Captures are a weak instrument for this (a 1920 × 1080 recompression of a 3660 × 3200 display), which is why the Mac study comes before another device build. - **The Mac study of the display-scale edge (2026-10-19, 17:15; branch `study/xr_presenter_edge_wobble` `7627fe83` on `ca92ed1e`, pushed, tests only: 99 in the filter, three study tests behind `XR_PRESENTER_WOBBLE_STUDY`, an always-on guard).** `XRPresenterEdgeWobbleTests` draws the straightness scene at 33.5 picture pixels per degree whatever the eye's density (2.6 px per texel for the default eye, 1.26 for the map at scale 1.5) and reads the cubes' top and side edges and the sphere's outline in picture pixels, by a mask as the captures were read, by the one-half crossing and by the coverage; the floor is the engine's eye resampled bilinearly into the same picture. **Top edges by the crossing, px rms**: the floor 0.19; shipped (cut 0.9, band) 0.20; the cut alone at 0.9 / 0.97 / 0.99 without the band 0.60 / 0.72 / 0.77 (exactly a pixel-centre alpha test, and RealityRenderer's own MSAA 4× leaves it there); cuts 0.97 / 0.99 with the band, and no core with a whole-grid band, 0.20; the view 3 mm off the drawing eye 0.19; the uniform 512 × 480 grid 0.44 and skirt 0 0.43 (worse: hard unblended stretches and a notch, the cause open); render scale 1.5 floor / shipped 0.13 / 0.14; the map's cells of 8 0.16 / 0.15, cells of 4 0.15 but the sides 0.30; map with scale 1.5 at cells 16 / 8 0.12 / 0.11 against a floor of 0.11; eight poses a tenth of a texel apart averaged: 0.21 → 0.05 (motion averages the texel staircase out; nothing is locked to cells or to display pixels). **Verdict**: no knob of the presenter moves a display-scale wobble at the shipped configuration; the chain equals the resampled texture to a hundredth of a pixel at every density, the cut's height does not matter with the band, and the raster candidate (the hard cut as a pixel-centre alpha test, 0.6 px) is real but fully covered by the band. The headset's density-independent 0.4 to 0.6 capture px (0.8 to 1.1 display px) is therefore not in the presenter as the Mac can run it: it lives in the device's own RealityKit render and compositor, or in the captures' 1.9× recompression with chroma subsampling read through a colour mask. **The experiment that decides it costs the engine nothing**: a native RealityKit cube beside the engine's in the example's space, captured in the same frame and measured with the same script. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside a wavy engine cube names the presenter's materials under the device's foveated render, and the thing to try then is a core that blends with depth kept (the no-core whole-grid-band row is the same picture on the Mac, at about 11 ms of device GPU as measured earlier). Side finding: cells under 4 texels (the 512 grid, the map's cells of 4) are worse than the shipped cell, with hard unblended stretches and a notch, and a smaller skirt does not mend them; the cause is open. - **The control cube, built and merged (2026-10-19, 17:34; example `feature/realitykit_presenter_example` fast-forwarded to `e5b4dc4`, pushed; example only).** Built in its own session: `ControlCube.swift`, a lit pale-blue RealityKit cube (`SimpleMaterial`, roughness 0.4) of the engine cubes' 0.2 m, one step to the right of the engine's three on the slab's plane (x 0.7, the same gap and height), no yaw so its far top edge is one straight line like the middle cube's, a child of the presenter's root beside the grid entities (world-locked like them, nothing the presenter manages); a live, remembered "Control cube" toggle, `--control-cube`, "control cube on" on the settings line. Release built and installed. The capture to take: both cubes in one frame, framed like the morning's, with the rate map and the scale as the user likes; `cube_tops2.py` on it reads the four top edges with one definition. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside wavy engine cubes names the presenter's materials under the device's render, and the next experiment is then a core that blends with depth kept. +- **The control cube decides (2026-10-19, 17:40, headset, Release, example `e5b4dc4`): the presenter's chain is at fault, not the device or the captures.** In one frame, the user sees the engine's cubes shimmer while the RealityKit cube beside them does not, and the control's edges "far better". The capture agrees: the control's top edge 0.35 px rms about a line (peak to peak 1.6, period 44 px) against the engine cubes' 0.82 / 0.64 / 0.55 px (peak to peak 2.5 to 3.3, periods 58 to 83 px), through the same device render and the same capture; the control's edge, drawn by RealityKit's own geometry pass with its anti-aliasing, is what the device can do, and the engine's edges through the presenter are twice as wavy and move. So the Mac study's candidates narrow to what the device renders differently from RealityRenderer on the Mac: the core's hard alpha cut rasterized without anti-aliasing (on the Mac the band covered it to the floor; on the device the band's blend and the cut's step apparently do not add up to a still edge), and the texel staircase marching under the head's motion (the "shimmer": the engine's texture changes every frame, the control's geometry does not). The next experiment, named by the study: a core that blends instead of cutting, with depth kept (`ShaderGraphMaterial.writesDepth`, so occlusion against Personas and RealityKit content stays), the whole grid through the band's blended graph and the band meshes off; on the Mac it is the same picture, on the device it removes the alpha test from the silhouette altogether. Its cost is the transparent pass for a full-screen mesh. If the shimmer stays with a blended core, it is the texture's own frame-to-frame change at the silhouette, and the remaining lever is temporal (the engine's TAA, judged "the same" for the static edge, to be judged for the shimmer). From a7dcccc9cd5ccf5e6d6e2da7e4a77e24a82c95d7 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:06:52 +0200 Subject: [PATCH 65/72] [Docs] XRRealityKitPresenter plan: the blended core, built and merged --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index a336087dd..e812ce5d2 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -257,3 +257,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The Mac study of the display-scale edge (2026-10-19, 17:15; branch `study/xr_presenter_edge_wobble` `7627fe83` on `ca92ed1e`, pushed, tests only: 99 in the filter, three study tests behind `XR_PRESENTER_WOBBLE_STUDY`, an always-on guard).** `XRPresenterEdgeWobbleTests` draws the straightness scene at 33.5 picture pixels per degree whatever the eye's density (2.6 px per texel for the default eye, 1.26 for the map at scale 1.5) and reads the cubes' top and side edges and the sphere's outline in picture pixels, by a mask as the captures were read, by the one-half crossing and by the coverage; the floor is the engine's eye resampled bilinearly into the same picture. **Top edges by the crossing, px rms**: the floor 0.19; shipped (cut 0.9, band) 0.20; the cut alone at 0.9 / 0.97 / 0.99 without the band 0.60 / 0.72 / 0.77 (exactly a pixel-centre alpha test, and RealityRenderer's own MSAA 4× leaves it there); cuts 0.97 / 0.99 with the band, and no core with a whole-grid band, 0.20; the view 3 mm off the drawing eye 0.19; the uniform 512 × 480 grid 0.44 and skirt 0 0.43 (worse: hard unblended stretches and a notch, the cause open); render scale 1.5 floor / shipped 0.13 / 0.14; the map's cells of 8 0.16 / 0.15, cells of 4 0.15 but the sides 0.30; map with scale 1.5 at cells 16 / 8 0.12 / 0.11 against a floor of 0.11; eight poses a tenth of a texel apart averaged: 0.21 → 0.05 (motion averages the texel staircase out; nothing is locked to cells or to display pixels). **Verdict**: no knob of the presenter moves a display-scale wobble at the shipped configuration; the chain equals the resampled texture to a hundredth of a pixel at every density, the cut's height does not matter with the band, and the raster candidate (the hard cut as a pixel-centre alpha test, 0.6 px) is real but fully covered by the band. The headset's density-independent 0.4 to 0.6 capture px (0.8 to 1.1 display px) is therefore not in the presenter as the Mac can run it: it lives in the device's own RealityKit render and compositor, or in the captures' 1.9× recompression with chroma subsampling read through a colour mask. **The experiment that decides it costs the engine nothing**: a native RealityKit cube beside the engine's in the example's space, captured in the same frame and measured with the same script. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside a wavy engine cube names the presenter's materials under the device's foveated render, and the thing to try then is a core that blends with depth kept (the no-core whole-grid-band row is the same picture on the Mac, at about 11 ms of device GPU as measured earlier). Side finding: cells under 4 texels (the 512 grid, the map's cells of 4) are worse than the shipped cell, with hard unblended stretches and a notch, and a smaller skirt does not mend them; the cause is open. - **The control cube, built and merged (2026-10-19, 17:34; example `feature/realitykit_presenter_example` fast-forwarded to `e5b4dc4`, pushed; example only).** Built in its own session: `ControlCube.swift`, a lit pale-blue RealityKit cube (`SimpleMaterial`, roughness 0.4) of the engine cubes' 0.2 m, one step to the right of the engine's three on the slab's plane (x 0.7, the same gap and height), no yaw so its far top edge is one straight line like the middle cube's, a child of the presenter's root beside the grid entities (world-locked like them, nothing the presenter manages); a live, remembered "Control cube" toggle, `--control-cube`, "control cube on" on the settings line. Release built and installed. The capture to take: both cubes in one frame, framed like the morning's, with the rate map and the scale as the user likes; `cube_tops2.py` on it reads the four top edges with one definition. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside wavy engine cubes names the presenter's materials under the device's render, and the next experiment is then a core that blends with depth kept. - **The control cube decides (2026-10-19, 17:40, headset, Release, example `e5b4dc4`): the presenter's chain is at fault, not the device or the captures.** In one frame, the user sees the engine's cubes shimmer while the RealityKit cube beside them does not, and the control's edges "far better". The capture agrees: the control's top edge 0.35 px rms about a line (peak to peak 1.6, period 44 px) against the engine cubes' 0.82 / 0.64 / 0.55 px (peak to peak 2.5 to 3.3, periods 58 to 83 px), through the same device render and the same capture; the control's edge, drawn by RealityKit's own geometry pass with its anti-aliasing, is what the device can do, and the engine's edges through the presenter are twice as wavy and move. So the Mac study's candidates narrow to what the device renders differently from RealityRenderer on the Mac: the core's hard alpha cut rasterized without anti-aliasing (on the Mac the band covered it to the floor; on the device the band's blend and the cut's step apparently do not add up to a still edge), and the texel staircase marching under the head's motion (the "shimmer": the engine's texture changes every frame, the control's geometry does not). The next experiment, named by the study: a core that blends instead of cutting, with depth kept (`ShaderGraphMaterial.writesDepth`, so occlusion against Personas and RealityKit content stays), the whole grid through the band's blended graph and the band meshes off; on the Mac it is the same picture, on the device it removes the alpha test from the silhouette altogether. Its cost is the transparent pass for a full-screen mesh. If the shimmer stays with a blended core, it is the texture's own frame-to-frame change at the silhouette, and the remaining lever is temporal (the engine's TAA, judged "the same" for the static edge, to be judged for the shimmer). +- **The blended core, built and merged (2026-10-19, 18:06; engine `feature/xr_realitykit_presenter` fast-forwarded to `da5cbc7a`, example `feature/realitykit_presenter_example` to `613dd24`, both pushed; 99 tests).** Built in its own session: `XRRealityKitPresenterConfiguration.silhouette`, `.cutAndBand` (the default, byte-identical) or `.blendedCore`, fixed for the next space opened; the presenter's `silhouette` says which is in force (the blended core falls back to the shipped chain with a warning if its graph does not load). The blended core is a third graph, `XRPresenterBlended.usda`: the band's blended graph on the grids themselves, loaded with `writesDepth` on so the grid still occludes RealityKit's content and the Personas the system composites by depth, plus a geometry modifier collapsing the OTHER eye's grid to a point through a vertex-stage Camera Index Switch, which was necessary: with blending every rasterized fragment writes depth, those at opacity 0 too, so the other eye's invisible grid would have written its parallax-offset depth into this eye and cut holes along its own silhouettes. No band meshes or entities in that mode; `hasEdgeBand` false; the ticket carries no band buffers and the kernel's band pass is not encoded. Verified on the Mac: a RealityKit box behind the cubes, forced after the grids in the transparent pass, is hidden under every covered cube texel and shows over the room six texels out (with `writesDepth` off the box covers the cubes; with the left grid disabled the collapsed right grid hides nothing); the picture through the blended core equals the engine's MSAA eye and the shipped chain's at 90 dB. A finding on the way: `writesDepth` reads true as loaded for every shader graph, the shipped band's too, so the shipped chain already hides the same 921-texel halo (the skirt texels and the near part of the stretched triangles at near-zero opacity): the blended core adds no depth footprint the band did not have. The example: "Blended core" in the next-space row, `--blended-core` / `--cut-and-band`, the settings line "silhouette blended core (depth written, no band)" or "cut and band (edge band on)"; the Edge band toggle has no effect under the blended core. On the headset: the settings line (a "graph did not load" means the device refused the geometry modifier or the vertex-stage switch), the picture whole in each eye (a blank eye means the vertex-stage switch read mono), the engine cubes' edges beside the control cube (still like the control's if the device's rasterization of the shipped cut was the shimmer; still shimmering if it is the texture's own frame-to-frame change, and the lever is then temporal), and the gpu on the numbers line (one blended pass over the grid against the alpha-tested core plus the band's cells). From bb860d3cf7cfa99f11618efb5726c95a1f9d0bad Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:24:23 +0200 Subject: [PATCH 66/72] [Docs] XRRealityKitPresenter plan: the blended core makes no difference; the waves point at the cells' depth under the compositor's reprojection --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index e812ce5d2..e1badb33f 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -258,3 +258,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The control cube, built and merged (2026-10-19, 17:34; example `feature/realitykit_presenter_example` fast-forwarded to `e5b4dc4`, pushed; example only).** Built in its own session: `ControlCube.swift`, a lit pale-blue RealityKit cube (`SimpleMaterial`, roughness 0.4) of the engine cubes' 0.2 m, one step to the right of the engine's three on the slab's plane (x 0.7, the same gap and height), no yaw so its far top edge is one straight line like the middle cube's, a child of the presenter's root beside the grid entities (world-locked like them, nothing the presenter manages); a live, remembered "Control cube" toggle, `--control-cube`, "control cube on" on the settings line. Release built and installed. The capture to take: both cubes in one frame, framed like the morning's, with the rate map and the scale as the user likes; `cube_tops2.py` on it reads the four top edges with one definition. The same wobble on the control names the device's render or the capture, and no presenter change can touch it; a straight control beside wavy engine cubes names the presenter's materials under the device's render, and the next experiment is then a core that blends with depth kept. - **The control cube decides (2026-10-19, 17:40, headset, Release, example `e5b4dc4`): the presenter's chain is at fault, not the device or the captures.** In one frame, the user sees the engine's cubes shimmer while the RealityKit cube beside them does not, and the control's edges "far better". The capture agrees: the control's top edge 0.35 px rms about a line (peak to peak 1.6, period 44 px) against the engine cubes' 0.82 / 0.64 / 0.55 px (peak to peak 2.5 to 3.3, periods 58 to 83 px), through the same device render and the same capture; the control's edge, drawn by RealityKit's own geometry pass with its anti-aliasing, is what the device can do, and the engine's edges through the presenter are twice as wavy and move. So the Mac study's candidates narrow to what the device renders differently from RealityRenderer on the Mac: the core's hard alpha cut rasterized without anti-aliasing (on the Mac the band covered it to the floor; on the device the band's blend and the cut's step apparently do not add up to a still edge), and the texel staircase marching under the head's motion (the "shimmer": the engine's texture changes every frame, the control's geometry does not). The next experiment, named by the study: a core that blends instead of cutting, with depth kept (`ShaderGraphMaterial.writesDepth`, so occlusion against Personas and RealityKit content stays), the whole grid through the band's blended graph and the band meshes off; on the Mac it is the same picture, on the device it removes the alpha test from the silhouette altogether. Its cost is the transparent pass for a full-screen mesh. If the shimmer stays with a blended core, it is the texture's own frame-to-frame change at the silhouette, and the remaining lever is temporal (the engine's TAA, judged "the same" for the static edge, to be judged for the shimmer). - **The blended core, built and merged (2026-10-19, 18:06; engine `feature/xr_realitykit_presenter` fast-forwarded to `da5cbc7a`, example `feature/realitykit_presenter_example` to `613dd24`, both pushed; 99 tests).** Built in its own session: `XRRealityKitPresenterConfiguration.silhouette`, `.cutAndBand` (the default, byte-identical) or `.blendedCore`, fixed for the next space opened; the presenter's `silhouette` says which is in force (the blended core falls back to the shipped chain with a warning if its graph does not load). The blended core is a third graph, `XRPresenterBlended.usda`: the band's blended graph on the grids themselves, loaded with `writesDepth` on so the grid still occludes RealityKit's content and the Personas the system composites by depth, plus a geometry modifier collapsing the OTHER eye's grid to a point through a vertex-stage Camera Index Switch, which was necessary: with blending every rasterized fragment writes depth, those at opacity 0 too, so the other eye's invisible grid would have written its parallax-offset depth into this eye and cut holes along its own silhouettes. No band meshes or entities in that mode; `hasEdgeBand` false; the ticket carries no band buffers and the kernel's band pass is not encoded. Verified on the Mac: a RealityKit box behind the cubes, forced after the grids in the transparent pass, is hidden under every covered cube texel and shows over the room six texels out (with `writesDepth` off the box covers the cubes; with the left grid disabled the collapsed right grid hides nothing); the picture through the blended core equals the engine's MSAA eye and the shipped chain's at 90 dB. A finding on the way: `writesDepth` reads true as loaded for every shader graph, the shipped band's too, so the shipped chain already hides the same 921-texel halo (the skirt texels and the near part of the stretched triangles at near-zero opacity): the blended core adds no depth footprint the band did not have. The example: "Blended core" in the next-space row, `--blended-core` / `--cut-and-band`, the settings line "silhouette blended core (depth written, no band)" or "cut and band (edge band on)"; the Edge band toggle has no effect under the blended core. On the headset: the settings line (a "graph did not load" means the device refused the geometry modifier or the vertex-stage switch), the picture whole in each eye (a blank eye means the vertex-stage switch read mono), the engine cubes' edges beside the control cube (still like the control's if the device's rasterization of the shipped cut was the shimmer; still shimmering if it is the texture's own frame-to-frame change, and the lever is then temporal), and the gpu on the numbers line (one blended pass over the grid against the alpha-tested core plus the band's cells). +- **The blended core on the headset, and what the user then saw (2026-10-19, 18:24, Release, engine `da5cbc7a`): no difference with the blended core on or off; the cubes' edges shimmer less with the head held still but still "wave", moving the head makes the waves at the edges move more, and even at rest the segments of a cube's edge are not one straight line.** The cut was never the cause (the blended core stays as an option). Segments at rest and waves under motion are the signature of depth, not of texels: each grid cell carries one depth plane, exact on a flat face seen head-on (the slab's front, whose edge measures at the floor) and wrong by up to the object's size on an edge-on sliver (a cube's top seen from just above: 20 cm of depth across one or two texels, whatever depth a cell's vertices take) or a curved limb (the sphere, mildly). On the device, RealityKit's frame is reprojected per pixel by its depth before display (the compositor's warp to the actual pose, which in mixed immersion also aligns the content with the passthrough), so pixels whose depth is wrong are warped wrongly: a kink per cell at rest (the prediction's residual, the device's own eye positions against the nominal 63 mm the engine renders from, the micro-motion of a held head) and waves that grow with the pose change under motion. The control cube has true depth and shows none of it; the Mac's RealityRenderer has no such reprojection, which is why every Mac study found the chain equal to the resampled texture at every density and every knob, and why the texel density, the cut, the band and TAA changed nothing on the device. The one-minute test that decides it is in the panel already: the fixed lead at 0 ms against 60 ms. If the waves grow with the lead (the pose change the compositor warps across), the depth is the cause. If they do not, the per-cell depth is wrong in a way the lead does not expose and the test is the view offset again, read for the waves rather than the rim. Remedies, if depth: the cells' depth closer to right where it matters (the rim and sliver vertices at the median of the covered texels' depths rather than the nearest; the MSAA depth resolve by nearest sample; cells split where the depth varies steeply, which the kernel can see per cell), a shorter lead (the warp is smaller, and the two-refresh pipeline lengthens it), and, the real one, depth per pixel for the compositor, which RealityKit's materials do not write: an item to raise with Apple, or a reason the hybrid keeps a halo of inexactness at silhouettes that the Compositor path does not have. From c5cf8013803d5178dcf5af2dc51275a8b1e20425 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:30:23 +0200 Subject: [PATCH 67/72] [Docs] XRRealityKitPresenter plan: the lead does not move the waves; the eyes or the cells' depth, and tomorrow's device experiments --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index e1badb33f..d5c7fa308 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -259,3 +259,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The control cube decides (2026-10-19, 17:40, headset, Release, example `e5b4dc4`): the presenter's chain is at fault, not the device or the captures.** In one frame, the user sees the engine's cubes shimmer while the RealityKit cube beside them does not, and the control's edges "far better". The capture agrees: the control's top edge 0.35 px rms about a line (peak to peak 1.6, period 44 px) against the engine cubes' 0.82 / 0.64 / 0.55 px (peak to peak 2.5 to 3.3, periods 58 to 83 px), through the same device render and the same capture; the control's edge, drawn by RealityKit's own geometry pass with its anti-aliasing, is what the device can do, and the engine's edges through the presenter are twice as wavy and move. So the Mac study's candidates narrow to what the device renders differently from RealityRenderer on the Mac: the core's hard alpha cut rasterized without anti-aliasing (on the Mac the band covered it to the floor; on the device the band's blend and the cut's step apparently do not add up to a still edge), and the texel staircase marching under the head's motion (the "shimmer": the engine's texture changes every frame, the control's geometry does not). The next experiment, named by the study: a core that blends instead of cutting, with depth kept (`ShaderGraphMaterial.writesDepth`, so occlusion against Personas and RealityKit content stays), the whole grid through the band's blended graph and the band meshes off; on the Mac it is the same picture, on the device it removes the alpha test from the silhouette altogether. Its cost is the transparent pass for a full-screen mesh. If the shimmer stays with a blended core, it is the texture's own frame-to-frame change at the silhouette, and the remaining lever is temporal (the engine's TAA, judged "the same" for the static edge, to be judged for the shimmer). - **The blended core, built and merged (2026-10-19, 18:06; engine `feature/xr_realitykit_presenter` fast-forwarded to `da5cbc7a`, example `feature/realitykit_presenter_example` to `613dd24`, both pushed; 99 tests).** Built in its own session: `XRRealityKitPresenterConfiguration.silhouette`, `.cutAndBand` (the default, byte-identical) or `.blendedCore`, fixed for the next space opened; the presenter's `silhouette` says which is in force (the blended core falls back to the shipped chain with a warning if its graph does not load). The blended core is a third graph, `XRPresenterBlended.usda`: the band's blended graph on the grids themselves, loaded with `writesDepth` on so the grid still occludes RealityKit's content and the Personas the system composites by depth, plus a geometry modifier collapsing the OTHER eye's grid to a point through a vertex-stage Camera Index Switch, which was necessary: with blending every rasterized fragment writes depth, those at opacity 0 too, so the other eye's invisible grid would have written its parallax-offset depth into this eye and cut holes along its own silhouettes. No band meshes or entities in that mode; `hasEdgeBand` false; the ticket carries no band buffers and the kernel's band pass is not encoded. Verified on the Mac: a RealityKit box behind the cubes, forced after the grids in the transparent pass, is hidden under every covered cube texel and shows over the room six texels out (with `writesDepth` off the box covers the cubes; with the left grid disabled the collapsed right grid hides nothing); the picture through the blended core equals the engine's MSAA eye and the shipped chain's at 90 dB. A finding on the way: `writesDepth` reads true as loaded for every shader graph, the shipped band's too, so the shipped chain already hides the same 921-texel halo (the skirt texels and the near part of the stretched triangles at near-zero opacity): the blended core adds no depth footprint the band did not have. The example: "Blended core" in the next-space row, `--blended-core` / `--cut-and-band`, the settings line "silhouette blended core (depth written, no band)" or "cut and band (edge band on)"; the Edge band toggle has no effect under the blended core. On the headset: the settings line (a "graph did not load" means the device refused the geometry modifier or the vertex-stage switch), the picture whole in each eye (a blank eye means the vertex-stage switch read mono), the engine cubes' edges beside the control cube (still like the control's if the device's rasterization of the shipped cut was the shimmer; still shimmering if it is the texture's own frame-to-frame change, and the lever is then temporal), and the gpu on the numbers line (one blended pass over the grid against the alpha-tested core plus the band's cells). - **The blended core on the headset, and what the user then saw (2026-10-19, 18:24, Release, engine `da5cbc7a`): no difference with the blended core on or off; the cubes' edges shimmer less with the head held still but still "wave", moving the head makes the waves at the edges move more, and even at rest the segments of a cube's edge are not one straight line.** The cut was never the cause (the blended core stays as an option). Segments at rest and waves under motion are the signature of depth, not of texels: each grid cell carries one depth plane, exact on a flat face seen head-on (the slab's front, whose edge measures at the floor) and wrong by up to the object's size on an edge-on sliver (a cube's top seen from just above: 20 cm of depth across one or two texels, whatever depth a cell's vertices take) or a curved limb (the sphere, mildly). On the device, RealityKit's frame is reprojected per pixel by its depth before display (the compositor's warp to the actual pose, which in mixed immersion also aligns the content with the passthrough), so pixels whose depth is wrong are warped wrongly: a kink per cell at rest (the prediction's residual, the device's own eye positions against the nominal 63 mm the engine renders from, the micro-motion of a held head) and waves that grow with the pose change under motion. The control cube has true depth and shows none of it; the Mac's RealityRenderer has no such reprojection, which is why every Mac study found the chain equal to the resampled texture at every density and every knob, and why the texel density, the cut, the band and TAA changed nothing on the device. The one-minute test that decides it is in the panel already: the fixed lead at 0 ms against 60 ms. If the waves grow with the lead (the pose change the compositor warps across), the depth is the cause. If they do not, the per-cell depth is wrong in a way the lead does not expose and the test is the view offset again, read for the waves rather than the rim. Remedies, if depth: the cells' depth closer to right where it matters (the rim and sliver vertices at the median of the covered texels' depths rather than the nearest; the MSAA depth resolve by nearest sample; cells split where the depth varies steeply, which the kernel can see per cell), a shorter lead (the warp is smaller, and the two-refresh pipeline lengthens it), and, the real one, depth per pixel for the compositor, which RealityKit's materials do not write: an item to raise with Apple, or a reason the hybrid keeps a halo of inexactness at silhouettes that the Compositor path does not have. +- **The lead does not move the waves (2026-10-19, 18:30): the fixed lead at 0 ms and at 60 ms look the same on the cube edges.** Both are wrong by a similar amount from the true 22 ms (20 ms behind, 38 ahead), so the test does not rule the compositor's warp in or out as cleanly as hoped, but the waves are there at rest, with no pose change to warp across, so the dominant cause is static. **What survives every reading of the day**: the kinks and waves live only where a cell's one depth plane cannot follow the surface (the cubes' edge-on top slivers, the sphere's limb) and not where it can (the slab's front face, at the floor); they are the same at double the texel density, under any cut, with the band or the blended core, with or without TAA, at any lead, at either pipeline depth, on the 512 grid, with the shell at 50 m, and, as judged this morning for the rim, with both eyes' viewpoints raised 3 cm; the control cube in the same frame is clean; the Mac's renderer, from the engine's eye and 3 mm off it, shows none of it. Two static mechanisms fit, both device-only: **(A) the eyes**: the engine renders and the grid is built from nominal eyes (63 mm apart, at the device anchor with no forward or vertical offset), RealityKit renders the grid from the device's actual eye transforms, and the difference, a few millimetres of IPD and whatever the eyes' real position is against the anchor, is a parallax that only a cell with the wrong depth shows (δ × Δz / z²: a centimetre of eye against 10 cm of depth error at 1.6 m is 0.4 mrad, about 1.3 display pixels, the size measured); **(B) the cells' depth at slivers**: the rim rule puts a vertex at the nearest depth in its footprint, so the texels of a top face 20 cm deep sit on a plane at the front face's depth, and any eye difference at all, including (A), shows as a kink per cell. Against (A): the 3 cm vertical offset of this morning, judged for the rim rather than for the waves. Tomorrow's device experiments, cheapest first, each read for the waves on the cube tops at rest: the IPD at 55 and 70 mm against 63 (a wrong IPD must change the kinks if the eyes are the cause); the view offset along each axis (forward, lateral, vertical), judged for the waves; the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m (the rim vertices' depths); then, if the eyes are the cause, the real eye transforms (RealityKit gives none; ARKit's device anchor and the Compositor path's drawable views do, and the latter cannot be queried from a RealityView: an item for Apple, or a calibration setting), and if the cells are, a rim rule by the median depth of the covered texels and cells split where the depth varies steeply. From 343de47d6bb1aaf73a41442e24b733dbb78911f9 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:39:48 +0200 Subject: [PATCH 68/72] [Docs] XRRealityKitPresenter plan: the eye settings, built and merged; the device matrix --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index d5c7fa308..f9912aae9 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -260,3 +260,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The blended core, built and merged (2026-10-19, 18:06; engine `feature/xr_realitykit_presenter` fast-forwarded to `da5cbc7a`, example `feature/realitykit_presenter_example` to `613dd24`, both pushed; 99 tests).** Built in its own session: `XRRealityKitPresenterConfiguration.silhouette`, `.cutAndBand` (the default, byte-identical) or `.blendedCore`, fixed for the next space opened; the presenter's `silhouette` says which is in force (the blended core falls back to the shipped chain with a warning if its graph does not load). The blended core is a third graph, `XRPresenterBlended.usda`: the band's blended graph on the grids themselves, loaded with `writesDepth` on so the grid still occludes RealityKit's content and the Personas the system composites by depth, plus a geometry modifier collapsing the OTHER eye's grid to a point through a vertex-stage Camera Index Switch, which was necessary: with blending every rasterized fragment writes depth, those at opacity 0 too, so the other eye's invisible grid would have written its parallax-offset depth into this eye and cut holes along its own silhouettes. No band meshes or entities in that mode; `hasEdgeBand` false; the ticket carries no band buffers and the kernel's band pass is not encoded. Verified on the Mac: a RealityKit box behind the cubes, forced after the grids in the transparent pass, is hidden under every covered cube texel and shows over the room six texels out (with `writesDepth` off the box covers the cubes; with the left grid disabled the collapsed right grid hides nothing); the picture through the blended core equals the engine's MSAA eye and the shipped chain's at 90 dB. A finding on the way: `writesDepth` reads true as loaded for every shader graph, the shipped band's too, so the shipped chain already hides the same 921-texel halo (the skirt texels and the near part of the stretched triangles at near-zero opacity): the blended core adds no depth footprint the band did not have. The example: "Blended core" in the next-space row, `--blended-core` / `--cut-and-band`, the settings line "silhouette blended core (depth written, no band)" or "cut and band (edge band on)"; the Edge band toggle has no effect under the blended core. On the headset: the settings line (a "graph did not load" means the device refused the geometry modifier or the vertex-stage switch), the picture whole in each eye (a blank eye means the vertex-stage switch read mono), the engine cubes' edges beside the control cube (still like the control's if the device's rasterization of the shipped cut was the shimmer; still shimmering if it is the texture's own frame-to-frame change, and the lever is then temporal), and the gpu on the numbers line (one blended pass over the grid against the alpha-tested core plus the band's cells). - **The blended core on the headset, and what the user then saw (2026-10-19, 18:24, Release, engine `da5cbc7a`): no difference with the blended core on or off; the cubes' edges shimmer less with the head held still but still "wave", moving the head makes the waves at the edges move more, and even at rest the segments of a cube's edge are not one straight line.** The cut was never the cause (the blended core stays as an option). Segments at rest and waves under motion are the signature of depth, not of texels: each grid cell carries one depth plane, exact on a flat face seen head-on (the slab's front, whose edge measures at the floor) and wrong by up to the object's size on an edge-on sliver (a cube's top seen from just above: 20 cm of depth across one or two texels, whatever depth a cell's vertices take) or a curved limb (the sphere, mildly). On the device, RealityKit's frame is reprojected per pixel by its depth before display (the compositor's warp to the actual pose, which in mixed immersion also aligns the content with the passthrough), so pixels whose depth is wrong are warped wrongly: a kink per cell at rest (the prediction's residual, the device's own eye positions against the nominal 63 mm the engine renders from, the micro-motion of a held head) and waves that grow with the pose change under motion. The control cube has true depth and shows none of it; the Mac's RealityRenderer has no such reprojection, which is why every Mac study found the chain equal to the resampled texture at every density and every knob, and why the texel density, the cut, the band and TAA changed nothing on the device. The one-minute test that decides it is in the panel already: the fixed lead at 0 ms against 60 ms. If the waves grow with the lead (the pose change the compositor warps across), the depth is the cause. If they do not, the per-cell depth is wrong in a way the lead does not expose and the test is the view offset again, read for the waves rather than the rim. Remedies, if depth: the cells' depth closer to right where it matters (the rim and sliver vertices at the median of the covered texels' depths rather than the nearest; the MSAA depth resolve by nearest sample; cells split where the depth varies steeply, which the kernel can see per cell), a shorter lead (the warp is smaller, and the two-refresh pipeline lengthens it), and, the real one, depth per pixel for the compositor, which RealityKit's materials do not write: an item to raise with Apple, or a reason the hybrid keeps a halo of inexactness at silhouettes that the Compositor path does not have. - **The lead does not move the waves (2026-10-19, 18:30): the fixed lead at 0 ms and at 60 ms look the same on the cube edges.** Both are wrong by a similar amount from the true 22 ms (20 ms behind, 38 ahead), so the test does not rule the compositor's warp in or out as cleanly as hoped, but the waves are there at rest, with no pose change to warp across, so the dominant cause is static. **What survives every reading of the day**: the kinks and waves live only where a cell's one depth plane cannot follow the surface (the cubes' edge-on top slivers, the sphere's limb) and not where it can (the slab's front face, at the floor); they are the same at double the texel density, under any cut, with the band or the blended core, with or without TAA, at any lead, at either pipeline depth, on the 512 grid, with the shell at 50 m, and, as judged this morning for the rim, with both eyes' viewpoints raised 3 cm; the control cube in the same frame is clean; the Mac's renderer, from the engine's eye and 3 mm off it, shows none of it. Two static mechanisms fit, both device-only: **(A) the eyes**: the engine renders and the grid is built from nominal eyes (63 mm apart, at the device anchor with no forward or vertical offset), RealityKit renders the grid from the device's actual eye transforms, and the difference, a few millimetres of IPD and whatever the eyes' real position is against the anchor, is a parallax that only a cell with the wrong depth shows (δ × Δz / z²: a centimetre of eye against 10 cm of depth error at 1.6 m is 0.4 mrad, about 1.3 display pixels, the size measured); **(B) the cells' depth at slivers**: the rim rule puts a vertex at the nearest depth in its footprint, so the texels of a top face 20 cm deep sit on a plane at the front face's depth, and any eye difference at all, including (A), shows as a kink per cell. Against (A): the 3 cm vertical offset of this morning, judged for the rim rather than for the waves. Tomorrow's device experiments, cheapest first, each read for the waves on the cube tops at rest: the IPD at 55 and 70 mm against 63 (a wrong IPD must change the kinks if the eyes are the cause); the view offset along each axis (forward, lateral, vertical), judged for the waves; the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m (the rim vertices' depths); then, if the eyes are the cause, the real eye transforms (RealityKit gives none; ARKit's device anchor and the Compositor path's drawable views do, and the latter cannot be queried from a RealityView: an item for Apple, or a calibration setting), and if the cells are, a rim rule by the median depth of the covered texels and cells split where the depth varies steeply. +- **The eye settings, built and merged (2026-10-19, 18:39; example `feature/realitykit_presenter_example` fast-forwarded to `fe379fb`, pushed; example only, the engine at `da5cbc7a`).** Built in its own session: an IPD stepper (50 to 75 mm, 63 by default) and a view offset along each axis of the device anchor's frame (x right, y up, z back; -5 to +5 cm, zero by default) in the next-space row, replacing the one-shot View +3 cm toggle; `--ipd 63`, `--view-offset x,y,z` (cm); the settings line names both. Release built and installed. The device matrix for the waves on the cube tops at rest, cheapest first: the IPD at 55 and 70 against 63 (a wrong IPD must change the kinks if the eyes are their cause: the parallax a wrong-depth cell shows is the eye difference times the depth error over the distance squared); the view offset at ±3 cm along each axis in turn (a real eye position off the anchor along an axis is found by the offset that flattens the kinks; a few centimetres that change nothing clear the eyes); then the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m, both live (the rim vertices' depths). From 1202bb97b73637c1247524c8a96485a62ee4eb5d Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:43:18 +0200 Subject: [PATCH 69/72] [Docs] XRRealityKitPresenter plan: the IPD does not move the kinks --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index f9912aae9..bcae0c01a 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -261,3 +261,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The blended core on the headset, and what the user then saw (2026-10-19, 18:24, Release, engine `da5cbc7a`): no difference with the blended core on or off; the cubes' edges shimmer less with the head held still but still "wave", moving the head makes the waves at the edges move more, and even at rest the segments of a cube's edge are not one straight line.** The cut was never the cause (the blended core stays as an option). Segments at rest and waves under motion are the signature of depth, not of texels: each grid cell carries one depth plane, exact on a flat face seen head-on (the slab's front, whose edge measures at the floor) and wrong by up to the object's size on an edge-on sliver (a cube's top seen from just above: 20 cm of depth across one or two texels, whatever depth a cell's vertices take) or a curved limb (the sphere, mildly). On the device, RealityKit's frame is reprojected per pixel by its depth before display (the compositor's warp to the actual pose, which in mixed immersion also aligns the content with the passthrough), so pixels whose depth is wrong are warped wrongly: a kink per cell at rest (the prediction's residual, the device's own eye positions against the nominal 63 mm the engine renders from, the micro-motion of a held head) and waves that grow with the pose change under motion. The control cube has true depth and shows none of it; the Mac's RealityRenderer has no such reprojection, which is why every Mac study found the chain equal to the resampled texture at every density and every knob, and why the texel density, the cut, the band and TAA changed nothing on the device. The one-minute test that decides it is in the panel already: the fixed lead at 0 ms against 60 ms. If the waves grow with the lead (the pose change the compositor warps across), the depth is the cause. If they do not, the per-cell depth is wrong in a way the lead does not expose and the test is the view offset again, read for the waves rather than the rim. Remedies, if depth: the cells' depth closer to right where it matters (the rim and sliver vertices at the median of the covered texels' depths rather than the nearest; the MSAA depth resolve by nearest sample; cells split where the depth varies steeply, which the kernel can see per cell), a shorter lead (the warp is smaller, and the two-refresh pipeline lengthens it), and, the real one, depth per pixel for the compositor, which RealityKit's materials do not write: an item to raise with Apple, or a reason the hybrid keeps a halo of inexactness at silhouettes that the Compositor path does not have. - **The lead does not move the waves (2026-10-19, 18:30): the fixed lead at 0 ms and at 60 ms look the same on the cube edges.** Both are wrong by a similar amount from the true 22 ms (20 ms behind, 38 ahead), so the test does not rule the compositor's warp in or out as cleanly as hoped, but the waves are there at rest, with no pose change to warp across, so the dominant cause is static. **What survives every reading of the day**: the kinks and waves live only where a cell's one depth plane cannot follow the surface (the cubes' edge-on top slivers, the sphere's limb) and not where it can (the slab's front face, at the floor); they are the same at double the texel density, under any cut, with the band or the blended core, with or without TAA, at any lead, at either pipeline depth, on the 512 grid, with the shell at 50 m, and, as judged this morning for the rim, with both eyes' viewpoints raised 3 cm; the control cube in the same frame is clean; the Mac's renderer, from the engine's eye and 3 mm off it, shows none of it. Two static mechanisms fit, both device-only: **(A) the eyes**: the engine renders and the grid is built from nominal eyes (63 mm apart, at the device anchor with no forward or vertical offset), RealityKit renders the grid from the device's actual eye transforms, and the difference, a few millimetres of IPD and whatever the eyes' real position is against the anchor, is a parallax that only a cell with the wrong depth shows (δ × Δz / z²: a centimetre of eye against 10 cm of depth error at 1.6 m is 0.4 mrad, about 1.3 display pixels, the size measured); **(B) the cells' depth at slivers**: the rim rule puts a vertex at the nearest depth in its footprint, so the texels of a top face 20 cm deep sit on a plane at the front face's depth, and any eye difference at all, including (A), shows as a kink per cell. Against (A): the 3 cm vertical offset of this morning, judged for the rim rather than for the waves. Tomorrow's device experiments, cheapest first, each read for the waves on the cube tops at rest: the IPD at 55 and 70 mm against 63 (a wrong IPD must change the kinks if the eyes are the cause); the view offset along each axis (forward, lateral, vertical), judged for the waves; the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m (the rim vertices' depths); then, if the eyes are the cause, the real eye transforms (RealityKit gives none; ARKit's device anchor and the Compositor path's drawable views do, and the latter cannot be queried from a RealityView: an item for Apple, or a calibration setting), and if the cells are, a rim rule by the median depth of the covered texels and cells split where the depth varies steeply. - **The eye settings, built and merged (2026-10-19, 18:39; example `feature/realitykit_presenter_example` fast-forwarded to `fe379fb`, pushed; example only, the engine at `da5cbc7a`).** Built in its own session: an IPD stepper (50 to 75 mm, 63 by default) and a view offset along each axis of the device anchor's frame (x right, y up, z back; -5 to +5 cm, zero by default) in the next-space row, replacing the one-shot View +3 cm toggle; `--ipd 63`, `--view-offset x,y,z` (cm); the settings line names both. Release built and installed. The device matrix for the waves on the cube tops at rest, cheapest first: the IPD at 55 and 70 against 63 (a wrong IPD must change the kinks if the eyes are their cause: the parallax a wrong-depth cell shows is the eye difference times the depth error over the distance squared); the view offset at ±3 cm along each axis in turn (a real eye position off the anchor along an axis is found by the offset that flattens the kinks; a few centimetres that change nothing clear the eyes); then the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m, both live (the rim vertices' depths). +- **Matrix, step 1 (2026-10-19, 18:43): the kinks on the cube tops look the same at IPD 55, 63 and 70 mm.** Fifteen millimetres of eye separation, 7.5 per eye, would move a parallax kink through a wrong-depth cell by about a display pixel at the slivers; nothing moved. The eye separation is cleared; the view offset along each axis is the remaining half of the eye hypothesis, then the rim vertices' depths (skirt, tear). From 21bd99f07a1f6914dc3637adcb9671b250651127 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:48:37 +0200 Subject: [PATCH 70/72] [Docs] XRRealityKitPresenter plan: the view offset does not move the kinks; the shimmer varies from space to space --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index bcae0c01a..929196c57 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -262,3 +262,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The lead does not move the waves (2026-10-19, 18:30): the fixed lead at 0 ms and at 60 ms look the same on the cube edges.** Both are wrong by a similar amount from the true 22 ms (20 ms behind, 38 ahead), so the test does not rule the compositor's warp in or out as cleanly as hoped, but the waves are there at rest, with no pose change to warp across, so the dominant cause is static. **What survives every reading of the day**: the kinks and waves live only where a cell's one depth plane cannot follow the surface (the cubes' edge-on top slivers, the sphere's limb) and not where it can (the slab's front face, at the floor); they are the same at double the texel density, under any cut, with the band or the blended core, with or without TAA, at any lead, at either pipeline depth, on the 512 grid, with the shell at 50 m, and, as judged this morning for the rim, with both eyes' viewpoints raised 3 cm; the control cube in the same frame is clean; the Mac's renderer, from the engine's eye and 3 mm off it, shows none of it. Two static mechanisms fit, both device-only: **(A) the eyes**: the engine renders and the grid is built from nominal eyes (63 mm apart, at the device anchor with no forward or vertical offset), RealityKit renders the grid from the device's actual eye transforms, and the difference, a few millimetres of IPD and whatever the eyes' real position is against the anchor, is a parallax that only a cell with the wrong depth shows (δ × Δz / z²: a centimetre of eye against 10 cm of depth error at 1.6 m is 0.4 mrad, about 1.3 display pixels, the size measured); **(B) the cells' depth at slivers**: the rim rule puts a vertex at the nearest depth in its footprint, so the texels of a top face 20 cm deep sit on a plane at the front face's depth, and any eye difference at all, including (A), shows as a kink per cell. Against (A): the 3 cm vertical offset of this morning, judged for the rim rather than for the waves. Tomorrow's device experiments, cheapest first, each read for the waves on the cube tops at rest: the IPD at 55 and 70 mm against 63 (a wrong IPD must change the kinks if the eyes are the cause); the view offset along each axis (forward, lateral, vertical), judged for the waves; the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m (the rim vertices' depths); then, if the eyes are the cause, the real eye transforms (RealityKit gives none; ARKit's device anchor and the Compositor path's drawable views do, and the latter cannot be queried from a RealityView: an item for Apple, or a calibration setting), and if the cells are, a rim rule by the median depth of the covered texels and cells split where the depth varies steeply. - **The eye settings, built and merged (2026-10-19, 18:39; example `feature/realitykit_presenter_example` fast-forwarded to `fe379fb`, pushed; example only, the engine at `da5cbc7a`).** Built in its own session: an IPD stepper (50 to 75 mm, 63 by default) and a view offset along each axis of the device anchor's frame (x right, y up, z back; -5 to +5 cm, zero by default) in the next-space row, replacing the one-shot View +3 cm toggle; `--ipd 63`, `--view-offset x,y,z` (cm); the settings line names both. Release built and installed. The device matrix for the waves on the cube tops at rest, cheapest first: the IPD at 55 and 70 against 63 (a wrong IPD must change the kinks if the eyes are their cause: the parallax a wrong-depth cell shows is the eye difference times the depth error over the distance squared); the view offset at ±3 cm along each axis in turn (a real eye position off the anchor along an axis is found by the offset that flattens the kinks; a few centimetres that change nothing clear the eyes); then the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m, both live (the rim vertices' depths). - **Matrix, step 1 (2026-10-19, 18:43): the kinks on the cube tops look the same at IPD 55, 63 and 70 mm.** Fifteen millimetres of eye separation, 7.5 per eye, would move a parallax kink through a wrong-depth cell by about a display pixel at the slivers; nothing moved. The eye separation is cleared; the view offset along each axis is the remaining half of the eye hypothesis, then the rim vertices' depths (skirt, tear). +- **Matrix, step 2 (2026-10-19, 18:48): the kinks look the same with the view offset at +3, 0 and -3 cm along z; and the edges shimmered less in the previous space than in this one, with nothing changed between them.** The eyes are cleared on both counts (the IPD and the eyes' position against the anchor), as this morning's vertical 3 cm had suggested. The run-to-run variation is the new fact: the shimmer's strength changes from one space to the next without a setting change, so its cause is something the device supplies differently each time (the tracking's jitter, which the engine's per-frame pose follows and RealityKit's render does not have to; the GPU's state and the frames' timing; the device's own foveated render), not a constant of the grid's geometry alone. The static kinks at rest remain unexplained by any parallax (no eye offset moves them) and are cell-periodic; what is left for them is the cell's own mapping of a sliver's texels where the two triangles of a cell lie in different planes, which from the engine's exact eye is invisible and from any other eye is a kink, so a per-frame eye difference of the tracking's size, variable from run to run, would give both the kinks and the shimmer. A diagnostic panel over the whole day's evidence is run overnight for a ranked list of mechanisms and the device experiment that separates them. From c5d408b1bbf0c52101a8f989af062da388d7e161 Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 18:53:42 +0200 Subject: [PATCH 71/72] [Docs] XRRealityKitPresenter plan: the tear threshold is the first knob that moves the shimmer; the skirt does nothing --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 929196c57..3d9233171 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -263,3 +263,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **The eye settings, built and merged (2026-10-19, 18:39; example `feature/realitykit_presenter_example` fast-forwarded to `fe379fb`, pushed; example only, the engine at `da5cbc7a`).** Built in its own session: an IPD stepper (50 to 75 mm, 63 by default) and a view offset along each axis of the device anchor's frame (x right, y up, z back; -5 to +5 cm, zero by default) in the next-space row, replacing the one-shot View +3 cm toggle; `--ipd 63`, `--view-offset x,y,z` (cm); the settings line names both. Release built and installed. The device matrix for the waves on the cube tops at rest, cheapest first: the IPD at 55 and 70 against 63 (a wrong IPD must change the kinks if the eyes are their cause: the parallax a wrong-depth cell shows is the eye difference times the depth error over the distance squared); the view offset at ±3 cm along each axis in turn (a real eye position off the anchor along an axis is found by the offset that flattens the kinks; a few centimetres that change nothing clear the eyes); then the skirt at 0 and 8 against 3 and the tear threshold at 0.1 and 3 m, both live (the rim vertices' depths). - **Matrix, step 1 (2026-10-19, 18:43): the kinks on the cube tops look the same at IPD 55, 63 and 70 mm.** Fifteen millimetres of eye separation, 7.5 per eye, would move a parallax kink through a wrong-depth cell by about a display pixel at the slivers; nothing moved. The eye separation is cleared; the view offset along each axis is the remaining half of the eye hypothesis, then the rim vertices' depths (skirt, tear). - **Matrix, step 2 (2026-10-19, 18:48): the kinks look the same with the view offset at +3, 0 and -3 cm along z; and the edges shimmered less in the previous space than in this one, with nothing changed between them.** The eyes are cleared on both counts (the IPD and the eyes' position against the anchor), as this morning's vertical 3 cm had suggested. The run-to-run variation is the new fact: the shimmer's strength changes from one space to the next without a setting change, so its cause is something the device supplies differently each time (the tracking's jitter, which the engine's per-frame pose follows and RealityKit's render does not have to; the GPU's state and the frames' timing; the device's own foveated render), not a constant of the grid's geometry alone. The static kinks at rest remain unexplained by any parallax (no eye offset moves them) and are cell-periodic; what is left for them is the cell's own mapping of a sliver's texels where the two triangles of a cell lie in different planes, which from the engine's exact eye is invisible and from any other eye is a kink, so a per-frame eye difference of the tracking's size, variable from run to run, would give both the kinks and the shimmer. A diagnostic panel over the whole day's evidence is run overnight for a ranked list of mechanisms and the device experiment that separates them. +- **Matrix, step 3 (2026-10-19, 18:53): the skirt at 0 and 8 changes nothing; the tear threshold is the first knob that moves the shimmer, and at 3 m it still shimmers.** The tear mask is a binary decision per cell per frame (a cell stretched across a depth jump above the threshold is discarded), so its boundary runs along cell edges, a polyline at rest, and cells flip in and out of it as the pose moves the silhouette across them, waves under motion; the skirt moves depths, not that decision. At 3 m the cube's own 20 cm jumps (a top face against its front) no longer tear, which lessens it; the jump from any silhouette to the 100 m shell exceeds any threshold the stepper offers, which is the part that stays. Why the Mac did not show it is still to be explained (the straightness scene ran with MSAA from the engine's eye; the discards may have fallen on texels of alpha 0 there, or the device's depth resolve or foveated raster puts the decision elsewhere). The direct test for tomorrow needs a build: a tear threshold beyond the shell (no tears at all) and, if the shimmer goes with it, a tear decision with hysteresis in time, or taken on the texture's alpha rather than on depth alone (a stretched cell whose texels are all transparent need not be decided at all), or the stretched cells blended out by the band rather than discarded. The shimmer's run-to-run variation fits a decision taken on noisy depth samples near a threshold. From 9cea8027dfbb870f20efa7b4da84c0ed5ce166ee Mon Sep 17 00:00:00 2001 From: Javier Segura Date: Mon, 19 Oct 2026 19:11:54 +0200 Subject: [PATCH 72/72] [Docs] XRRealityKitPresenter: the no-tears build (tears switch, what to read, the remedies) --- docs/proposals/XRRealityKitPresenter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/proposals/XRRealityKitPresenter.md b/docs/proposals/XRRealityKitPresenter.md index 3d9233171..22abb2741 100644 --- a/docs/proposals/XRRealityKitPresenter.md +++ b/docs/proposals/XRRealityKitPresenter.md @@ -264,3 +264,4 @@ Done: CoolSaber playable with a Persona; the finding recorded in `SpatialPersona - **Matrix, step 1 (2026-10-19, 18:43): the kinks on the cube tops look the same at IPD 55, 63 and 70 mm.** Fifteen millimetres of eye separation, 7.5 per eye, would move a parallax kink through a wrong-depth cell by about a display pixel at the slivers; nothing moved. The eye separation is cleared; the view offset along each axis is the remaining half of the eye hypothesis, then the rim vertices' depths (skirt, tear). - **Matrix, step 2 (2026-10-19, 18:48): the kinks look the same with the view offset at +3, 0 and -3 cm along z; and the edges shimmered less in the previous space than in this one, with nothing changed between them.** The eyes are cleared on both counts (the IPD and the eyes' position against the anchor), as this morning's vertical 3 cm had suggested. The run-to-run variation is the new fact: the shimmer's strength changes from one space to the next without a setting change, so its cause is something the device supplies differently each time (the tracking's jitter, which the engine's per-frame pose follows and RealityKit's render does not have to; the GPU's state and the frames' timing; the device's own foveated render), not a constant of the grid's geometry alone. The static kinks at rest remain unexplained by any parallax (no eye offset moves them) and are cell-periodic; what is left for them is the cell's own mapping of a sliver's texels where the two triangles of a cell lie in different planes, which from the engine's exact eye is invisible and from any other eye is a kink, so a per-frame eye difference of the tracking's size, variable from run to run, would give both the kinks and the shimmer. A diagnostic panel over the whole day's evidence is run overnight for a ranked list of mechanisms and the device experiment that separates them. - **Matrix, step 3 (2026-10-19, 18:53): the skirt at 0 and 8 changes nothing; the tear threshold is the first knob that moves the shimmer, and at 3 m it still shimmers.** The tear mask is a binary decision per cell per frame (a cell stretched across a depth jump above the threshold is discarded), so its boundary runs along cell edges, a polyline at rest, and cells flip in and out of it as the pose moves the silhouette across them, waves under motion; the skirt moves depths, not that decision. At 3 m the cube's own 20 cm jumps (a top face against its front) no longer tear, which lessens it; the jump from any silhouette to the 100 m shell exceeds any threshold the stepper offers, which is the part that stays. Why the Mac did not show it is still to be explained (the straightness scene ran with MSAA from the engine's eye; the discards may have fallen on texels of alpha 0 there, or the device's depth resolve or foveated raster puts the decision elsewhere). The direct test for tomorrow needs a build: a tear threshold beyond the shell (no tears at all) and, if the shimmer goes with it, a tear decision with hysteresis in time, or taken on the texture's alpha rather than on depth alone (a stretched cell whose texels are all transparent need not be decided at all), or the stretched cells blended out by the band rather than discarded. The shimmer's run-to-run variation fits a decision taken on noisy depth samples near a threshold. +- **The no-tears build (2026-10-19, 19:10; engine `feature/xr_realitykit_presenter` at `ead5ee48`, example `feature/realitykit_presenter_example` at `0a41427`, both pushed; 65 tests in the presenter filter, one new).** The test of the tear decision, for the headset. `XRRealityKitPresenterConfiguration.tears` (`XRPresenterSettings.tears`, live while the space is open): off, the unprojection kernel leaves every vertex's tear mask zero (flag bit 2), so no cell stretched across a depth jump is discarded and the grid stays one surface. The triangles between an object's rim and what lies behind it are drawn, and since each vertex's uv is its own projection they read the texels between the two, the engine's render around the silhouette, transparent but for the rim's partly covered texels, which the alpha test cuts and the band blends. The threshold keeps its other job, the placement (the texel nearest the vertex's centre on the near side of a footprint's depth jump) and the rim flags, so the two runs differ in the decision alone; a threshold past the shell, the first idea, would also have moved every vertex onto its centre texel and taken the rims and the skirt with it. The example has a "Tears" toggle beside the threshold's stepper and `--no-tears`; the settings line says NO TEARS. Release built and installed (the launch from the Mac timed out as usual; it opens from Home). What to read: the cube tops at rest and under a slow turn with Tears off against on, beside the control cube. If the shimmer goes with the tears, the remedy is a tear decision that cannot flicker: hysteresis in time (a cell torn only after some frames on the same side of the threshold), or the decision taken on the texture's alpha with the depth (a jump into transparent texels needs no tear, the alpha test cuts it), or the stretched cells handed to the band and blended out instead of discarded. If it stays, the tear decision joins the cleared knobs and the panel's list takes over. What a no-tears space may show on its own: under the cut, the rim's silhouette rides a stretched surface whose parallax from the predicted to the shown pose is not the object's (a fraction of a pixel with the lead right); under the blended core with its depth kept, transparent texels across the stretch may write depth and hide what RealityKit draws behind them, the control cube included.