Repository navigation
Sync fork develop with upstream: 0.23.0, glass against frames, specular AA, project layout (#1366), plus the headset rendering PRs for testing - #76
Merged
Conversation
…toldengine#1331) * [Bugfix] Glass keeps its place against its frame and in a headset Seen on an Apple Vision Pro with a BIM site, through the editor's Mac-rendered preview, and fixed on a try-out before coming here. A pane modelled into its frame shares a plane with the frame, and the two fought for depth along the pane's edges at every move of the head (on a Mac, a noisy edge whenever the camera moves). Glass now draws a few units of depth resolution behind where it is, and two more per unit of slope, under either depth convention, so the frame wins wherever the two coincide. The side of a pane the viewer sees was decided from the smooth shading normal, which bends at a pane's edges, so an edge pixel could count as the far side for one eye and the near side for the other. The face itself decides now: the derivative normal, with the vertex normal only choosing which way it points, and the shading normal for a degenerate face. The shading normal of a pane always faces the viewer, whichever of the two goes draws the face. A headset's compositor reprojects every pixel by the depth it finds. A pane that mostly shows itself (dark, frosted: less than half of the light crosses it, Material.paneShowsItself) now writes its depth outside passthrough, so its tint and reflections stay on the pane; a clear pane leaves the depth of what is seen through it, which is what the eye follows. Before, every pane carried the depth of what was behind it and dark car windows flickered. Tests: a glowing block whose near face lies in the plane of a glass cube's near face shows the block in every pixel of the middle (it fails without the bias); the depth rule on clear, smoked, frosted, half-there and grey glass. The seven shader libraries are rebuilt. * [Patch] Name the head-on transmittance of a non-metal that the pane rule uses The 0.96 in Material.paneShowsItself is one minus the dielectric reflectance of 0.04 that TransparencyShader.metal takes as F0, collapsed to its head-on value since the rule has no view angle or roughness per pixel. It is a named constant now, dielectricHeadOnTransmittance, with a comment pointing at the shader, so a change of the shader's F0 does not leave the rule behind. --------- Co-authored-by: Javier Segura <javier@miolabs.com>
Normal detail finer than a pixel (a knurled dart barrel, sisal fibres, the curvature of a thin wire) gives a tight specular lobe a different peak every frame as the view moves, which reads as sparkle on Vision Pro, where the head never holds still. Multisampling resolves coverage, not this. The model fragment shader now widens the roughness it writes to the G-buffer by the screen-space variance of the shading normal (Kaplanyan 2016; Tokuyoshi and Kaplanyan 2019): sigma^2 = 0.25, kernel clamped at 0.18, the values Unreal and Frostbite ship. A flat surface sees no change; the light pass is untouched because it already reads roughness from the material attachment. Kernels rebuilt for every platform. Co-authored-by: Javier Segura <javier@miolabs.com>
* [Release] preparing release 0.23.0 * [Docs] Fill in missing v0.23.0 CHANGELOG entries The release commit only captured half the commits since v0.22.0; add the fixes, performance, enhancement, and chore/test entries that were left out.
…EADME (untoldengine#1349) * Re-host starter asset packs on GitHub Releases, fix CLI install bundling The AWS/CloudFront distribution serving starter asset packs was decommissioned, breaking `untoldengine assets install`. Point manifest.json at the assets-v2 GitHub Release instead, with corrected pack sizes. Also fix install-untoldengine-create.sh: it only copied the untoldengine binary to /usr/local/bin, never the sibling UntoldEngineCLI_UntoldEngineCLI.bundle SwiftPM generates for manifest.json, so `assets install`/`assets list` failed with "Asset manifest not found in CLI bundle" on any installed copy. * Promote the Untold Editor in the README/docs landing page, refresh intro video, restructure direction section Move "Try the Untold Editor" ahead of the demo/code-first sections in both README.md and docs/index.md so visual, no-code scene authoring is visible before the Swift/Metal quick start -- the Editor is now stable enough to lead with for game-dev-focused readers. Swap the intro video/thumbnail from the old CoolSaber demo to the "State of the Engine 2026" video. Replace "Core Direction" with "Our Direction": Vision, Mission, and an Engineering Principles summary linking to docs/Contributor/EngineeringPrinciples.md.
…ntoldengine#1345) * [Project Layout] Introduce shared ProjectLayout type Replaces the 5 independent copies of the Sources/<Project>/GameData path shape and the 9-category GameData folder list with a single ProjectLayout type that both engine and CLI call sites go through, and that supports detecting either the legacy nested layout or the upcoming root-level one. Fixes untoldengine#1334 * [Project Layout] Bundle GameData from project root in templates & XcodeGen spec New projects now get GameData, Resources and Docs at the project root instead of GameData nested three levels deep under Sources/<Project>/. Package.swift keeps its code under Sources/<Project>/ (per the SwiftPM spike in untoldengine#1333) and scopes sources/resources explicitly rather than moving the target itself. The XcodeGen spec bundles GameData as a root-level folder reference for every platform target. Also folds in the .gitignore and README fixes from untoldengine#1336, since they touch the same templates. Fixes untoldengine#1335 Fixes untoldengine#1336 * [Project Layout] Resolve GameData under new or legacy layout in update/assets install update and assets install already located GameData by name rather than by a fixed prefix, so they worked against the root-level layout unmodified. findGameDataDirectory did have a real bug though: it returned on the first match instead of continuing to search, so a project with both a root-level GameData and a stale legacy Sources/<Project>/GameData (mid-conversion) would silently resolve to just one of them instead of surfacing the existing ambiguity error. Fixes untoldengine#1338 * [Project Layout] Make pack manifest's sourceAsset a resolvable relative path sourceAsset was a bare filename nothing read back. The exporter now writes it as a path relative to the pack's own directory (the same convention models' own "path" already uses), via the existing relative_asset_uri() helper, so it resolves correctly whether the source still sits beside the pack or lives in a separate Resources/ tree. Added UntoldPackData.resolvedSourceAssetURL(packURL:) on the read side; a bare filename is just the degenerate case of a relative path with no intermediate directories, so one resolution path handles both old and new manifests. Fixes untoldengine#1339 * [Project Layout] Add untoldengine convert command Moves an existing project from the legacy layout (GameData nested under Sources/<Project>/GameData) to the root-level one: moves GameData, creates empty Resources/ and Docs/, writes a .gitignore if missing, rewrites project.yml via a line-based patch (so hand-edits beyond the GameData folder reference survive) and regenerates the Xcode project, then checks that every pack's sourceAsset still resolves -- which it does without any rewrite, since GameData and every pack (plus whatever source currently sits beside it) move together as one unit. Refuses to run if the project is already converted or the legacy GameData doesn't exist. Fixes untoldengine#1337 * [Project Layout] Update docs for root-level GameData path Every Sources/<Project>/GameData path and generated-tree diagram now reflects GameData/Resources/Docs/Sources at the project root. Also fixed UsingAssetPaths.md's category table, which was missing LUT/ (a gap independent of the layout move), and documented the new untoldengine convert command in UsingUntoldEngineCLI.md. Fixes untoldengine#1340 * [Project Layout] Add ProjectLayout.detect(gameDataDirectory:) reverse lookup The editor only has assetBasePath (the GameData URL), not the project root/name it came from. Found the editor had independently reinvented this same inference five times (EditorController.projectName, EditorView+Project.openProject, CreateProjectView.startBuild, ComponentSourceLocator.projectRoot(forAssetBasePath:), EditorBuildTargetSettings.projectRoot(of:)), all hardcoding the legacy 3-levels-up assumption. One shared reverse lookup for untoldengine#168 to adopt instead of patching all five independently. Also fixed a stale comment on the forward detect(projectRoot:projectName:) that still said new projects default to the legacy layout -- untoldengine#1335 changed that. * [Chores] fixed formatting * [Bugfix] Roll back GameData move and flag unconvertible project.yml in untoldengine convert If project.yml, Resources/Docs creation, or xcodegen regeneration fails after GameData has already moved to the project root, convertProjectLayout now moves GameData back to its legacy location instead of leaving the project half-converted. It also warns instead of reporting success when a hand-edited project.yml still references the legacy GameData path but doesn't match the generator's exact textual shape. * [Chores] fixed formatting
…ntoldengine#1356) Add a Common Failures table to UsingCodeComponents.md covering issues found in packaged/generated-game testing (final members, discoverInApp bundle scanning, dlopen path matching, mask(for:) hiding pending destruction, packaging script pitfalls, sRGB marker textures), kept as a reference since the fixes explain existing API constraints. Note in EditorCodeComponents.md that the UntoldEngine side of feature/component_kit is merged to develop, and flag that the UntoldEditor-side branch and UntoldGaussianTwins adoption (stage 5) still need separate re-verification.
…te maps (foveation) With foveation enabled in the layer configuration, visionOS hands each eye a rasterization rate map and textures of its physical size; the engine ignored the maps, so apps had to keep foveation off, which also forbids a render quality above the platform default. The stereo path now attaches the eye's map to every pass that rasterizes the scene at drawable size and sets the eye's screen-space viewport there; the full-screen passes keep copying physical pixels one to one. SSAO rebuilds view positions from a texel's screen position, and the occlusion culls (mesh, Gaussian chunk, per-splat) move their projected rects into the pyramid's physical layout through the map the pyramid was drawn with, so culling stays on under foveation. The Gaussian fragment takes its offset from the splat centre from a varying rather than from its position. On by default when the drawable provides maps; `.foveatedRendering(.disabled)` ignores them for diagnosis. Kernels rebuilt for every platform.
A layer configured with foveation or a render quality aborts the app when
the frame is queried with the single-drawable queryDrawable() ("BUG IN
CLIENT: called cp_frame_query_drawable() which does not support setting
render quality or fovea"). The frame now takes the drawable array on
visionOS 26 (the old call on visionOS 2), presents every drawable, and
draws the scene into a capture drawable only while its textures match the
frame's resources; otherwise it is presented cleared rather than
reallocating every sizeable resource twice a frame.
… map is attached Under a rasterization rate map Metal reads a descriptor's renderTargetWidth and renderTargetHeight, like the viewport, in screen pixels. The scene descriptors carried the physical texture size there, which clipped every foveated frame to the top-left physical-size corner of the screen: on the headset a screen-fixed rectangle, different in each eye, outside which the board vanished. The attached descriptors now leave the size at 0 (the attachments' own) and get the texture size back when the map is detached; the eye's viewport is taken from the map's screen size.
The projection is jittered by a sub-pixel Halton offset before each frame's graph is built, and a resolve pass after the look pass blends the frame with the eye's history, reprojected through the depth buffer and the two frames' unjittered view-projections and clipped to the colour box of the pixel's neighbourhood. Sub-pixel detail a single sample flickers on — thin wires, fine normal maps, the periphery of a foveated frame where a texel spans several screen pixels — averages out over frames. Under a rasterization rate map the reprojection goes physical → screen → world → previous screen → previous physical through both frames' maps. Camera motion is handled; moving objects rely on the clip. Kernels rebuilt for every platform.
…rate map A pixel-thin feature — a wire, a line of text — is rasterized in some jittered frames and not others, and a min/max box of a frame that missed it clipped it out of the history, so it flickered at distance; the clip box is now the 3×3 mean ± clipGamma (1.25) standard deviations. Under foveation a spot the gaze has just moved onto has a history from the periphery, coarser than the new frame, which showed as a shadow fading over twenty frames; the history's weight is cut by the ratio of the two maps' local rates. Kernels rebuilt.
…and moving-camera tests A pixel at the clear depth — the background pass, a transparent quad drawn over passthrough — has no position to reproject from; fetched from the far plane its history ghosted or wiped it under a moving camera (the HUD pill and label flickered on and off). Such a pixel now keeps its own frame. The history is sampled with a Catmull-Rom filter (nine bilinear taps), which resampling every frame smears less than bilinear. The rate map on renderInfo.xrFoveation now also goes on the scene passes of the macOS path, so the foveated resolve is testable on a Mac: the new tests draw a still scene through a rate map whose dense zone jumps every frame, and move the camera, in both temporal modes; the residuals sit at the held-still floor. Kernels rebuilt.
The model pass draws without culling, so an open mesh shows its back faces — a dart's flight, a leaf — and those arrived with the front side's normal, pointing away from the viewer, and shaded as if unlit. A face whose normal points away from the viewer is now lit with it turned around. The test is geometric rather than [[front_facing]]: the pass sets no winding and assets arrive wound either way. Kernels rebuilt.
… or the depth pyramid On visionOS 26 a frame carries a second drawable while the view is recorded or streamed. The engine drew the scene into it as into the built-in display, so each eye's temporal history and the occlusion pyramid were overwritten from the capture's pose on every frame: the display's next resolve read a history that alternated between its own pose and the capture's, and thin objects blinked while recording. A secondary drawable now resolves its frame as is and leaves the history, the matrices, the map and the pyramid to the built-in display. The frame's drawable count is logged when it changes.
…arsest A rate map is coarser in one axis than the other wherever its rows and columns differ, and Metal quantizes the requested rates, so a frame can be 0.6 texels per screen pixel in y where the history's frame was 0.25. The history weight was scaled by a ratio taken from the larger step of the two axes and judged such frames equal in rate, keeping a history 2.4× coarser at full weight: its blurred edge overrode the frame's and a thin object lost its edge rows on alternate frames (a headset HUD strip blinked at the one distance where it sat on such a zone). The ratio is now per axis, the smaller taken. The moving-map test now draws a map coarse in both axes and asserts, with the jitter off, that no lit pixel loses its colour to a history drawn through the other map. Kernels rebuilt.
…GPU time per frame, the tile light stage, opaque geometry without discards Six branches of 2026-10-12/13 on top of foveation and specular anti-aliasing: skip the Gaussian pass on a frame without splat entities, skip the pre-composite pass on a passthrough frame it would only copy, TAA resolving into the history alone, a frame's GPU time beyond the previous frame's, the tile light stage with the multisample-count setting and the up-front tile-memory check, and opaque geometry through a model pipeline without discards. Kernels rebuilt for every platform.
Two switches, off by default, that take one kind of shading out of the combined pass and leave the rest of the frame as it is, so a measurement pass with one on gives the frame's cost without that shading. Flat material: the opaque draws go through the plain model pipeline compiled with the material shading out (the material's factors and the vertex normal, no texture sample, no derivative). Flat lighting: the light quad compiled to return the albedo untouched, no image-based light, shadows or lights; the tile stage stands aside. Both are function-constant specialisations of the existing shaders (`modelFunctionConstantFlatMaterial`, `lightFunctionConstantFlatShading`), selected through `setRendering(.flatMaterialExperiment)` / `.flatLightingExperiment`. Kernels rebuilt for every platform.
The model pass draws both sides of every triangle, so a closed mesh sends its back faces through the tiler and the rasteriser too. As an experiment, `setRendering(.backFaceCullingExperiment(.counterClockwiseFront))` (or `.clockwiseFront`, for an asset wound the other way) culls the faces wound the other way in the G-buffer pass; the light quad is never culled, and off, the default, the pass draws as it always has. An open mesh loses its back with it on: the product version needs a per-material double-sided flag from the exporter.
`GPUPassTiming` carries a counter sample buffer through each frame's command buffer: every render pass writes a timestamp at the start and end of its vertex and fragment stages, every culling compute pass and the depth copy at their bounds, and the buffer is resolved when the command buffer completes. Each pass's time is averaged over the recent frames (`averages()`), the last frame is available as it came (`lastFrame()`), and the same label in one frame, one pass per eye, adds up. The GPU's clock is calibrated against the media clock, since its ticks are not documented. Off by default, `setRendering(.gpuPassTiming(.enabled))`; nothing is measured, and nothing changes, on a GPU without stage-boundary sampling. This is the per-pass view a Metal System Trace gives, on a device that cannot record one.
Under MSAA the deferred light quad runs once per sample — a fragment function that reads the framebuffer is invoked per sample whether or not it declares a sample index — so every pixel's image-based light and cascade shadow were evaluated four times at 4×. The model fragment shader, which runs once per fragment, now evaluates them and carries them in the emissive attachment (the indirect light added to the emissive, the shadow factor in alpha), and the light pass adds them from there, keeping only the direct lights per sample. A function constant on each shader selects it (`modelFunctionConstantIndirectInGBuffer`, `lightFunctionConstantIndirectFromGBuffer`); the model and light pipelines follow `setRendering(.perFragmentIndirectLighting)` on the next frame, enabled by default. The picture is the same within the G-buffer's precision, and the tile light stage stands aside while it is on. Kernels rebuilt for every platform.
…de something Every frame copied its opaque depth and built the occlusion test's pyramid from it, whether or not the test would run on the next frame: with the test disabled (a camera step of zero) or a scene of one mesh entity, which hides nothing but itself, that was the frame's cost for nothing. The copy and the build now run only when the test is on and the culling set has at least `HZBOcclusionCulling.minimumEntities` mesh entities (two by default, `setRendering(.occlusionCullingMinimumEntities(_:))`) or the frame has splat entities, whose cull reads the pyramid; a frame without a pyramid marks it invalid, so no test reads a stale one.
…would only copy it With temporal anti-aliasing the output transform pass read the resolve, encoded it for the display and copied the opaque depth for the compositor: a full-screen pass per eye to move the frame one texture over. The resolve now does that itself (`fragmentTAAPresentShader`): the display's colour texture is its second colour attachment, the display's depth its depth attachment, the encoding and the far-depth clamp are the output pass's, and the output pass stands down for the frame. Only when nothing is registered between the resolve and the output in the graph, the view is the lit one, and the present textures exist at the resolve's size; otherwise the frame runs as before. Kernels rebuilt for every platform.
…t quad stands down The combined G-buffer + light pass lit every multisample of every pixel through the light quad, reading the G-buffer from the tile: under 4× MSAA four evaluations of the direct light per pixel, and under a varying rasterization rate map (XR foveation) each of them at the map's multiplier. The indirect light moved to the geometry a commit ago; now the whole of the light does: the model fragment shader evaluates the sun through its cascade shadow, the point, spot and area lights with theirs, over the indirect light it already carries, and writes the lit colour to the attachment the quad would write. The quad then has nothing to do and the pass ends after the geometry — the forward way to shade, once per fragment, which under MSAA is once per pixel but at the edges. - `computeDirectLighting` (LightShader.metal, declared in ShadersUtils.h): the direct part of `lightTBDRSample`, factored out for the geometry to share; the quad's per-sample path calls it and is unchanged in what it computes. - `fragmentModelShader`: function constant `modelFunctionConstantLitInGBuffer` (`kModelLitInGBuffer`, index 5) binds the light blocks, the LTC tables and the point and spot shadow maps at new model-pass indices, and writes `GBufferOut.lit` ([[color(5)]]); the pipeline masks the attachment off when the constant is false. `kModelSunNeeded` keeps the sun's parameters bound for either per-fragment mode. - `renderInfo.perFragmentLighting` (on by default; needs `perFragmentIndirectLighting`, and stands aside for the flat-lighting experiment, which runs through the quad), `perFragmentLightingBuilt`, and `setRendering(.perFragmentLighting(.enabled | .disabled))`; the pipelines follow the setting on the next frame as the indirect one does. - The combined pass uploads the point, spot and area lights once, before the geometry, and binds them to whichever lights with them; with the geometry lit, the tile light stage stands aside and the quad's sub-pass returns after ending the encoder, on device and in the simulator. - Tests: PerFragmentLightingTests — the same picture as the quad at one sample and under 4× MSAA (a box over a slab under the sun, a point light and the image-based light), the frame one draw short with the quad gone, and the point light there. - All seven metallibs rebuilt.
… what its fragments read With the geometry lighting itself per fragment, nothing read the five G-buffer attachments, yet the pass still carried them in its imageblock: 48 bytes per sample at 4× MSAA, which holds the GPU to 16×8 tiles. The opaque pass now has a second descriptor for that mode — the lit colour in the light pass's slot and the depth, 12 bytes per sample — and the model pipelines built for it declare that attachment alone: the shader's G-buffer outputs are compiled out. On Apple GPUs every interpolated float the vertex stage emits is written to the parameter buffer by the tiler and read back per tile, so the vertex-to- fragment layout is a vertex-stage cost in itself (measured on the board: the stage halves with the material's inputs gone). `VertexOutModel` carried the object-space position and normal, which the fragment stage transformed again, beside the world-space normal it also carried, and three members nothing ever wrote. It now carries the world position, the world normal, the tangent and the uv: twelve floats beside the position. - `GBufferOut` lives with its function constants in modelShader.metal; each member is a function-constant output: the G-buffer's by `kModelGBufferOut` (the geometry not lighting itself), the lit colour by `kModelLitInGBuffer`. - `renderInfo.forwardRenderPassDescriptor` (built beside the G-buffer's, foveated like it); `renderInfo.opaquePassDescriptor` picks the descriptor the pipelines were built for, which the combined pass and the two stand-alone model passes take. The combined pass labels itself "Forward", skips the G-buffer attachment setup and the tile stage in that mode. - `perFragmentLightingWanted` also stands aside for the G-buffer debug views, which read what the forward pass does not write. - `vertexModelShader` transforms the position to world space once; the model, transparency and gizmo fragment stages take it and the world normal as they are. - All seven metallibs rebuilt.
…engine stats The XR loop recorded its update phase and its whole period, which includes the wait for the compositor's optimal input time, so the CPU's own work per frame could not be read from the stats. `EngineTimingStats.xrWaitMs` is now that wait, `encodeMs` the submission phase (the drawables' command buffers encoded and committed), and `updateMs` + `encodeMs` the CPU's work. The loop also prints the compositor's frame timing once: optimal input time to rendering deadline, and to presentation.
An app's Debug configuration builds the engine package at -Onone, and a frame that fits comfortably with the engine optimized is then bound by the CPU encoding it: on a headset the opaque draws took five times longer to encode and the GPU downclocked while it waited, so a measurement timed the compiler. `EngineBuild.isOptimized` reads the engine module's own assert configuration, never inlined, for a measurement to print it.
…ess settings say The cascade shadow filtered every fragment with the whole 16-point Poisson disk, a hardware-compared sample each: on the board under a foveal map that was a fifth of the whole opaque fragment stage (Mac, release: 0.63 ms with 16 taps, 0.52 with 8, 0.49 with 4). `ShadowSoftnessSettings.filterTaps` (16 by default, else 8 or 4) now picks every second or fourth point of the disk, which keeps its spread with fewer compares and leaves a grainier penumbra that a temporal resolve smooths. On a headset `xrFilterTaps` applies, 8 by default: the frame is drawn twice at a high resolution under a temporal resolve, and the whole disk cost 1.5 ms of an 11 ms frame there (judged acceptable at 8 on the board). - `CSMUniforms.shadowFilterTaps`, in the Metal struct and its Swift mirror, carried by `ShadowSystem.makeUniforms`. - Tests: the headset takes 8 taps by default and the Mac 16; a count asked for is the disk or every second or fourth point of it. - `sampleCSMCascade` runs one of three loops with a constant trip count, which the compiler unrolls: a single loop over a count read from the uniforms ran a quarter slower at 16.
…ives and the history with five taps Under foveation the resolve decoded each rasterization rate map twice per pixel: once for the pixel's position and once a pixel over, for the map's local rate, which weights a history coarser than the frame. The second decode of each map is gone: the mapping is now computed for every pixel before any pixel leaves, and the rates are the derivatives of it across the 2×2 quad — this frame's screen position per texel, the history's texel per screen pixel. The history's Catmull-Rom filter takes the five taps that carry its weight instead of nine; the four corner taps weighed a few thousandths at most, and the five are renormalised. - Same picture on the Mac within the temporal tests' tolerances (the still scene's settling test fails as it did before this change, at 0.00428 against 0.004; it failed at that figure on the branch's integration commit). - All seven metallibs rebuilt.
…last frame, and the late-frame line shows the loop's split Asked for two refreshes, the pacing still took three: its wait was aimed at a presentation instant on the refresh grid, and a query made there landed just after the compositor's hand-out for that refresh and blocked until the next one, a refresh lost on every frame. The pacing now waits until the cadence's refreshes after the compositor handed it the last frame, less three milliseconds, and lets the query's own block align the frames to the refreshes. The per-second line adds how long the query blocked, how long the loop took between frames, how far after the query the presentation was predicted and how far before it the deadline sat, so a run shows where a paced frame's time goes.
…art, and the line reports the presentations' spacing The compositor hands out a frame on demand (the query blocked 0.0 ms) and predicts its presentation about three refreshes after the query. A wait aimed at the previous frame's presentation instant therefore made every cadence three refreshes; a wait of a cadence less a three-millisecond lead spaced the frames 16.6 ms apart at a cadence of 20, shown two refreshes apart, then one. The queries now fall on the pacing's own grid, the instant it started from, every cadence exactly, a query up to a quarter refresh late still taking its slot. The per-second line adds the mean, shortest and longest spacing between consecutive predicted presentations, so a run reads its evenness instead of assuming it.
…past the deadline's margin, and the pose lands by that margin
The compositor's rendering deadline sits 15 ms before the presentation on the
headset, so a frame a few milliseconds past it is still shown on time: the
full-rate runs with nearly every frame a millisecond late dropped nothing, and
the heavy run's paced frames, 3 to 6 ms past the deadline, were posed a refresh
ahead of where they were shown ("landing +1"). The policy now takes the deadline's
margin into account, less a 2 ms safety (XRHalfRatePolicy.usableMarginMs):
- The pose lands on the compositor's prediction while the frames finish within
the usable margin, and as many refreshes later as the rest of the mean overrun
reaches.
- The pacing enters when, with no wait left to skip, the display skipped a
refresh for one frame in ten or more (counted in the loop from the spacing of
consecutive predicted presentations, beyond the pacing's own cadence) or one
frame in ten or more finished past the usable margin, for two seconds running.
Frames merely past the deadline no longer count: the heavy run's second before
it paced had 90 frames a millisecond past it, shown on time, and fewer hops
than one in ten are fewer than the steps the half rate adds.
The loop counts both per second (secondMissed, secondSkipped) and prints them on
the per-second line ("N past the deadline's margin, N refreshes skipped"), which
now also appears for a second with skipped refreshes and no late frame; the GPU
completion handler receives the drawable's presentation time for the margin.
The proposal's pacing paragraph describes the final mechanism (the fixed grid,
the landing, the four lessons from the device) and records the by-eye result of
the heavy run: the even 45 fps shows fewer jumps than the hopping full rate, and
a visible stepping remains, the frame rate itself; the cure for that
configuration is the frame's GPU cost, not the timing.
…odel A model loaded from a file is one entity per node under the entity it was set on, so the viewer's single dartboard put many mesh entities into the culling set, and the pyramid's gate (minimumEntities, 2 by default) built the depth copy and the pyramid every frame, and ran the test for an object that hides nothing but itself: on the headset 1 to 2.5 ms of a 13 to 15 ms frame. The culling set now counts objects: an entity with its children, the snapshot's object(of:) being the entity's scenegraph parent or the entity itself. Both culling paths set hzbCandidateCount to that count; the gate, the setting and the accessor are documented in those terms. A render test puts three parts under one entity beside a box and sees two objects: the pyramid at a minimum of two, none at three, where four entities would have built it.
…draw The pass made its encoder, loading and storing the lit colour and the depth of the whole frame, before it looked for a transparent surface: with none, every eye paid the encoder, its bindings and three light uploads to draw nothing. The draw list is gathered first now and the pass returns when it is empty, as the wireframe and occluder-shell passes do. A render test reads the GPU's own pass labels: no "Transparency" with an opaque box, the pass once its material blends.
…ge, not their vertex stage Every scene pass waited on the shared fence before its vertex stage, after the previous pass updated it after its fragment stage: the tiler of one pass could not run while the renderer of the previous one shaded, and the Forward pass's binning of a model's many parts (3.5 ms of vertex work on the headset) sat behind the shadows' and culling's fragments. A pass's vertex stage reads nothing a previous pass's fragments wrote (meshes, quads, joint streams), and every resource is hazard-tracked, so the wait moves to the fragment stage at all 23 sites, as the temporal resolve already had it; every update stays as it was.
…ay compress them The lit colour (the MSAA resolve target), the environment, the composite, the look output and the anti-aliasing output were created with shaderWrite usage, which no kernel uses and which, per Apple's texture guidance, forbids the GPU's lossless compression of a texture: every pass that resolved into, read or wrote them moved uncompressed pixels. They keep their shared storage, so the tests' CPU reads stay as they were.
The look pass drew through the post-process descriptor, which carries the frame's depth loaded and stored for the passes that read it as an attachment; the look pass never tests or writes depth, so each eye loaded and stored a full-size depth texture for nothing. It has its own descriptor now, colour alone, its pipeline built with no depth attachment; executePostProcess takes the descriptor to draw into and sets the depth load only when there is one. The other post-process passes keep the frame's depth. A render test checks the descriptor and that the look output still carries the lit frame.
The viewer's sun switch leaves the directional light in place at intensity 0 with castsShadow on, and the cascades were drawn every frame regardless: two 2048x2048 depth maps of every part of the board, 0.5 to 2.9 ms a frame on the headset with no sun. The shading multiplies the sun's shadow by its intensity, so the cascade update now refuses a sun at zero and the shadow passes stand down; the picture is the same. A render test dims the scene's sun to zero and sees no "Shadows" pass, then lights it again.
…um, and the frame's own GPU time The per-second line named only the three heaviest passes, so the device's numbers could not be read against the frame's span: the shadows, the look, the culling and the transparency passes were timed and never shown. A second well past the deadline now lists every pass, heaviest first, with their sum (a label sums both eyes); the rest of the span lies between passes. The GPU figure adds the frame's own time after the frame before finished: a queued frame's command buffer is taken before the previous one is done, so the raw span overstates the frame's work.
…at held, what remains in the shaders
…set starts without one The renderer's setup made a camera and a directional light for every path, the headset included: a sun at intensity 1 casting cascades in a passthrough room that has its own light, until the app replaced it. The defaults are one function now (createDefaultEntities), the camera on every path, since the headset writes the head pose into it, and the sun by UntoldRendererConfig.defaultSun: unset, the platform decides, a sun everywhere but the headset, where the passthrough room is lit by its probe and the immersive environment by its own light; a project sets it to keep a sun on the headset (for shadows its environment cannot cast) or to start without one elsewhere. A render test covers both cases, and the XR architecture doc's comparison table has the row.
… pixel: the sun by its intensity, the maps by their flags, the lookups once, the resolve's early-out On the headset the Forward pass took 8.0 ms for both eyes (5.5 of it fragment work) and the temporal resolve 2.1 against an 11.1 ms refresh, with a sun the viewer switches off, materials without every map, a room probe that is never rotated, and a frame that is mostly cleared. Four cuts in the shaders, each a branch on a per-draw or per-frame constant (uniform across the draw) or a value computed once: - The sun's cascade shadow taps and its Cook-Torrance BRDF are evaluated only when the sun's intensity is not zero: in emissiveWithIndirect and computeDirectLighting (the forward path and the tile light pass), the texture light pass, lightTBDRSample and the transparency shader. Any other intensity takes the old path in the old order of products, bit for bit. The CPU's cascade gate (ShadowSystem.updateCascades) says the same: not zero. - Each material map is sampled only when the material has it (base colour, normal with its decode and tangent transform, roughness, metallic, emissive); the flags are per-draw constants, so the implicit-derivative samples stay in uniform control flow, as the height-map branch already was. The reviewers put this at nothing measurable (the removed samples were on nil textures); it stays as a cleanup. - The view vector, the environment's rotation in radians and the BRDF lookup table's pair are computed once per pixel (brdfIntegration, with both reflectances derived from the pair; the texture-taking signatures stay as wrappers); the rotation of the lookup direction is skipped at angle zero, the room probe's case, with the normalize kept: the simulator's light pass hands in a G-buffer normal as stored, and the equirectangular latitude's asin is not scale-invariant. That asin is clamped now, which removes a pre-existing chance of a NaN pixel at the poles. - The temporal resolve returns early for a pixel with no history or at the clear depth, before the rate-map decodes, the reprojection and the velocity fetch. The foveation rates are read as texels per screen pixel over the same screen interval from both maps, a second decode of each, instead of derivatives across the quad, which forbade any early return: the ratio is exactly 1 wherever the two maps agree, zone boundaries included; where they differ, a band at most a pixel wide before a boundary weighs a little differently than the quad's span did. The kernel libraries are rebuilt for every platform (buildkernels.sh). Twelve reviewers tried to refute the cuts on Metal semantics, picture equivalence and performance; what they found is folded in above. The material shading, lighting, glass, probe, post-effects and temporal tests pass on the rebuilt library.
… device result of the Swift-side cuts
…ault A sun switched on in a headset scene cost about 1.5 ms a frame, half of it the second cascade: every caster drawn once more into a 2048x2048 depth map, and the band where the two cascades meet sampled twice, for a distance a room or a tabletop never shows. The cascade count is a setting now, setRendering(.shadowCascades(1 or 2)): unset, the headset draws one and the desktop every cascade the maps hold. The count flows from the setting through the cascade splits (one cascade reaches the whole shadow distance), the frustum fitting, both shadow passes and the shader's uniforms, which already read the count; the depth array keeps its slices, and the slots past the cascades in use repeat the last one. With maxShadowCastingDistance set to the scene's size, one cascade is sharp on a small scene. A render test checks the setting, the splits and the headset default; the proposal and the XR architecture doc record the device measurements behind it.
… serializeScene (untoldengine#1366) Avoids spurious 'Does not have a Physics component' (1028) errors logged for every entity on Play. Fixes untoldengine#1363
…is shown to cast When a sun stops casting (castsShadow off, or no sun active) the cascades are no longer drawn, but the shader kept sampling the last maps through the last matrices, so the stale shadows stayed on the scene. The uniforms of a sun that does not cast now carry matrices that land every point outside the maps, which the shader already reads as lit. Found by a render test that renders a box on a floor with the sun casting and not, and counts the floor pixels the shadow darkens: with two cascades and with one, which must shadow about the same pixels; the unit test of the uniforms' packing states that its system is an active one, and a sibling checks the non-casting matrices.
…nOS and in the XR loop, deferred elsewhere From the review of untoldengine#1359: the two per-fragment settings defaulted to on, which made forward the default on every platform, while only the headset had been measured. `RenderInfo.defaultRenderPath` is forward on visionOS and deferred on every other platform, so macOS and iOS run the deferred renderer as develop does; the two settings take their defaults from it. The XR loop adopts forward as it initialises unless the game chose a path (`renderInfo.renderPathChosen`, set by `.renderPath` and by either toggle; `adoptXRRenderPathUnlessChosen`), so a Mac rendering a remote immersive space gets the headset's path too. RenderPathTests checks the platform's default and the adoption; the profile and architecture docs say where each path is the default.
…heir path and restore it `.hybrid` was named only in RenderPathTests, which checks that the two settings compose it; no test drew it. HybridRenderPathTests does: the same picture as deferred at one sample and under 4× MSAA, the same picture as forward, the light quad's draw back (forward's draws + 1) with the tile light stage standing aside, and the point light's contribution — the frame with it less the frame without — equal to the deferred path's, so a direct light the geometry added as well would show as twice it. Measured on an M4 Max: hybrid against deferred mean 0.00004 and max 0.002 per channel, against forward mean 0.00003 and max 0.001; the point light's contribution differs by 0.0005 at most. The two existing per-fragment classes pin their paths with `.renderPath` (deferred and hybrid, hybrid and forward) and, with TileLightStageTests and XRProbeOrientationTests, restore the path they found instead of "enabled", so none leaves a path to the next test in the same worker now that the platform's default is deferred.
…nd the docs say which path writes motion The deferred and hybrid paths write no motion vectors, so under TAA an object moving on its own relies on the neighbourhood clip there. TAAMovingObjectTests runs its box on the three paths at two strides: 0.16 a frame, most of the box's width, where the clip alone rejects the stale history, and 0.02 a frame, a few pixels, the ghosting case. Measured on an M4 Max (a plain box over a plain background): the resolve's excess over a resolve held still is 0.1–0.3 % of the box's motion with the motion written and 0.2–0.4 % without, under `.taa` and `.msaaTaa`; the test bounds all three paths at a tenth of the motion and prints the figures. The architecture doc's TAA paragraph said there were no motion vectors yet; it says now that the forward pass writes them and the other paths rely on the clip, as do the `.taa` and TemporalAntiAliasing.swift comments.
…y their components `VertexOutModel.currentClip` and `prevClip` hold the clip position's x, y and w, not xyz: they are `currentClipXYW` and `prevClipXYW`, and `modelVelocity` names the w it divides by. The seven kernel metallibs are rebuilt.
…set, specular anti-aliasing, root-level project layout (untoldengine#1366)
…ne#1359: foveation, temporal anti-aliasing, a forward path and RealityKit's look in a mixed space
…sed for its presentation time, the headset's frame cost on the Swift side, the Forward pass and resolve doing less per pixel
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the fork's
developup to upstreamdevelopatebfa68beand, on top of it, the headset rendering work that is still in review, so every session tests the same engine.What upstream brings (10 merges since the last sync)
Material.heightEnableddefaults to false, auto-enabled on height-map loadsserializeSceneThe upstream merge is clean; the tree after it is upstream plus the fork's 13 proposal documents.
What is merged on top, for testing before it lands upstream
feature/xr_headset_renderingat2832d5fb, the head of Headset rendering: foveation, temporal anti-aliasing, a forward path, and RealityKit's look in a mixed space untoldengine/UntoldEngine#1359: foveation through the compositor's rate maps, temporal anti-aliasing (.taa,.msaaTaa), the forward render path (the headset's default), RealityKit's look in a mixed space (tone map.none, Display P3 primaries, the room probe convolved), back faces lit towards the viewer, the depth-pyramid gate and the pass skips.feature/xr_shader_costateac4e3e6, the stacked follow-ups [Feature] The frame is posed for its predicted presentation time: no judder for a moving object on the headset #73, [Feature] The headset's frame cost, on the Swift side: the pyramid's gate, the sun, the look pass, the fences, the textures #74 and [Feature] The headset's Forward pass and temporal resolve do less per pixel, and the sun casts in one cascade #75: the frame posed for its predicted presentation time, the headset frame's cost on the Swift side, the Forward pass and the temporal resolve doing less per pixel, one sun cascade on the headset.One conflict between the two headset lineages, in
RenderResources.swift: both sides reformatted the same property and its comment; the merge keeps the formatted property and the newer comment. The seven Metal libraries were rebuilt from the merged sources (make compile-shaders), and the source hash with them.Not merged: the large-scene stage 2 branches (instanced draws, entity walks, pack batches). They are local, not yet opened as PRs, and they rework the model fragment shader and the model pipelines in a way that conflicts with the forward path of untoldengine#1359 (13 hunks in
modelShader.metal,RenderPipeLines.swift,RenderPasses.swift,EngineSettingsAPI.swiftand friends). They need a rebase onto the headset work by their own session before they can join.Verified locally
UntoldEngineTests): 1,755 pass, 2 skipped, 0 failures.UntoldComponentKitTests): 59 pass.xcodebuild -scheme UntoldEngine): BUILD SUCCEEDED.Since untoldengine#1357 the package carries a build-tool plugin (the stale-kernels check):
xcodebuildbuilds of apps that take the engine as a package need-skipPackagePluginValidation, and Xcode asks once to trust the plugin.When untoldengine#1359 and the stacked PRs merge upstream, the next sync brings their final form; the fork's copy is a snapshot for testing.