Repository navigation
Merge the large-scene stage 2 chain for testing: instanced draws, entity walks and pack single batches, rebased onto the headset rendering - #77
Merged
Conversation
added 20 commits
October 15, 2026 23:02
A scene that places one model many times asks for the same draw over and over with a different transform. These are the vertex stages for drawing such entities together: one draw with an instance count, and what belongs to each instance read from a buffer. vertexModelInstancedShader takes the transform of a surface (model-view, model and normal matrices, and the model matrix of the previous frame for the velocity) from a ModelInstanceUniform record, indexed by the instance id; vertexShadowInstancedShader takes a model matrix per instance. They read the four vertex streams every mesh has (the position alone for a shadow): a mesh drawn this way has no skeleton. The instanced model stage hands the fragment stage the VertexOutModel of vertexModelShader, computed by the same arithmetic on the same values, and the fragment function of an instanced pipeline is fragmentModelShader itself, built with the same function constants: a surface shades the same whichever way it is drawn, on the deferred, hybrid and forward paths alike, to the last bits of what the two vertex stages hand it (two compilations of the same arithmetic differ in those). The opacity of a ghost channel is the draw's, in the material parameters, as for a single draw. vertexModelShader, fragmentModelShader, vertexShadowShader, Uniforms and VertexOutModel are as they were: a render extension that draws with them is not affected.
The seven prebuilt Metal libraries, rebuilt with buildkernels.sh for the instanced model and shadow shaders, and the source hash the stale check compares against. The deployment targets are the package's (macOS 14, iOS 17, tvOS 17, visionOS 2).
…one instanced draw The opaque pass and the shadow cascades encoded every entity on its own: its vertex streams, its material and a draw call per submesh, however many entities drew the same thing. A scene that places a model a thousand times paid for a thousand draws. Both now gather the entities that have nothing of their own in a draw but their transform, and encode each group of them that draws the same meshes with the same materials as one instanced draw per submesh. A group is found without anything being declared. A render component keeps a small number, its instance key, that two components share exactly when their mesh lists send the same things to the GPU: the same Metal meshes and local transforms and, per submesh, the same index data, textures, samplers and material values. RenderInstanceKeyTable gives the numbers by comparing the descriptions in full, not their hashes, so two different materials never share a draw. Any change to RenderComponent.mesh (a material edited, a texture streamed in, a LOD switch) clears the key, and it is looked up again when the entity is next drawn: an entity whose material is edited leaves its group, and joins it again if the material is set back. Entities on a ghost channel join the group of their key with their opacity: the opacity is the draw's. An entity with something of its own in a draw is drawn alone by the code that drew it before: a LOD, tile or mesh fade in progress, a skeleton, a deformation, and every entity while a debug colouring is on. Transparent submeshes stay with the transparency pass, which sorts them. What a group's draws bind is prepared once per component (InstancedSubmeshDraw) from the function the single-entity path fills its parameter block with, and kept until the meshes change. The records of the instances (model-view, model and normal matrices, and the model matrix of the previous frame; a model matrix for a shadow) go into a buffer borrowed from a pool and handed back when the frame's command buffer completes. The buffer is bound once per pass and each draw names its first record as its base instance. A device that cannot do that (before the A9) gets no instanced pipelines and draws as before. The instanced model pipelines are the model pipelines with the instanced vertex stage: the same fragment function and function constants, so a group shades as a single draw does on every render path, the lighting inputs the forward path binds once per pass serving both; the same pair with and without the discards, so a group whose material never discards goes through the pipeline that lets the GPU resolve visibility first; and the velocity each pixel writes on the forward path comes from where the instance was the previous frame, which its record carries. A frame is the same picture drawn either way, to the last bits of a few pixels: a vertex stage compiled apart from another differs from it in the last bits of what it computes (the G-buffer attachments at their own precision are identical), and the derivatives the fragment stage takes turn that into a visible difference at a quad or two along the silhouettes. The tests compare the two ways in fifteen scenes, on the deferred, hybrid and forward paths, the velocity of a moving instance included. The order of the draws changes (single entities, batches, then the groups), so where two surfaces lie exactly on each other the other one may show. setRendering(.instancedDraws(.disabled)) turns it off. 10,000 cubes of one mesh, 1,888 of them casting into a shadow cascade (release build, M4 Max, before the forward path): 11,896 draws become 10, and the CPU time of a frame goes from 10.8 ms to 4.9 ms.
The shadow pass went over the entities of the scene once to fit the cascades and once more per cascade to choose its casters, working out the same world boxes each time. With tens of thousands of entities that was a fifth of the CPU time of a frame, shadows drawn or not. The pass now walks the scene once a frame. The walk gives the bounds the cascades are fitted to, which ShadowSystem.updateCascades takes from its caller when it has them, and the casters with their boxes, which each cascade then tests by distance and frustum without touching the scene again. A cascade draws the casters it drew before, in the same order. The fit makes the light-space box of a caster (eight transforms) only when the sphere around it (one transform) could overlap a cascade seen from the light. The sphere holds the box, so the cascades come out the same; DirectionalShadowWalkTests compares them with those the shadow system fits on its own. Whether an entity casts is read from its render component as the walk passes it. A caster made or switched since the last frame no longer waits for invalidateShadowEntityCache; the spot and point passes still use the candidate cache.
The transparency pass asked every visible entity whether any of its submeshes is blended by going through copies of its meshes and their materials, every frame, to find in nearly every case that none is. A render component now keeps the answer until its meshes change (the didSet that clears its instance key clears this too), and the pass asks it before anything else about the entity.
LODSystem.update asked the scene for the LOD component and the two transform components of every entity with levels, each a lock, a copy of the scene and a component-id lookup. It now reads them through one RenderSceneSnapshot, as the render passes do. The fade transitions are advanced the same way. entityDistanceToCamera and entityDistanceAndRadius gain a form that takes the components; the forms that take an entity id call it.
Every tick, GeometryStreamingSystem.update asks the octree for the entities within 500 units and goes through them six times, asking the scene each time whether the entity exists and has a StreamingComponent or a TileComponent. In a scene loaded whole none has either, and all of it was spent finding that out. The entities the query returns are now narrowed once, with one read of the scene, to those that exist and have one of the two components. The passes see the entities they acted on before. The checks that ask whether the query returned an entity still answer for the whole query; the set they use is only made when a pass has an entity to ask about.
The culling gather made the world box of every entity twice (once for the small-object test, once for the cull) and built an array per entity to hold its one to three bounding boxes before appending them to the list for the GPU. The box is now made once, and appendSegmentedEntityAABBs adds an entity's bounding boxes to the list in place. makeSegmentedEntityAABBs is kept and calls it.
How the shadow pass walks the scene once a frame, and that the LOD selection, the streaming update and the transparency pass follow the rule the render passes follow: read the scene once, and keep on the render component what the meshes alone decide.
The batches of a cell are built per material: meshes that share a material are merged, and a mesh that is alone with its material is left to its entity. But the render passes draw an entity either through its batches or by itself. An entity with two materials, one shared with a neighbour and one of its own, was skipped as batched while only its shared mesh was in a batch: nothing drew the other mesh. A lone mesh now gets a batch of its own when its entity is in a batch for another of its meshes. A mesh whose entity shares nothing is left to the entity, as before.
…t, sheen or tints A batch draws all of its meshes with the material of one of them, so the key that decides which meshes are merged has to tell apart everything the opaque pass sends per material. It left out the edge tint, the specular tint, subsurface, anisotropy, sheen, its tint, clear coat and its gloss: two materials alike in the rest were merged, and one of them was drawn with the values of the other.
The batches of a cell are built on a background queue and applied by a later tick. The tick returns early when it has no pending entity, no dirty cell and nothing to retire, and it did so before looking at the builds that had finished. The last cell of a scene that then stood still was never applied: its entities stayed on a draw each, with their batch waiting in the queue, until something else in the scene changed. The tick now goes on while a finished build waits.
With batching enabled, the shadow passes skipped every entity marked static: its batch was expected to cast for it. An entity that no batch took had no shadow at all, from the sun, from spot lights or from point lights. That is an entity alone with its material in its cell, every entity of a cell over the runtime limits, and an entity marked static that the batching system was never told of. The passes now ask the batching system (castsShadowByItself). An entity in a batch is drawn by the batch. An entity of a streamed cell that waits for its batches, or of a tile that is fading in or out, is still skipped: that is what keeps a tile that has just come in from adding a shadow draw per entity. Any other marked entity casts for itself, and the cascades are fitted to it. The spot and point passes keep a list of candidates that leaves out the marked entities a batch draws. The batching system invalidates the list when batches are built or removed.
Marked entities reach the batching system in two ways: through generateBatches(), which rebuilds every batch of the scene inside the call, or as streamed tiles, whose cells are built once they have been seen and only under the per-cell size limits. notifyStaticEntitiesPlaced(_:) is for marked entities that are loaded and in the scene to stay. Their cells are built by the tick, on the background queue and within the per-tick budgets, but without waiting to be seen and whatever their size, as generateBatches() builds them. Until its batch is built an entity is drawn by itself, shadow included.
…eaves its batch A batch holds the vertices of its entities where they stood when it was built, and nothing told the batching system when that stopped being true. An entity moved with the transform functions was still drawn at its old place by its batch. One that lost its StaticBatchComponent, or was destroyed, stayed in the batch until something else rebuilt its cell. The editor works around the first case by unmarking the entity and rebuilding every batch of the scene. The transform functions now note the entity that moves and every descendant that moves with it. Before the frame is drawn those entities are out of their batches, and their cells fall back to a draw per entity. A moved entity keeps its mark: after 30 frames at rest it is batched again where it stands. Unmarking and destroying take the entity out at once, and its cell is rebuilt without it. The transform functions are called from any thread, so they only append the entity to a list behind a lock, and only while there is an entity that a move could take out of a batch.
…t is batched Each pass asked the batching system about every visible entity, a dictionary lookup per entity and pass: 7 % of the CPU time of a frame with 17,000 visible entities and batches on. Only an entity with a StaticBatchComponent can be in a batch, now that one that loses the component leaves its batch at once, and the component is a bit of the snapshot the pass already holds.
A model that a pack places several times is drawn with instancing. The models it places a single time have nothing to share a draw with, and a building exported from a BIM tool is mostly that. The pack loader now marks those parts static: the parts of a model that the pack places once and that has no LOD chain, without the ones no batch takes (a transparent material, a skeleton, an animation). When the pack is in, it hands them to the batching system, which merges them by material in the background. With batching disabled the marks do nothing, and a later generateBatches() uses them. setBatching(.packSinglePlacements(false)) turns the marking off. On a site of 21,576 parts the loader marks 4,373, and 4,227 of them end up in 213 batches of 37 MB, built over the 22 frames after the load. Six views at 1080p, three alternating rounds against the build before: a frame goes from 19.6 to 18.2 ms from above and from 25.5-31.3 to 20.8-26.8 ms among the trees, its CPU time from 12.3-13.3 to 9.8-11.7 ms, the draws from 7,340 to 4,236. Three of the six views are the same pixel for pixel. In another a pair of coplanar parts is drawn in the other order (682 pixels of 2 million), and close to a tree the faint self-shadows of the leaves are gone (15,000 pixels, most by 1 or 2 of 255): the first cascade is fitted to the boxes of the batches and covers 30 m in depth where it covered 24.
The scenes of StaticBatchMembershipTests were lit by the renderer's default light, whose shadow of a post is 140 pixels of the frame in their view: about as many as the edge pixels that a batch and an entity's own draw may differ by. The class now makes its sun the active light, from a side where that shadow is 3,500 pixels, so that a shadow missing or drawn twice cannot pass for edge differences.
…hat casts no shadow A batch drew all of its entities and cast a shadow for all of them, whatever their render components said. An entity that was hidden went on being drawn by its batch, and one told not to cast a shadow went on casting through it. An editor that hides an entity by clearing isVisible had no way to hide a batched one. A hidden entity is no longer taken into a batch. castsShadow is now part of what groups entities: the ones that cast no shadow get batches of their own, which the shadow passes skip, so they keep the saving in draws. The render component tells the batching system when either flag changes. Before the frame is drawn, an entity that was hidden is out of its batch, one that is shown again is handed back, and one whose castsShadow changed moves to a batch of entities that cast as it does. They keep their mark throughout.
… chain: instanced draws, entity walks, pack single batches), rebased onto the headset rendering The three local branches of the large-scene plan's stage 2, rebased in order from upstream develop 1085580 onto this develop (10d1fbe: upstream ebfa68b, the head of untoldengine#1359 and the xr_shader_cost chain), so that every session tests them with the headset rendering: - feature/instanced_draws (452c8c4): entities that share their meshes and materials drawn as one instanced draw, on the deferred, hybrid and forward paths; - feature/entity_walks (7f3a74a): one shadow walk per frame, LOD through a snapshot, the cull gather in place; - feature/pack_single_batches (16a2c93): six fixes to batching, the parts a pack places once batched. Resolved against the forward path: the instanced model pipelines take the model pipeline's own fragment function and function constants (there is no instanced fragment function any more), come with and without the discards like the model pipelines, and write the velocity of each instance from the previous model matrix its record carries; the opacity of a ghost channel is the draw's, so entities that differ in it are in groups of their own; the transparency pass's gathering, moved before its encoder by the fork, takes the chain's checks there. The seven Metal libraries are rebuilt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the large-scene stage 2 chain, rebased onto this develop (
10d1fbec: upstream developebfa68be, the head of untoldengine#1359 and thexr_shader_costchain of #73–#75), so that every session tests it together with the headset rendering. The chain was local only until now, on upstream develop1085580c; the upstream pull requests come after this, one per branch.The chain, in order (each rebased onto the previous):
feature/instanced_drawsat452c8c40(3 commits): entities that share their meshes and materials are encoded as one instanced draw, in the opaque pass and the shadow cascades.feature/entity_walksat7f3a74af(6 commits): one walk of the scene per frame for the directional shadows, LOD selection through a snapshot, the streaming update narrowed, the cull gather without an array per entity, a flag for blended submeshes.feature/pack_single_batchesat16a2c931(10 commits): six fixes to batching (a batched entity's lone mesh, materials that differ in clear coat/sheen/tints, a batch built in the background, a static entity that no batch draws, one that moves or loses its mark, a hidden entity), batching in the background the static entities placed to stay, and the parts a pack places once marked static.What was resolved (13 hunks against the forward path, plus the seven metallibs):
modelShader.metal: the headset turnedfragmentModelShaderinto a function-constant-driven stage (kModelMayDiscard,kModelIndirectInGBuffer,kModelLitInGBuffer; the lit and velocity attachments; the world position and the clip positions from the vertex stage). The instanced path no longer has a fragment function of its own:vertexModelInstancedShaderhands the fragment stage the sameVertexOutModelasvertexModelShader, computed by the same arithmetic (the world position, the clip positions of this frame and the previous one for the velocity), and the instanced pipelines are built withfragmentModelShaderitself and the same function constants. So a group shades as a single draw does on the deferred, hybrid and forward paths, and the lighting inputs the forward pass binds once per pass serve both.InstancedDrawListgroups by key and opacity). The record (ModelInstanceUniform) carries the model-view, model and normal matrices and the model matrix of the previous frame; the opaque pass takes the latter fromObjectMotionas it does for a single draw, so a moving instance writes its own velocity for the temporal resolve.RenderPipeLines.swift/RenderPipelineType.swift/RenderInitializer.swift: onemakeModelPipeline(instanced:mayDiscard:name:)builds the four model pipelines (.model,.modelOpaque,.modelInstanced,.modelInstancedOpaque), all rebuilt together when the sample count or the render path changes; the opaque pass picks the instanced pipeline without discards for a group whose material never needs them, as it does per single draw, and counts it inopaqueDrawsPlain.ShaderTypes.h: the headset's fragment buffer indices kept, the chain'smodelPassFragmentInstanceIndexdropped (the fragment stage binds nothing per instance),modelPassInstanceIndexandshadowPassInstanceIndexkept.RenderPasses.swift: the opaque pass's per-fragment lighting bindings and the instanced draw list set-up both stay; the transparency pass's gathering was moved before its encoder by the fork ([Feature] The headset's Forward pass and temporal resolve do less per pixel, and the sun casts in one cascade #75: no pass for a frame with nothing transparent), so the chain's two changes to it (hasBlendedSubmeshesasked first, marked before batched) were applied there.EngineSettingsAPI.swiftanddocs/API/UsingEngineAPI.md: both sides' settings.make compile-shaders(.metalsource.sha256included).One finding on the way: with the fragment stage shared, the instanced and the single way of drawing still differ in the last bits of a few pixels (about 20 of a million in the test scenes). The G-buffer attachments (position, normal, emissive, depth) are identical at their own precision; the vertex stages, compiled apart, differ in the last bits of what they hand the fragment stage, and the derivatives the fragment stage takes (the texture footprint, the specular anti-aliasing) turn that into a visible difference at a quad or two along the silhouettes. The picture tests now hold the two ways to the last bits (a channel difference beyond 0.004 counts, at most 0.01 % of the covered pixels may show one). The claim "the same pixel for pixel" of the instancing commit is reworded accordingly.
Verified (Xcode 27.0 / Swift 6.4, on the chain's tip
16a2c931):swift build, and the seven metallibs rebuilt (make compile-shaders).swift test --disable-swift-testing --filter UntoldEngineTests: 1,755 tests, 2 skipped, 1 failure,ExternalRenderExtensionPackageTests(the checkout folder is not namedUntoldEngine; the known folder-name failure, unrelated to the change).CI=true UNTOLD_PSNR_THRESHOLD=33.5, filterInstancedDrawListTests|RenderInstanceKeyTests|InstancedDrawRenderTests|DirectionalShadowWalkTests|BlendedSubmeshFlagTests|StaticBatchMembershipTests|UntoldPackStaticPartsTests|HybridRenderPathTests|TemporalAntiAliasingTests|OpaqueHiddenSurfaceRemovalTests|Shadow|Batch|Culling|LOD|Streaming: 384 render tests, 16 skipped, 0 failures (among them InstancedDrawRenderTests 16, with the hybrid and forward path scenes and the velocity of a moving instance; HybridRenderPathTests 5; TemporalAntiAliasingTests 6; OpaqueHiddenSurfaceRemovalTests 5; DirectionalShadowWalkTests 4; BlendedSubmeshFlagTests 2; StaticBatchMembershipTests 17; UntoldPackStaticPartsTests 7; InstancedDrawListTests 10; RenderInstanceKeyTests 9). The same filter also runs two core tests ofLODSystemTestsafterEngineSettingsAPITestsin one process, wheretestSetLODUpdatesSharedConfigleaves the LOD defaults changed and they fail; alone, and in the core run, they pass (a test-isolation matter of the existing tests, not of the chain).feature/entity_walksbefore the batches were rebased on it: 291 tests, 10 skipped, 0 failures.make strict-concurrency-check(a clean strict build withMAX_UNIQUE_CONCURRENCY_WARNINGS=0): build pass, 0 warning sites.wrapIfStatementBodiesfindings of the local 0.62.1.The
keep/*_pre_1015branches hold the three tips as they were on upstream develop, until the upstream pull requests are out.