Repository navigation
[Performance] Phase A counters and Phase B benchmark: compositor deadlines, per-system CPU time, GPU pass timer, hitches, recorder, runtime stats switch, PerfBench - #53
Draft
miogds wants to merge 17 commits into
Conversation
miogds
changed the title
added 17 commits
October 16, 2026 13:43
…per frame On visionOS the frame loop waited for the compositor's optimal input time but recorded nothing about the outcome, so the stats could not say whether a frame was submitted late or finished on the GPU after its rendering deadline. - Add EngineCompositorStats to the stats snapshot: update phase ms, input slack against optimalInputTime, submission ms, GPU margin against the drawable's renderingDeadline and presentationTime, missed-deadline flag and cumulative counts (with a miss rate), missing-anchor count, view count and per-view drawable size. - Record them in UntoldEngineXR.renderNewFrame/executeXRSystemPass; the GPU margins come from the command buffer completion handler. - Add timing.semaphoreWaitMs on the macOS and visionOS paths. - Add a Compositor signpost category (CompositorUpdate, CompositorWaitForInput, CompositorSubmission) and a MissedDeadline point event so Instruments shows the phases next to the Metal System Trace GPU track. - Print a Compositor line in the verbose stats output when views > 0, and document the fields in UsingProfiler.md.
The Update scope was one number; when it grew there was no way to tell scene-graph propagation from animation, scripting, physics or the game callback without a Time Profiler session. - Wrap traverseSceneGraph, extension updates, LOD selection, animation, USC scripting, each fixed-step physics update, custom systems and the game update callback in a profileSystem helper that emits a signpost in a new Systems category and, when engine stats are compiled in, adds the CPU time to new per-system fields on EngineTimingStats (physics and custom systems sum over the fixed steps; physicsStepCount records the steps taken). - Print a Systems line in the verbose stats output and document it.
GPU time was one number per command buffer, which on visionOS holds culling, Gaussian sorting, both eyes and the HZB build. There was no way to attribute it to a pass without a Metal System Trace session. The engine's GPUPassTiming (Profiling/GPUPassTiming.swift) times the passes: MTLCounterSampleBuffer timestamps at stage boundaries, a ring of sample buffers per in-flight frame, resolution when the command buffer completes, passes merged by label (per-eye passes sum), switched on with setRendering(.gpuPassTiming(.enabled)). - GPUPassTimingSnapshot: the timer's last resolved frame as the engine stats carry it (the passes, their sum, ms(for:)), published into EngineStatsSnapshot.gpuPasses once per frame. - The verbose stats output prints the heaviest passes; the timer and the snapshot are documented in docs/API/UsingProfiler.md. - Passes the timer did not cover are timed too: the three Gaussian compute passes, the Gaussian occlusion depth copy, the extension surfaces and the scene render target. Picking encodes on a command buffer of its own and stays untimed. - GPUPassStatsTests: the stats carry the timed passes.
…l state and GPU memory Mean and p99 frame times hide single long frames, and a benchmark run had no way to leave a file behind that a script could compare. - EngineHitchStats: cumulative frame-time histogram (buckets straddling the 90 Hz and 60 Hz periods), frames over a configurable budget (EngineStatsMonitor.frameBudgetMs, set to the 90 Hz period by the visionOS runtime), and a rolling one-second window with its worst frame. - EngineStatsRecorder with startEngineStatsRecording(to:interval:) and stopEngineStatsRecording(): every published snapshot (or the last of each second) written as JSON Lines on a utility queue, plus a summary line with frame percentiles, frames over budget, mean GPU time, deadline misses, mean GPU time per pass and peak GPU allocation. Every stats struct is now Codable and Sendable. - Memory stats gain MTLDevice.currentAllocatedSize, os_proc_available_memory (iOS/visionOS) and ProcessInfo.thermalState; thermal transitions emit a ThermalStateChanged signpost event. - The AR path times its passes too: GPUPassTiming's frame begins and ends around its command buffer. - Hitches line, memory fields and the recorder documented; tests added.
…build configuration ENGINE_STATS_ENABLED was defined for debug builds only, so a release build (the one that ships and the one to measure on a device) had no stats at all unless the package was rebuilt with -DENGINE_STATS_ENABLED. - Define ENGINE_STATS_ENABLED for the engine and XR targets in every configuration. - EngineStatsMonitor gains isCollecting (an OSAllocatedUnfairLock<Bool> read on every frame-loop call without the monitor lock): on by default in debug builds, off in release, overridable with UNTOLD_STATS=1. All per-frame entry points return at once when it is off, publishEngineStats skips gathering the snapshot, and RenderStatsCollector samples the flag at frame start so the per-draw path reads one Bool. - setEngineStatsCollection(enabled:) / isEngineStatsCollectionEnabled(); setEngine(.metrics(.enabled/.disabled)) toggles collection together with the profiler. - Tests for the switch; UsingProfiler.md build notes updated.
A repeatable way to measure a build: the same scenes, drawable size and camera motion, the engine stats and the GPU pass timer on, in release, on macOS, iOS or Apple Vision Pro, with the results pulled into one folder and compared against a stored baseline. Examples/PerfBench (XcodeGen project, engine referenced by path): - Seven scenes: 1k individual primitives, 10k batched primitives, 64 point lights, the post-processing chain, 16 skinned characters, the stadium and grass assets, a Gaussian splat. Assets come from the render-test resources and are copied into the bundle. - BenchRunner builds, warms up, records (JSON Lines per frame through the engine recorder), tears down and moves on, then writes summary.json with the device, the render configuration (platform, viewport, view count, layout, foveation, immersion, refresh rate) and every scene's summary and last snapshot; configuration comes from UNTOLD_BENCH_* environment variables so devicectl can drive it, and the app exits when done. - macOS: fixed 1920x1080 drawable, orbiting camera. iOS: SwiftUI host. visionOS: CompositorLayer with the engine's dedicated layout, scenes placed in front of the user, full or mixed immersion. - The hitch budget is one and a half refresh intervals (a skipped refresh). scripts/perf/run_bench.sh builds the target (arm64 only), runs it locally or installs and launches it on a device with the environment, optionally records an Instruments trace, copies the run folder back and calls scripts/perf/compare_baseline.py, which compares frame percentiles, GPU time, over-budget and missed-deadline rates and per-pass GPU means against perf/baselines/<model>/<platform>/<scene>.json and refuses to compare different render configurations.
15 s per scene, 1920x1080, display-paced at 120 Hz, release build of 0226477, so later runs on this Mac have something to compare with.
…s; aggregate repeated benchmark runs Two 15 s runs of the same build on this Mac differed by 20 to 50 % in per-pass GPU and per-system CPU times: the machine idles most of each display-paced frame and its clocks drift, which only ever inflates a time. - EngineStatsRecordingSummary gains timingMeanMs (every EngineTimingStats field and the compositor phases) and gpuPassMinMs. - compare_baseline.py accepts several summaries, aggregates time metrics by their minimum and rates by their mean, compares the per-system CPU means at the frame-time tolerance and the per-pass GPU minimums instead of the means; --update stores the aggregate. - run_bench.sh --repeat N runs the scene set N times and passes every summary to the comparer; the README explains the noise and the remedy.
10 s per scene, three repeats, aggregated by the minimum of each time metric; includes the per-system CPU means and per-pass GPU minimums.
…r information Three repeats of the same build still varied 40 to 70 % in mean GPU time on an idle GPU. The recording summary gains minGPUExecutionMs; the comparer judges that and prints the mean as information only. Existing baselines lack the field and skip the check until re-recorded.
On visionOS the engine calls the game update from the XR render thread, so MainActor.assumeIsolated in the PerfBench callback trapped on the first frame. BenchRunner is no longer main-actor bound: its state machine runs on whatever thread the engine updates from (main on macOS and iOS, the XR render thread on visionOS, inside the frame's update phase where the engine expects scene mutations), start() only requests the run, and the observable UI properties are published on the main thread. The scenes and the config lose their main-actor annotations for the same reason.
…visionOS 27) One 15 s run per scene from the headset: dedicated layout, no foveation, 2048x1984 per eye, full immersion, release build of ac740ee. Every scene misses the compositor deadline on most frames; this is the reference the visionOS work is measured against.
…nchmark scene The scene's teardown set anti-aliasing to .none, so every scene after it ran without the engine's default FXAA: on the headset the stadium lines shimmered. Capture the mode at build time and put it back.
A cube on a plane is the frame's fixed cost and the scene to judge image quality on a headset without load. UNTOLD_BENCH_AA selects fxaa, smaa, msaa or none for every scene (the post-FX scene keeps SMAA); the mode is recorded in the run and is part of the baseline configuration key.
…y baselines by refresh rate On a Mac with several displays the bench opened on NSScreen.main, the screen with keyboard focus, so the same command was paced at 60, 100 or 120 Hz depending on where the terminal was. It now runs on the fastest connected display, or on one with the rate given by --refresh / UNTOLD_BENCH_REFRESH, and keeps its window inside that display. The window shows the fixed 1920x1080 drawable one drawable pixel per device pixel. Stretched over a Retina display, the window server rescaled it every frame on the GPU being measured: mean GPU time of the light scenes was 3.3 to 5.6 ms and varied between runs, against 1.2 to 1.8 ms and stable at 1:1. Comparer: - The refresh rate is part of a baseline's configuration (times taken at different rates are not comparable); a mismatch now says which setting differs. - Rates are aggregated by the best run, like times. Other activity on the machine makes the window server skip refreshes in bursts, which only inflates the over-budget rate; a real regression raises it in every run. - GPU times are judged only when the baseline shows the GPU busy for at least half the frame, as on a headset, and printed as information otherwise: with the GPU mostly idle they follow its clock state (one frame's minimum GPU time measured 0.8 and 2.2 ms in consecutive runs of the same build). - A GPU or CPU time also has to move by more than an absolute slack to count.
Three-run aggregate of all eight scenes (the single-cube scene had none) at 88737f3, on the built-in display at 120 Hz with the 1:1 window. The earlier baseline predates the merge of develop and the window change. Per-system CPU means repeat within a few percent across the three runs (encode 6.32 / 6.39 / 6.34 ms in primitives-1k, 17.5 / 18.0 / 18.1 ms in primitives-10k). One of the three runs was free of skipped refreshes; the other two had bursts from other activity on the machine, which the best-run aggregation leaves out.
miogds
force-pushed
the
feature/perf_counters
branch
from
October 16, 2026 14:34
4bb0bb5 to
51873ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phases A and B of the performance work: the counters the rest of the plan depends on, and a benchmark that can be re-run (see
docs/proposals/PerformanceMeasurementResearch.md, section 4). Fork-only for now; nothing goes upstream until the whole effort is done.Phase A: counters (
Sources/UntoldEngine/Profiling)EngineCompositorStats: update-phase ms, input slack againstoptimalInputTime, submission ms, the GPU's margin againstrenderingDeadlineandpresentationTime, missed-deadline counts and rate, missing-anchor count, view count and per-view drawable size.Compositorsignpost category and aMissedDeadlineevent.timing.semaphoreWaitMson macOS and visionOS.Systemscategory) and a field onEngineTimingStats.GPUPassTimersamplesMTLCounterSampleBuffertimestamps at stage boundaries for every labelled render, compute and blit pass of the frame command buffer (two samples per pass, no extra GPU work). Passes attach throughmakeRenderCommandEncoder(descriptor:passLabel:),makeComputeCommandEncoder(passLabel:)andmakeBlitCommandEncoder(passLabel:), which fall back to the plain Metal calls; 50 pass sites converted, labels unchanged. Enable withGPUPassTimer.shared.isEnabled = trueorUNTOLD_GPU_PASS_TIMING=1. A test checks the pass sum stays in the same order of magnitude asgpuEndTime - gpuStartTime(the first calibration was off by the mach timebase, 41x; fixed in a follow-up commit).EngineHitchStats(histogram with buckets straddling the 90 Hz and 60 Hz periods, frames over a configurable budget, rolling one-second window),startEngineStatsRecording(to:interval:)/stopEngineStatsRecording()writing JSON Lines plus a summary (percentiles, over-budget frames, mean GPU time, deadline misses, mean per-pass GPU time, peak GPU allocation), every stats structCodableandSendable,MTLDevice.currentAllocatedSize,os_proc_available_memoryandProcessInfo.thermalStatein the memory stats with aThermalStateChangedevent.ENGINE_STATS_ENABLEDis compiled into every configuration; collection is a runtime flag (OSAllocatedUnfairLock<Bool>read without the monitor lock), on in debug, off in release,UNTOLD_STATS=1orsetEngineStatsCollection(enabled:)/setEngine(.metrics(.enabled))to turn it on. When off every per-frame entry point returns at once and the per-draw counter reads one Bool.docs/API/UsingProfiler.mddocuments every new line and field.Phase B: benchmark (
Examples/PerfBench, decision D2)XcodeGen project with macOS, iOS and visionOS targets, engine referenced by path. Seven scenes (1k individual primitives, 10k batched, 64 point lights, the post chain, 16 skinned characters, stadium and grass, a Gaussian splat), assets from the render-test resources.
BenchRunnerbuilds, warms up, records every frame through the engine recorder, tears down, and writessummary.jsonwith device, render configuration (platform, viewport, view count, layout, foveation, immersion, refresh rate) and every scene's summary and last snapshot. Configured throughUNTOLD_BENCH_*environment variables sodevicectlcan drive it; exits when done.scripts/perf/run_bench.sh <macos|ios|visionos>builds (arm64 only), runs locally or installs and launches on a device with the environment, optionally records an Instruments trace, pulls the run folder intoperf/results/and callsscripts/perf/compare_baseline.py, which compares frame percentiles, GPU time, over-budget and missed-deadline rates and per-pass GPU means againstperf/baselines/<model>/<platform>/<scene>.jsonand refuses to compare different render configurations.Examples/PerfBench/README.mdhas the details and how to read a regression.Measurement methodology (learned from the first runs)
Two runs of the same build on this Mac differ by 20 to 50 % in per-pass GPU and per-system CPU times when a scene fits in a frame: the machine idles most of each display-paced frame and its clocks drift, which only ever inflates a time. So the recording summary now carries per-system CPU means (
timingMeanMs), per-pass GPU minimums (gpuPassMinMs) and the fastest GPU frame (minGPUExecutionMs);run_bench.sh --repeat Nruns the scene set N times and the comparer aggregates by the minimum of each time metric, judges the CPU means, the GPU minimums and the frame percentiles, and prints the GPU mean as information only. The committed Mac16,5 baseline is a three-run aggregate. The CPU-bound scene (primitives-10k) is the stable one and already profiles the engine at scale: about 17 ms encoding, 3 ms culling and 1.5 ms scene-graph walk per frame against 4 ms of GPU time.Not in here
Verified locally (Xcode 27.0)
swift build(debug and release) andmake strict-concurrency-check.xcodebuild build -scheme UntoldEngineXRfor the visionOS 27 simulator.GPUPassTimerTests(6, real frames on the macOS renderer, including the pass-sum consistency check),EngineStatsMonitorTests(30),RenderStatsCollectorTests,FrameMetricsCollectorTests.UntoldEngineRenderTestsover the converted passes: the only failures are the 8 PSNR comparisons, which fail on this machine for lack ofcv2regardless of the change.UntoldEngineTests: the only failure is the external-package test that needs the checkout folder to be namedUntoldEngine.scripts/perf/run_bench.sh macosran all seven scenes end to end on this Mac and produced a comparable summary.