[Performance] Phase A counters and Phase B benchmark: compositor deadlines, per-system CPU time, GPU pass timer, hitches, recorder, runtime stats switch, PerfBench

.gitignore

@@ -97,3 +97,7 @@ scripts/tests/__pycache__/
 
 # Local SwiftPM scratch directories other than .build (e.g. .build-strict for the strict-concurrency build).
 .build*/
+
+# PerfBench run output (scripts/perf/run_bench.sh)
+perf/results/*
+!perf/results/.gitkeep

Examples/PerfBench/.gitignore

@@ -0,0 +1,4 @@
+*.xcodeproj
+.build/
+DerivedData/
+results/

Examples/PerfBench/README.md

@@ -0,0 +1,123 @@
+# PerfBench
+
+A benchmark app for the engine on macOS, iOS and visionOS. It builds a fixed set of scenes one after
+another, warms each up, records every frame through the engine's stats recorder, and writes one
+JSON Lines file per scene plus a `summary.json` for the run. `scripts/perf/run_bench.sh` builds it,
+runs it on a Mac or a paired device, collects the run folder and compares it with the stored
+baseline for that device model.
+
+The point is repeatability: the same scenes, the same drawable size, the same camera motion, the
+engine stats and the GPU pass timer on, in a release build, so a number from before a change can be
+compared with a number from after it.
+
+## Scenes
+
+| id | What it measures |
+|---|---|
+| `cube` | One lit, shadowed cube on a plane: the fixed per-frame cost of the pass chain, and the scene to judge image quality on a headset. |
+| `primitives-1k` | 1024 individual cubes and spheres, 4 point lights, cascaded shadows: per-entity CPU cost and encoder overhead. |
+| `primitives-10k` | 10 000 primitives merged by the static batcher: batched draw cost and light-loop cost at scale. |
+| `lights-64` | 1024 batched primitives with 64 point lights: the per-pixel light loop. |
+| `postfx` | The 1k grid with SSAO, bloom, depth of field and SMAA: the full-screen pass chain. |
+| `animation-16` | 16 skinned characters running: the animation system and GPU skinning. |
+| `stadium` | The stadium and grass assets: material binds and textures. |
+| `gaussian` | A small Gaussian splat plus primitives: the splat cull, depth-key and radix-sort passes. |
+
+Assets come from `Tests/UntoldEngineRenderTests/Resources`; the project copies the `Models` and
+`Animations` folders and the splat file into the app bundle.
+
+On macOS and iOS the camera orbits each scene (`BenchCameraOrbit`). On visionOS the head is the
+camera, so the scenes are built three metres in front of the user and the orbit is ignored; keep
+still during a run, or accept the head motion as part of the measurement.
+
+Frames are display-paced on every platform (the display's maximum refresh rate on macOS and iOS,
+the compositor on visionOS), so a scene that fits in a frame reports the refresh interval as its
+frame time. Read the cost of such a scene from `meanGPUExecutionMs`, the per-pass GPU means and the
+per-system CPU times in the snapshots; frame time and the over-budget count become the signal only
+when a scene no longer fits. A frame counts as over budget when it took more than one and a half
+refresh intervals, that is when a refresh was skipped.
+
+On a Mac with several displays the bench opens on the fastest one, whichever screen has the
+keyboard focus, or on one running at the rate given with `--refresh`. Times taken at different
+refresh rates are not comparable (the GPU stretches its work over the slack a slower display
+leaves), so the refresh rate is part of a baseline's configuration and the comparer skips a
+baseline recorded at another rate. The Mac16,5 baseline is recorded at 120 Hz, on the built-in
+display. The window shows the fixed 1920 by 1080 drawable one drawable pixel per device pixel, so
+the window server does not rescale it on the GPU that is being measured.
+
+## Running
+
+```bash
+brew install xcodegen                      # once
+
+scripts/perf/run_bench.sh macos                                  # this Mac
+scripts/perf/run_bench.sh visionos --device "Javier's Apple Vision Pro"
+scripts/perf/run_bench.sh ios --device <udid>
+
+scripts/perf/run_bench.sh macos --scenes primitives-1k,postfx --seconds 10
+scripts/perf/run_bench.sh macos --update-baseline                # store this run as the baseline
+scripts/perf/run_bench.sh visionos --device <udid> --xctrace "Metal System Trace"
+```
+
+Runs land in `perf/results/<timestamp>-<platform>/` with `<scene>.jsonl`, `summary.json` and the
+console log. The comparison prints one block per scene and exits non-zero on a regression (10 %
+on frame-time, GPU-time and per-system CPU metrics, 0.5 points on the over-budget and
+missed-deadline rates, 20 % on a per-pass GPU minimum). A GPU or CPU time also has to move by
+more than an absolute slack to count (0.5 ms for a frame's minimum GPU time, 0.25 ms for a pass,
+0.1 ms for a per-system CPU mean).
+
+Single runs are noisy on a machine that idles most of each frame: CPU and GPU clocks drift between
+runs, other activity makes the window server skip refreshes, and the same build can differ by 20
+to 50 % in a per-pass or per-system time. `--repeat 3` runs the scene set three times and the
+comparer aggregates by the minimum of each time metric and of each rate (interference only ever
+inflates them, and a real regression raises them in every run); record the baseline the same way.
+
+What to trust on a Mac: the per-system CPU means (`timingMeanMs`) repeat within a few percent, and
+so does everything in the CPU-bound scenes (`primitives-10k`). GPU times of the light scenes do
+not: with the GPU idle most of the frame they follow its clock state, and one frame's minimum GPU
+time measured 0.8 and 2.2 ms in consecutive runs of the same build. The comparer therefore judges
+GPU times only when the baseline shows the GPU busy for at least half the frame, and prints them
+as information otherwise; judge GPU cost on the headset, where the GPU is busy and run-to-run
+noise is under 10 %.
+
+The app can also be opened in Xcode (`xcodegen generate`, then `PerfBench.xcodeproj`) and started
+from its Start button; it reads its configuration from the environment:
+
+| Variable | Meaning | Default |
+|---|---|---|
+| `UNTOLD_BENCH_SCENES` | Comma-separated scene ids, or `all` | `all` |
+| `UNTOLD_BENCH_WARMUP` | Seconds rendered before recording, per scene | `3` |
+| `UNTOLD_BENCH_SECONDS` | Seconds recorded per scene | `15` |
+| `UNTOLD_BENCH_OUTPUT` | Output directory | `Documents/PerfBench` |
+| `UNTOLD_BENCH_RUN_ID` | Run folder name | timestamp |
+| `UNTOLD_BENCH_LABEL` | Free text stored in the summary | empty |
+| `UNTOLD_BENCH_KEEP_OPEN` | `1` keeps the app open after the last scene | exit |
+| `UNTOLD_BENCH_AUTOSTART` | `0` waits for the Start button | start on launch |
+| `UNTOLD_BENCH_IMMERSION` | `full` or `mixed` (visionOS) | `full` |
+| `UNTOLD_BENCH_PER_FRAME` | `0` writes one line per second instead of per frame | per frame |
+| `UNTOLD_BENCH_AA` | `fxaa`, `smaa`, `msaa` or `none` for every scene (the post-FX scene keeps SMAA) | engine default, FXAA |
+| `UNTOLD_BENCH_REFRESH` | macOS: run on a connected display with this maximum refresh rate (`run_bench.sh --refresh`) | the fastest connected display |
+
+On a device the app exits when the run is done so that `devicectl ... --console` returns; the run
+folder is then pulled from the app's Documents container. With the app opened from Xcode, set
+`UNTOLD_BENCH_KEEP_OPEN=1` in the scheme to read the report on screen.
+
+## What a run records
+
+Every line of `<scene>.jsonl` is a full `EngineStatsSnapshot` (frame timing, per-system CPU time,
+GPU pass times, culling, streaming, batching, memory, thermal state, compositor deadline margins on
+visionOS, hitch histogram), and the last line is the `EngineStatsRecordingSummary` for the scene.
+`summary.json` holds the device (model, OS, GPU), the render configuration (platform, viewport,
+view count, texture layout, foveation, immersion) and every scene's summary and last snapshot.
+See `docs/API/UsingProfiler.md` for the meaning of each field.
+
+## Reading a regression
+
+1. Frame p95/p99 up, GPU time flat: CPU-side. Open the run's `jsonl` and look at `timing`: which
+   system grew (`animationMs`, `scenegraphMs`, ...), or `encodeMs` for draw submission.
+2. GPU time up: `gpuPasses` says which pass. Reproduce the scene in the app from Xcode and take a
+   GPU frame capture of that pass.
+3. Missed deadlines up on visionOS with everything else flat: look at `compositor.inputSlackMs`
+   (late update phase) and `timing.semaphoreWaitMs` (GPU pacing the CPU), then a Metal System
+   Trace with `--xctrace`.
+4. `worstThermalState` above 1: the device throttled; rerun cool before believing the numbers.

Examples/PerfBench/Sources/PerfBenchCore/BenchConfig.swift

@@ -0,0 +1,83 @@
+//
+//  BenchConfig.swift
+//  PerfBench
+//
+// Copyright (C) Untold Engine Studios
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+import Foundation
+
+/// Run configuration, read from the environment so the driver script and `devicectl` can set it.
+///
+/// | Variable | Meaning | Default |
+/// |---|---|---|
+/// | `UNTOLD_BENCH_SCENES` | Comma-separated scene ids, or `all` | `all` |
+/// | `UNTOLD_BENCH_WARMUP` | Seconds rendered before recording starts, per scene | `3` |
+/// | `UNTOLD_BENCH_SECONDS` | Seconds recorded per scene | `15` |
+/// | `UNTOLD_BENCH_OUTPUT` | Output directory | `<Documents>/PerfBench` |
+/// | `UNTOLD_BENCH_RUN_ID` | Run folder name | timestamp |
+/// | `UNTOLD_BENCH_LABEL` | Free text stored in the summary (commit, branch, note) | empty |
+/// | `UNTOLD_BENCH_KEEP_OPEN` | `1` keeps the app running after the last scene | exit when done |
+/// | `UNTOLD_BENCH_AUTOSTART` | `0` waits for the Start button (visionOS) | start on launch |
+/// | `UNTOLD_BENCH_IMMERSION` | `full` or `mixed` (visionOS) | `full` |
+/// | `UNTOLD_BENCH_PER_FRAME` | `0` records one line per second instead of per frame | per frame |
+/// | `UNTOLD_BENCH_AA` | `fxaa`, `smaa`, `msaa` or `none`: anti-aliasing for every scene (the post-FX scene keeps SMAA) | engine default (FXAA) |
+/// | `UNTOLD_BENCH_REFRESH` | macOS: run on a connected display with this maximum refresh rate, for example `120` | the fastest connected display |
+public struct BenchConfig: Sendable {
+    public var sceneIDs: [String]
+    public var warmupSeconds: Double
+    public var measureSeconds: Double
+    public var outputDirectory: URL
+    public var runID: String
+    public var label: String
+    public var exitWhenDone: Bool
+    public var autoStart: Bool
+    public var immersion: String
+    public var perFrame: Bool
+    /// Anti-aliasing mode name from the environment, or nil to leave the engine's default.
+    public var antiAliasing: String?
+    /// Refresh rate of the display to run on (macOS), or nil for the fastest connected display.
+    public var refreshHz: Int?
+
+    public static func fromEnvironment(_ environment: [String: String] = ProcessInfo.processInfo.environment) -> BenchConfig {
+        func double(_ key: String, _ fallback: Double) -> Double {
+            guard let raw = environment[key], let value = Double(raw), value > 0 else { return fallback }
+            return value
+        }
+        let scenesRaw = environment["UNTOLD_BENCH_SCENES"] ?? "all"
+        let sceneIDs = scenesRaw == "all"
+            ? BenchScenes.all.map(\.id)
+            : scenesRaw.split(separator: ",").map { $0.trimmingCharacters(in: .whitespaces) }.filter { !$0.isEmpty }
+
+        let documents = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first
+            ?? FileManager.default.temporaryDirectory
+        let outputDirectory = environment["UNTOLD_BENCH_OUTPUT"].map { URL(fileURLWithPath: $0, isDirectory: true) }
+            ?? documents.appendingPathComponent("PerfBench", isDirectory: true)
+
+        let formatter = DateFormatter()
+        formatter.dateFormat = "yyyyMMdd-HHmmss"
+        formatter.locale = Locale(identifier: "en_US_POSIX")
+
+        return BenchConfig(
+            sceneIDs: sceneIDs,
+            warmupSeconds: double("UNTOLD_BENCH_WARMUP", 3.0),
+            measureSeconds: double("UNTOLD_BENCH_SECONDS", 15.0),
+            outputDirectory: outputDirectory,
+            runID: environment["UNTOLD_BENCH_RUN_ID"] ?? formatter.string(from: Date()),
+            label: environment["UNTOLD_BENCH_LABEL"] ?? "",
+            exitWhenDone: environment["UNTOLD_BENCH_KEEP_OPEN"] != "1",
+            autoStart: environment["UNTOLD_BENCH_AUTOSTART"] != "0",
+            immersion: environment["UNTOLD_BENCH_IMMERSION"] ?? "full",
+            perFrame: environment["UNTOLD_BENCH_PER_FRAME"] != "0",
+            antiAliasing: environment["UNTOLD_BENCH_AA"].map { $0.lowercased() },
+            refreshHz: environment["UNTOLD_BENCH_REFRESH"].flatMap { Int($0) }.flatMap { $0 > 0 ? $0 : nil }
+        )
+    }
+
+    public var runDirectory: URL {
+        outputDirectory.appendingPathComponent(runID, isDirectory: true)
+    }
+}

Examples/PerfBench/Sources/PerfBenchCore/BenchResults.swift

@@ -0,0 +1,136 @@
+//
+//  BenchResults.swift
+//  PerfBench
+//
+// Copyright (C) Untold Engine Studios
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+import Foundation
+import Metal
+import UntoldEngine
+
+/// What the run was measured on. Baselines are keyed by `model`.
+public struct BenchDeviceInfo: Codable, Sendable {
+    public var model: String
+    public var osVersion: String
+    public var gpuName: String
+    public var processorCount: Int
+    public var physicalMemoryBytes: UInt64
+    public var thermalStateAtStart: Int
+    public var isLowPowerMode: Bool
+
+    public static func current() -> BenchDeviceInfo {
+        BenchDeviceInfo(
+            model: modelIdentifier(),
+            osVersion: ProcessInfo.processInfo.operatingSystemVersionString,
+            gpuName: MTLCreateSystemDefaultDevice()?.name ?? "unknown",
+            processorCount: ProcessInfo.processInfo.processorCount,
+            physicalMemoryBytes: ProcessInfo.processInfo.physicalMemory,
+            thermalStateAtStart: ProcessInfo.processInfo.thermalState.rawValue,
+            isLowPowerMode: ProcessInfo.processInfo.isLowPowerModeEnabled
+        )
+    }
+
+    private static func modelIdentifier() -> String {
+        #if os(macOS)
+            let key = "hw.model"
+        #else
+            let key = "hw.machine"
+        #endif
+        var size = 0
+        sysctlbyname(key, nil, &size, nil, 0)
+        guard size > 0 else { return "unknown" }
+        var buffer = [CChar](repeating: 0, count: size)
+        sysctlbyname(key, &buffer, &size, nil, 0)
+        return String(cString: buffer)
+    }
+}
+
+/// Render settings the app chose, so a baseline is never compared across different ones.
+public struct BenchRenderSettings: Codable, Sendable {
+    public var platform: String
+    public var viewportWidth: Int = 0
+    public var viewportHeight: Int = 0
+    public var viewCount: Int = 1
+    public var layout: String = "single"
+    public var foveation: Bool = false
+    public var renderQuality: Double = 1.0
+    public var immersion: String = "none"
+    /// Whether presentation is paced by the display. True everywhere: on macOS and iOS the view
+    /// presents at the display's maximum refresh rate, on visionOS the compositor paces frames.
+    /// Frame time is therefore bounded below by the refresh interval; GPU time, per-pass and
+    /// per-system times are the cost metrics for scenes that fit in a frame.
+    public var vsync: Bool = true
+    /// Refresh rate the frame budget is derived from (0 when unknown).
+    public var displayRefreshHz: Double = 0.0
+    /// Anti-aliasing mode the scenes ran with (the post-FX scene always uses SMAA).
+    public var antiAliasing: String = "fxaa"
+
+    public init(platform: String) {
+        self.platform = platform
+    }
+}
+
+public struct BenchSceneResult: Codable, Sendable {
+    public var id: String
+    public var title: String
+    public var notes: String
+    /// Recording file, relative to the run directory.
+    public var file: String
+    public var warmupSeconds: Double
+    public var measureSeconds: Double
+    public var summary: EngineStatsRecordingSummary
+    /// The last published snapshot of the scene, for the compositor, hitch and pass details.
+    public var lastSnapshot: EngineStatsSnapshot
+}
+
+public struct BenchRunSummary: Codable, Sendable {
+    public var runID: String
+    public var label: String
+    public var startedAt: String
+    public var finishedAt: String = ""
+    public var device: BenchDeviceInfo
+    public var render: BenchRenderSettings
+    public var scenes: [BenchSceneResult] = []
+
+    public init(runID: String, label: String, device: BenchDeviceInfo, render: BenchRenderSettings) {
+        self.runID = runID
+        self.label = label
+        startedAt = ISO8601DateFormatter().string(from: Date())
+        self.device = device
+        self.render = render
+    }
+
+    /// Writes `summary.json` into `directory` and returns the encoded data.
+    @discardableResult
+    public func write(to directory: URL) throws -> Data {
+        let encoder = JSONEncoder()
+        encoder.outputFormatting = [.prettyPrinted, .sortedKeys]
+        encoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "inf", negativeInfinity: "-inf", nan: "nan")
+        let data = try encoder.encode(self)
+        try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+        try data.write(to: directory.appendingPathComponent("summary.json"))
+        return data
+    }
+
+    /// One line per scene for the console.
+    public var consoleReport: String {
+        var lines: [String] = []
+        lines.append("PerfBench \(runID) on \(device.model) (\(device.gpuName), \(device.osVersion)) \(render.platform) \(render.viewportWidth)x\(render.viewportHeight) x\(render.viewCount)")
+        for scene in scenes {
+            let s = scene.summary
+            let miss = s.deadlineSamples > 0
+                ? String(format: " deadlineMiss %.2f%%", Double(s.missedDeadlines) / Double(s.deadlineSamples) * 100)
+                : ""
+            lines.append(String(
+                format: "  %@: frames %d | mean %.2f p95 %.2f p99 %.2f worst %.2f ms | over budget %d | gpu %.2f ms%@ | thermal %d",
+                scene.id, s.frames, s.meanFrameMs, s.p95FrameMs, s.p99FrameMs, s.worstFrameMs,
+                s.framesOverBudget, s.meanGPUExecutionMs, miss, s.worstThermalState
+            ))
+        }
+        return lines.joined(separator: "\n")
+    }
+}