[native] Investigation: GC bridge gref churn is not the cause of CoreCLR ART GC regression (informational, do not merge)

[jonathanpeppers]

Note

Informational only — not intended for review or merge.

This branch records a measured negative result for dotnet/runtime#131370, so the next person doesn't re-run this experiment. The instrumentation is correct and verified, but it does not fix the reported problem, and it kills one more hypothesis. Nobody needs to review it.

Background

dotnet/runtime#131370 reports periodic frame-rate drops (60→54 FPS every ~5–8 s) in a .NET MAUI + SkiaSharp game after moving from Mono to CoreCLR (the .NET 11 default). A prior investigation (draft PR #12262) established that the entire delta lives in ART's concurrent copying-GC phase — with an identical managed root set (~840 grefs), identical Runtime.gc() trigger, identical stop-the-world pause (µs), and identical heap occupancy — and killed three hypotheses (larger gref root set, bridge thread priority, missing WaitForGCBridgeProcessing barrier).

This branch tests the last structural difference #12262 called out: does CoreCLR's GC bridge issue far more JNI weak/strong global-reference transitions per bridge round than Mono? That churn is invisible to the managed JniEnvironment.Runtime.GlobalReferenceCount because it is a native-side count.

What I did

Both GC bridges use the identical algorithm, confirmed by reading the code:

• prepare: iterate every SCC × object, flip each peer strong→weak (NewWeakGlobalRef + DeleteGlobalRef)
• java.lang.Runtime.gc()
• cleanup: iterate every SCC × object, flip each peer weak→strong (NewGlobalRef + DeleteWeakGlobalRef)

= exactly 4 JNI global-ref ops per peer per round on both runtimes.

• CoreCLR: BridgeProcessingShared::process() — src/native/clr/host/bridge-processing.cc
• Mono: OSBridge::gc_cross_references() — src/native/mono/monodroid/osbridge.cc

I added one unconditional log_warn per round to each, so the per-round churn can be compared directly against ART's GC wall time:

GCBRIDGE_CHURN runtime=<coreclr|mono> sccs=<n> peers=<n> xrefs=<n> jni_transitions=<n>

Both instrumented native libs were byte-verified inside the packaged APKs before trusting any numbers.

The result: performance-neutral / inverted

Same local SDK, same device (Pixel 5, Android 14, arm64, refresh rate pinned to 60 Hz), 60 s per arm, two rounds in reversed order to control for ordering. Steady state (startup + first bridge round dropped as warm-up), no contamination:

Arm	Rnd	peers/round	JNI transitions/round	ART GC avg	ART GC max	FPS
CoreCLR	1	442	1768	20.9 ms	22.0 ms	59.77
CoreCLR	2	442	1767	22.9 ms	26.4 ms	59.76
Mono	1	503	2012	10.5 ms	10.9 ms	59.87
Mono	2	502	2010	10.6 ms	11.1 ms	59.82

ART GC columns are Explicit concurrent copying GC total wall time, filtered to the benchmark process.

• CoreCLR issues ~12% fewer JNI global-ref transitions per round than Mono (≈1767 vs ≈2011).
• Yet CoreCLR's ART concurrent GC is ~2.1× slower (≈21.9 ms vs ≈10.6 ms).
• The relationship is inverted: the runtime doing more gref churn (Mono) is the faster one.

So per-round native global-ref churn does not explain the ~13 ms concurrent-phase delta.

Bonus: also rules out the SCC/cross-ref machinery

On both runtimes, every round reported sccs == peers and xrefs == 0. Every bridged object is a singleton strongly-connected component with no cross-references, so monodroidAddReference / monodroidClearReferences, the circular-reference wiring, and the temporary-peer path are never exercised by this benchmark. Those are not the differentiator either.

What this rules out (cumulative with #12262)

Hypothesis	Result
CoreCLR holds a larger JNI global-ref root set	Dead (#12262) — ~840 on all arms; CoreCLR holds ~3% fewer
Bridge thread runs at a lower priority	Dead (#12262) — both nice −10 / PRI 29
Missing WaitForGCBridgeProcessing barrier	Dead (#12262) — restored, verified firing, no gain
CoreCLR issues more native gref transitions per round	Dead (this branch) — CoreCLR issues fewer, yet is 2.1× slower
SCC circular-ref / add/clear-reference machinery	Dead (this branch) — never exercised; xrefs==0, sccs==peers

Where I'd look next

The delta is in ART's concurrent phase under an identical bridge algorithm, identical trigger cadence, identical managed root set, and now near-identical (slightly lower for CoreCLR) native gref churn. The cause is not the count of bridge JNI operations. Remaining structural suspects:

• Per-object ART work resolving each weak global — differences in the Java-side peer wrapper objects / GCHandle registration between the two runtimes, rather than the number of transitions.
• Scheduling of Runtime.gc() relative to ART's concurrent worker — whether the bridge thread's timing causes ART's concurrent copy to overlap differently with app allocation on CoreCLR.

The code, for the record

Two files, +20 lines, instrumentation only — one per-round summary log_warn appended to each runtime's existing bridge round entry point. Gated by nothing (always on) purely so the experiment is reproducible; it would be removed or made #if DEBUG before any real use.

Fixes nothing. Closes nothing. Recorded so the next person doesn't repeat it. Related: dotnet/runtime#131370, #12262.

[jonathanpeppers]

Root cause found via Perfetto: DVFS frequency starvation of the GC-bridge thread

Following the negative gref-churn result above, I captured matched Perfetto system traces (ftrace sched_switch + cpu_frequency + ART atrace) for both runtimes on the Pixel 5, three Explicit concurrent copying GC events inside each 20 s window. This identifies the actual mechanism.

1. The ART GC runs single-threaded on the bridge thread, with zero contention

On both runtimes the ART Explicit concurrent copying GC runs synchronously, single-threaded, on one big core (cpu6), as a single unbroken scheduling segment. HeapTaskDaemon sleeps the whole time. Wall-time == on-CPU time, so it is not CPU contention, not scheduling preemption, and not more threads competing.

• CoreCLR carrier: Thread-7 (the dedicated GC-bridge pthread) — cpu6, 1 segment, state=Running the whole window.
• Mono carrier: GLThread 3 (the render thread) — cpu6/cpu7, 1 segment.

2. Every ART scan/copy sub-phase is uniformly ~2.2× slower on CoreCLR — but the heap is identical

Sub-phase totals across the 3 GCs (ms):

ART sub-phase	CoreCLR	Mono	ratio
CopyingPhase	64.73	28.98	2.23×
└ Process mark stacks & References	46.94	20.73	2.26×
ScanImmuneSpaces	7.96	3.81	2.09×
VisitConcurrentRoots	7.96	3.48	2.29×
ReclaimPhase / RecordFree	5.13 / 4.71	2.75 / 2.50	~1.9×
SweepSystemWeaks	0.53	0.47	1.1× (parity)

The ART heap logs are identical: CoreCLR 3693–3745 KB/27 MB, 86 % free, ~400 KB freed; Mono 3648–3664 KB/27 MB, 87 % free, ~450 KB freed. Same live bytes, same garbage freed, same core — yet every phase costs 2.2×. That means ART does the same work at ~half the clock speed. (SweepSystemWeaks at parity independently re-confirms weak-global churn is not the differentiator.)

3. The CPU is clocked ~2.3× lower during CoreCLR's GC

Time-weighted frequency of the carrier core inside each GC window:

	carrier thread	avg freq during GC	GC wall-time
CoreCLR	Thread-7 (dedicated GC-bridge pthread)	866 / 941 / 1094 MHz	27 / 25 / 20 ms
Mono	GLThread 3 (render thread)	2090 / 2208 / 2400 MHz	12 / 11 / 10 ms

The frequency ratio (~2.3×) matches the GC-time ratio (~2.2×). Governor is schedutil (utilization-driven DVFS). CoreCLR's cpu6 was seen at 653 MHz — the lowest step in the table — 18 ms into the GC, only ramping to 941 MHz by 25 ms: the governor ramps during the GC but the burst ends before it reaches max.

4. Why: the bridge thread is idle 99.6 % of the time

Carrier-thread duty cycle over the 20 s trace:

carrier	on-CPU	duty	segments
CoreCLR Thread-7	79.8 ms	0.4 %	9 (sleeps 7–8 s on a semaphore between bridge rounds)
Mono GLThread 3	19022 ms	95.3 %	1350 (busy every frame)

GCBridge::bridge_processing() (src/native/clr/host/gc-bridge.cc:101) loops: wait_for_shared_args() (long semaphore sleep) → process() which calls Runtime.gc() — and ART runs the explicit concurrent-copying GC synchronously on this thread. Because the thread has near-zero scheduler utilization when it wakes, schedutil keeps its core near the bottom of the frequency table for the entire short GC burst. Mono has no dedicated bridge thread here — the explicit GC is driven from the always-hot render thread, whose core is already pinned at max clock, so the identical GC finishes in ~half the time.

This is the whole 13 ms concurrent-phase delta, and it is a .NET-side (dotnet/android) design difference, not app behavior and not anything ART-specific.

Proposed fix (dotnet/android, .NET-side — not an app workaround)

Raise the GC-bridge thread's util-clamp floor so schedutil ramps the CPU up during the rare, latency-critical bridge GC: sched_setattr() with SCHED_FLAG_UTIL_CLAMP_MIN and sched_util_min near 1024, set once after pthread_create in GCBridge::start_bridge_processing_thread (or around the process() round). uclamp_min only affects frequency while the thread is actually on-CPU (0.4 % duty), so the power cost is negligible. Optionally combine with big-cluster affinity. Expected effect: CoreCLR explicit-GC wall-time ~24 ms → ~10 ms (Mono parity), removing the periodic 60→54 FPS dips. This still needs a build + on-device A/B to confirm.

(No root on the device, so I could not force the governor to performance as an independent check; the uclamp build is the validation path.)

Related: dotnet/runtime#131370.