Whole-system hard hang under sustained GPU rendering — no Xid, no kernel log, RTX 4060 Laptop

[najmmShora]

Summary

On an RTX 4060 Laptop with nvidia-open 610.57.04, sustained GPU rendering by a Vulkan
application hard-hangs the entire host — not the GPU, not the application. The machine stops
executing: display frozen, network dead, no response to input, power cycle required.

The hang is instantaneous and completely silent. There is no Xid, no NVRM error, no oops, no
soft- or hard-lockup report, and nothing in pstore — including when the kernel log is streamed
off-box in real time so that a dead filesystem cannot be the explanation.

Observed rate: roughly one hang per 29 minutes of sustained GPU rendering (4 events across
117 minutes of instrumented exposure). A matched CPU-only load is clean.

Environment

GPU	NVIDIA GeForce RTX 4060 Laptop (AD107M, 8188 MiB), 0000:01:00.0
Driver	nvidia-open-dkms 610.57.04 (GSP firmware active, EnableGpuFirmware: 18)
Kernel	Arch 7.2.3-arch1-3, CONFIG_PCIEAER=y, CONFIG_PCIEPORTBUS=y
Platform	ASUS ROG Strix G614JV, i9-13980HX, BIOS G614JV.333
Display	reproduced in both MUX-discrete (NVIDIA drives eDP) and hybrid (Intel drives eDP, dGPU used via -graphicsadapter=1)
Application	Unreal Engine 5.7, Vulkan RHI, SF_VULKAN_SM6, offscreen rendering via scene-capture components

Module parameters in force: NVreg_PreserveVideoMemoryAllocations=0,
NVreg_DynamicPowerManagement=0x00, nvidia_drm modeset=1.

Reproducer

A headless UE 5.7 application repeatedly creates a 1920x1080 scene-capture stream, renders for a
few seconds, tears it down, and repeats. GPU load during the run is steady at ~80 W / 92-94%
utilisation, VRAM ~4.3 GiB of 8 GiB.

The hang occurs at a random point, typically 10-40 minutes in. It is not deterministic and not
tied to a particular frame, resource, or code path we have been able to isolate.

The hang is a whole-platform stop, not a GPU or driver-visible fault

Three separate hangs were observed with a second machine watching over the LAN. Every independent
clock stops within the same one-second window:

sim application log (fsync'd)                   18:04:04
2 s telemetry sampler on the box (fsync'd)      18:04:04
1 Hz sampler on the box (fsync'd)               18:04:04
userspace heartbeat over SSH      (OFF-BOX)     18:04:04
ICMP echo reply                   (OFF-BOX)     18:04:04   <-- kernel softirq

ICMP echo replies are generated in kernel softirq context — no userspace, no filesystem, no
display. They stop simultaneously with everything else, which rules out a display/compositor wedge
and an I/O stall with a live kernel. The machine then fails ARP from the peer (Host is down).

Immediately before the stop, telemetry is entirely normal: GPU 87 C / 78.7 W / 94% util, VRAM 4331 MiB, CPU package 97 C. There is no precursor of any kind in any instrument.

Nothing is logged — and this is not an artifact of an unwritable disk

dmesg -w was streamed over SSH to the second machine for the duration, so kernel messages left
the box in real time and did not depend on the failing machine's filesystem. Across three hangs it
captured zero kernel messages in the minutes leading up to and including each event.

At the same time, all of the following were armed and produced nothing:

• kernel.panic_on_oops = 1, kernel.hardlockup_panic = 1, kernel.softlockup_panic = 1
• nmi_watchdog = 1 / watchdog = 1 (NMI watchdog confirmed enabled, consuming a hw-PMU counter)
• efi_pstore — proven working on this machine by a deliberate echo c > /proc/sysrq-trigger,
  which produced a complete symbolised panic record

So: no oops, no BUG, no soft lockup, no hard lockup, and no panic of any kind. The CPUs appear to
stop executing outright.

What we ruled out, and how

Each of these was tested rather than assumed:

Hypothesis	Result	Method
VRAM exhaustion	Not the trigger	Last durable sample before a hang: 5373 / 8188 MiB, ~2.7 GB free, mid-render and still climbing. The run with the largest allocator storm (4429 NV_ERR_NO_MEMORY + 5409 Vulkan alloc failures, peak 7835 MiB) survived.
Xid 109 -> 31	Not the signature	Fired 5 times; 4 were harmless per-process faults ending in an application SIGSEGV + coredump, host alive 6-44 min longer. Only 1 coincided with a hang, and that one produced no coredump.
Thermal	Not the variable	A CPU-only load at a hotter envelope (91.1 C mean, 44.5% of time >=95 C) ran 40 min clean; hangs occurred at 70-73 C means.
CPU / VRM instability	Not reproduced	32 threads of AES-NI at 91-96 C for 40 min, GPU idle at 2.3 W / 0% util — clean.
SMM / firmware stall	No SMI activity	MSR 0x34 (SMI_COUNT) sampled at 1 Hz: flat across all 3532 samples of a run that ended in a hang.
Driver-session age	Refuted	One boot survived 382 minutes with 36-40 application launches and 41,570 allocation failures, then shut down cleanly.
PCIe completion timeout	Excluded	Root port and endpoint both have completion timeout enabled (50us to 50ms, TimeoutDis-); AER status clean (UESta/CESta all clear), no AER errors ever logged.
Storage / IO stall with live kernel	Excluded	Off-box ICMP stops simultaneously (see above).
Display / compositor wedge	Excluded	Same.
MUX mode	Not relevant	Reproduced in both discrete and hybrid.

Controlled experiment — GPU rendering is necessary

All arms on the same machine, same day, GPU load held constant where applicable:

arm	GPU	exposure	hangs
CPU-only (32x AES-NI, GPU idle 2.3 W / 0%)	idle	39.9 min	0
GPU rendering, low stream churn	81.7 W / 94% util	40.2 min	0
GPU rendering, medium churn	79.7 W / 92% util	19.3 min	1
GPU rendering, high churn	79.0 W / 92% util	20.9 min	0
GPU rendering + segmentation passes	70-85 W / 91-94%	36.2 min	3

P(0 hangs in the CPU-only arm | the GPU-render rate) = 0.037.

We can state that GPU rendering is necessary. We cannot say which property of the workload
matters — with only 4 events, stream-churn rate and workload type are not separable
(p ~ 0.09), and two hypotheses we initially favoured were each refuted by the next experiment.

What we could not do — and the specific ask

We attempted to read GSP-RM firmware logs, which is the one layer our instrumentation cannot see:

nvidia 0000:01:00.0: Direct firmware load for nvidia/610.57.04/gsp_log_ga10x.bin failed, error -2
NVRM: RmFetchGspRmImages: Failed to load gsp_log_*.bin, no GSP-RM logs will be printed (non-fatal)

gsp_log_*.bin is not shipped. The driver source confirms this is deliberate —
kernel-open/common/inc/nv-firmware.h declares the firmware set as "gsp_ and ucodes_* (but not
gsp_log_)", and libos_log.h notes the log metadata and format strings are "stripped from the
image as the final build step".

Asks:

1. Is there any supported way to obtain GSP-RM log output on a release driver? That is the only
   remaining instrument on our side.
2. Are there NVreg_RmMsg / NVreg_ResmanDebugLevel settings you would like set for a repro run?
3. Is a fault of this shape — CPU execution stopping with no Xid and no driver-visible error —
   consistent with any known GSP or display-engine failure mode on AD107?

We can run any instrumented build or parameter set against the reproducer on request.

Attachments available

• Kernel and full journal logs for each hang boot
• Application logs with -forcelogflush
• 2 s GPU/CPU telemetry CSV spanning all events (~32k samples)
• 1 Hz SMI_COUNT + heartbeat CSV
• Off-box ping / SSH-heartbeat / dmesg -w captures from the observing machine
• nvidia-bug-report.sh output

One methodological note

Two instruments reported success while being non-functional, which may be worth knowing if you ask
us to enable something: the iTCO_wdt watchdog logged initialized. heartbeat=30 sec but is
decorative on this chassis (a controlled test showed timeleft frozen at 29 for 90 s with no
reset), and NVreg_EnableGpuFirmwareLogs=1 reported armed while emitting nothing. Every instrument
quoted in this report was verified by its observable effect, not by its own status message.