[610.57.04 / GA102] Mapping-node failures in gpu_vaspace.c (both 4635/2066 and 4551/2021 forms) after sustained Vulkan load pins VRAM at ~96% under Wayland

[ivorycrayon]

Summary

On an RTX 3080 (GA102) with nvidia-open-dkms 610.57.04, Deadlock (Source 2)
running its native Vulkan renderer under a Wayland compositor throws repeated
duplicate-mapping-node failures in gpu_vaspace.c after roughly 100 minutes of
sustained load. The GPU does not
hang and no Xid is logged, but frametimes degrade sharply in the same window
and the game process eventually exits.

This is the NV_ERR_INVALID_ARGUMENT / duplicate-node variant, not the
NV_ERR_NO_MEMORY exhaustion variant covered by #1165 and #1140. VRAM peaked at
6.4 GiB of 10 GiB, so nothing was close to full.

Error signature

Three identical bursts, at 102, 103 and 104 minutes into a 105-minute session:

NVRM: nvCheckFailedNoLog: Check failed: 0 == (pMapNode->gpuMask & gpuMask) @ gpu_vaspace.c:4551
NVRM: nvCheckOkFailedNoLog: Check failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT]
      (0x0000001F) returned from _gvaspaceMappingInsert(pGVAS, pGpu, pVASBlock, vaLo, vaHi, flags)
      @ gpu_vaspace.c:2021
NVRM: nvCheckFailedNoLog: Check failed: NV_OK == status @ virt_mem_allocator_gm107.c:2560
NVRM: dmaAllocMapping_GM107: can't update VA space for mapping @vaddr=0xa200000

The third burst reported @vaddr=0x2a3e00000; the first two both reported
@vaddr=0xa200000, the same address twice.

0 == (pMapNode->gpuMask & gpuMask) failing means the driver found a mapping
node already covering the requested GPU for that VA range, so this is a
duplicate DMA mapping registration rather than an allocation failure.

Environment

GPU	RTX 3080 10 GiB, GA102, 10de:2206 rev a1
Driver	nvidia-open-dkms 610.57.04, open kernel module
Kernel	7.2.3, Arch Linux
Compositor	Hyprland (Wayland)
Application	Deadlock (appid 1422450), Source 2 native Vulkan renderer via -vulkan. Windows build under Proton experimental-11.0-20260910b. DXVK is not in the path
IOMMU	iommu=pt, GPU group domain type identity
PCIe	Gen4 x16, CPU-direct root port, 2-device IOMMU group
ReBAR	active, BAR1 16384 MiB (EnableResizableBar module param reads 0, so this is firmware)
Peak VRAM	6.4 GiB of 10 GiB

Reproduction

Not reliably reproducible on demand. It correlates with session length rather
than with any particular action:

• Sessions of 1.8, 2.8 and 0.6 minutes: clean.
• A 104.7-minute session: three bursts, all after the 102-minute mark.
• A subsequent session was still clean at 55 minutes.

This matches the 30-to-45-minute onset other reporters describe for the
exhaustion variant, but stretched, and without exhaustion being reached.

Measured impact

Frametime percentiles per 10-minute window of the 104.7-minute session, MangoHud
at a 100 ms log interval, first 60 s of load-in dropped:

window	p99 (ms)	p99.9 (ms)	frames >33 ms
first 80 minutes, eight windows	10.2 to 11.9	12 to 33	0 to 5
minutes 80 to 90	13.23	91.29	18
minutes 90 to 100	9.53	12.78	0
minutes 100 to 105	30.54	36.42	7

The final window, which contains all three bursts, has a p99 three times every
clean window's. Median frametime stays flat at about 6 ms throughout, so this is
purely a tail effect.

What was ruled out

• Not thermal. 79 to 81 C, HW Thermal Slowdown and SW Thermal Slowdown
  counters both at 0 us. Only SW Power Cap was active, which is the card at
  its 320 W board limit.

• Not VRAM exhaustion. 6.4 GiB peak of 10 GiB.

• Not DXVK. Source 2 ships rendersystemvulkan.dll, and the game is launched
  with -vulkan so D3D11 translation is out of the path entirely. Verified by
  thread names rather than mapped DLLs, since Source 2 probes and maps both
  render backends at startup either way. Full launch options:

  gamemoderun mangohud __NV_PRIME_RENDER_OFFLOAD=1 VK_LOADER_DRIVERS_SELECT=nvidia_icd.json %command% -vulkan
  

  Worth noting that this game also produced Xid 109 CTX SWITCH TIMEOUT hard
  hangs when it ran through DXVK, which stopped when the renderer was switched.
  Those are a separate failure mode and are not what this report is about; the
  mapping fault below fires under the native Vulkan renderer with no Xid at
  all.

• Not IOMMU translation. iommu=pt was applied specifically to test this and
  the fault survived it unchanged. Domain type reads identity.

• Not PCIe topology. The card was moved from a chipset-attached Gen4 x4 link
  to a CPU-direct Gen4 x16 link, and the fault survived that too.

• No Xid was logged at any point, and the compositor stayed responsive
  throughout, so this is distinct from a GPU hang.

Related

• [GTX 1650 4GB] VRAM fragmentation causes nvAssertFailedNoLog in virt_mem_allocator_gm107.c leading to Wayland crash and system deadlock on shutdown #1165 and RTX 4090 (AD102): dmaAllocMapping_GM107 VA space exhaustion → RC watchdog GPU lock after 7 days uptime (nvidia-open 595.58.03) #1140 are the NV_ERR_NO_MEMORY exhaustion variant. Same call path,
  different failure mode.
• The duplicate-node variant appears in one public report I can find, an RTX 5080
  on 575.153.02 under Wayland, with the same 0 == (pMapNode->gpuMask & gpuMask)
  check and the same _gvaspaceMappingInsert rejection:
  https://forums.developer.nvidia.com/t/575-153-02-btree-mapping-crash-during-dune-awakening/336878
  That report received no reply.
• NVIDIA internal ticket #5762513 tracks the dmaAllocMapping_GM107 family
  as a Wayland-only issue, per
  https://forums.developer.nvidia.com/t/bug-report-wayland-only-dmaallocmapping-gm107-va-mapping-failures-does-not-reproduce-on-x11/353598
  That thread lists GA102 among affected hardware and gives an affected range of
  580.105.08 through 610.43.02. This report is on 610.57.04, past that range.

Happy to run instrumented sessions or collect additional logs on request.

[ivorycrayon]

Correction from the reporter: this is VRAM exhaustion, and my original framing was wrong

I need to retract the central claim of this report. I wrote that VRAM "peaked at
6.4 GiB of 10 GiB, so nothing was close to full". That figure was a single
nvidia-smi spot reading taken after the session had ended, not a measurement of
the session itself. Having gone back to the 1 Hz telemetry, both faulting
sessions run the card essentially full:

session	samples	median	p99	peak
104.7 min	6288	8112 MiB	9813 MiB	9866 MiB
62 min	3718	8240 MiB	9823 MiB	9860 MiB

Against a 10240 MiB card, that is 96.3%. So this belongs with the exhaustion
cases in #1165 and #1140 after all, and the "not VRAM exhaustion" in the title is
wrong. Apologies for the noise.

What the telemetry does add

The approach to the ceiling is gradual, and the fault does not fire on arrival.
VRAM from the second session, 1 Hz sampling:

22:01   7891 MiB   u=98%  292W
22:15   7367 MiB   u=98%  288W      <- flat around 7.4 GiB for ~30 min
22:29   7955 MiB   u=98%  284W
22:32   8730 MiB   u=99%  278W      <- ramp begins
22:34   9264 MiB   u=98%  274W
22:40   9520 MiB   u=99%  282W
22:43   9829 MiB   u=98%  285W      <- ceiling reached
22:57   9804 MiB   u=97%  289W
22:59   9792 MiB   u=94%  283W      <- FAULT (both bursts)
23:00   9637 MiB   u=100% 174W      <- power collapses at full utilisation

Two things worth noting:

1. The card sat at 95 to 96% for roughly 16 minutes before the first burst.
   Hitting the ceiling is not sufficient; something accumulates while pinned
   there. That is consistent with the fragmentation framing in [GTX 1650 4GB] VRAM fragmentation causes nvAssertFailedNoLog in virt_mem_allocator_gm107.c leading to Wayland crash and system deadlock on shutdown #1165 rather than
   a plain out-of-memory.
2. Power collapses from ~288 W to ~174 W while utilisation reads 100%. The
   GPU is busy and not doing work, which is what the frametime tail is measuring.

The other signature

The bursts in this second session are the intermapRegisterDmaMapping form:

NVRM: intermapRegisterDmaMapping: Failed to insert new mapping node for range 0xDF57FB0000-0xDF57FBFFFF!
NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:4635
NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: NV_OK == status @ gpu_vaspace.c:2066

So this box produces both forms on the same driver build: the 4635 / 2066
insert failure above, and the 4551 / 2021 gpuMask duplicate-node rejection
in the original report. Both fire near the VRAM ceiling. Whether they are two
symptoms of one fragmentation problem or two separate paths is beyond what I can
determine from outside the driver, but the 4551 duplicate-node form is the one
I could find only one other public report of, so it may still be the more useful
signal.

The rest of the original report stands: no Xid, compositor survives, native
Vulkan renderer with DXVK out of the path, and both iommu=pt and a move to a
CPU-direct Gen4 x16 link tested without effect.

[Patrick-Sutton]

I’m seeing the same gpu_vaspace.c:4551 / _gvaspaceMappingInsert() / dmaAllocMapping_GM107 failure on Fedora 44 with the NVIDIA 610.57.04 open kernel module under KDE Plasma Wayland.

System:

Fedora Linux 44 (KDE Plasma Desktop Edition)
Kernel: 7.2.4-200.fc44.x86_64
GPU: NVIDIA GeForce RTX 4080 SUPER
Driver: 610.57.04
Session: KDE Plasma / Wayland

With the open kernel module, I repeatedly observed:

NVRM: nvCheckFailedNoLog: Check failed: 0 == (pMapNode->gpuMask & gpuMask) @ gpu_vaspace.c:4551
NVRM: nvCheckOkFailedNoLog: Check failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT] (0x0000001F) returned from _gvaspaceMappingInsert(pGVAS, pGpu, pVASBlock, vaLo, vaHi, flags) @ gpu_vaspace.c:2021
NVRM: nvCheckFailedNoLog: Check failed: NV_OK == status @ virt_mem_allocator_gm107.c:2560
NVRM: dmaAllocMapping_GM107: can't update VA space for mapping @vaddr=0x9600000

I also saw repeated:

[drm:nv_drm_gem_alloc_nvkms_memory_ioctl [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NVKMS memory for GEM object

During the same period, I was also seeing newly introduced stuttering in Forza Horizon 6 and intermittent Steam Game Recording export/encode/decode issues.

I then performed an A/B test using the same 610.57.04 userspace driver, same Fedora install, same kernel, and same hardware. I rebuilt the RPM Fusion akmod-nvidia package with open-module auto-detection disabled so that the proprietary 610.57.04 kernel module was loaded instead.

Before the switch:

license: Dual MIT/GPL
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 610.57.04

After the switch:

license: NVIDIA
NVRM version: NVIDIA UNIX x86_64 Kernel Module 610.57.04

After rebooting with the proprietary module, I tested for more than 30 minutes while:

• playing Forza Horizon 6
• screen sharing through Discord
• recording intermittently
• using roughly 11.4 GiB of VRAM during gameplay

The system remained stable. The previous game/video issues did not recur, and:

journalctl -k -b | grep -iE 'NVRM|Xid|NVKMS|gpu_vaspace|dmaAllocMapping'

only returned the normal module-load message:

NVRM: loading NVIDIA UNIX x86_64 Kernel Module 610.57.04

There were no gpu_vaspace, dmaAllocMapping, NVKMS allocation, or NVIDIA Xid errors during that test.

I generated nvidia-bug-report.log.gz while the system was still running the failing open kernel module, and it contains the repeated VA-space and NVKMS failures. I’m attaching a sanitized copy here, along with a cleaned baseline captured while the open module was loaded.

On my system, the issue currently appears specific to the open kernel module path, since proprietary 610.57.04 has remained clean under a heavier version of the same workload.

nvidia-bug-report-public.log.gz
nvidia-open-baseline.txt