feat: arm64 CUDA prebuilt binaries

[giladgd]

Description of change

• feat: arm64 CUDA prebuilt binaries
• docs: reference Semantic Golfer

Fixes #651

Pull-Request Checklist

• Code is up-to-date with the master branch
• npm run format to apply eslint formatting
• npm run test passes with this change
• This pull request links relevant issues as Fixes #0000
• There are new or updated unit tests validating the change
• Documentation has been updated to reflect this change
• The new commits and pull request title follow conventions explained in pull request guidelines (PRs that do not follow this convention will not be merged)

[hassanhabib]

@giladgd Ran it on the GB10. The CUDA build works and every inference test passes. 27 of 357 fail, and as far as I can tell none of them are about arm64 or CUDA. Details below so you can judge for yourself.

Tested eadb243d6024ffe34b05ceb0c3eb3dbc733fba6b ("build: fix arm64 CUDA install") on gilad/cudaArm64.

Environment

DGX Spark, NVIDIA GB10, compute capability 12.1, driver 580.178.04
Ubuntu 24.04.5 LTS (aarch64), DGX OS
CUDA 13.0.88, cmake 3.28.3, g++ 13.3.0, Node 22.23.2

Build and detection

npm ci, npm run build, then a source build with CUDAARCHS=121 all succeeded. inspect gpu:

node-llama-cpp: 0.1.0
Prebuilt binaries: latest
Cloned source: v0.5.0

CUDA: available
Vulkan: The Vulkan prebuilt binaries module is missing

CUDA device: NVIDIA GB10
CUDA used VRAM: 30.82% (37.5GB/121.69GB)
CUDA free VRAM: 69.17% (84.18GB/121.69GB)
CUDA unified memory: 121.69GB (100%)

Results

npm run test:typescript   PASS
npm run lint:eslint       PASS
npm run test:vitest       Test Files  5 failed | 64 passed (69)
                          Tests  27 failed | 322 passed | 8 skipped (357)
                          Duration 359.44s

Everything that actually runs a model passes: chat sessions, function calling, LoRA, embeddings, reranking, parallel sequences, structured decisions, token prediction, sequence state, chunking, async load. So the binary is good and the kernels are correct on sm_121.

Also worth noting that predicted VRAM usage should match actual VRAM usage and the same test when using gpuLayers both pass, which is the pair I would have expected to break first on a unified memory part.

The 27 failures, grouped by what I think causes them

0/ A timezone-dependent test. Not arm64.

test/standalone/chatWrappers/MuseChatWrapper.test.ts

- Current date: 11-08-2026.
+ Current date: 10-08-2026.

The test passes todayDate: new Date("2026-08-11T00:00:00Z") and expects 11-08-2026. This machine is America/Los_Angeles, so UTC midnight on the 11th is 5pm on the 10th locally and the formatter renders the local date. I would expect this to fail on any machine behind UTC and to pass in CI. Nothing to do with this PR.

1/ A token metadata difference. Probably not arm64.

test/modelDependent/llama3.2/promptCompletion.test.ts

- new SpecialToken("EOS"),
+ new SpecialToken("EOT"),

The expectation is hardcoded to EOS. Worth checking whether the Llama 3.2 GGUF that downloadAllTestModels pulls has been re-uploaded, because this looks like model file drift rather than anything platform specific.

2/ A snapshot pinned to other hardware.

test/modelDependent/functionary/gguf/ggufInsights.test.ts > estimated model memory footprint stays the same

-   "gpuVram": "442.52MB",
+   "gpuVram": "410.99MB",

A recorded constant rather than a prediction-versus-reality check. The prediction-versus-reality tests in the same file pass.

3/ GPU layer and context resolution. This is the one that is genuinely about unified memory.

functionaryModelGpuLayersOptions.test.ts and stableCodeModelGpuLayersOptions.test.ts, 24 of the 27 failures. The resolved context sizes come out slightly larger than expected:

Expected: "1280"     Expected: "6656"
Received: "1536"     Received: "6912"

The failing cases are named no RAM, some RAM, some unified RAM, with swap, 5GB RAM, 8GB RAM, so they simulate memory conditions. On this part vram.total, ram.total and unifiedSize are all 121.69GB, and I suspect the simulation does not model a machine where VRAM and RAM are the same pool. That is a question about the test fixtures rather than about the binary, and you will know immediately whether the new numbers are correct behaviour or a real defect. I did not want to guess.

Offer

Happy to re-run any of this, run a subset, or try a patch. The repo is still checked out with all 13 test models cached, so a re-run is quick now.

On the self hosted runner, I am working through what it would take to do it safely. I will follow up.

[giladgd]

@hassanhabib Thank you very much for running the testsuite :)
Did you see any worker crashed or similar error from Vitest?

The results of some of the tests come out a bit differently across different GPU backends and machine configurations (and also llama.cpp updates), so I usually align them to pass on my mac machine to make it easier for me to develop (and also since mac machines tend to have fewer variations and so the compute results often match, like in the CI for example where I use a mac machine for the tests), so in the meantime I just manually look at the failing tests when running with CUDA and Vulkan to detect a trend or a larger issue that needs fixing, rather than neccessarily have all of them strictly pass.
So by your test results summary it looks like everything works correctly on your machine.

Another thing that I'd love if you could do is to run these commands inside the repo:

node ./dist/cli/cli.js inspect measure hf:giladgd/gemma-4-31B-it-GGUF:Q4_K_M
node ./dist/cli/cli.js inspect measure hf:giladgd/gpt-oss-20b-GGUF/gpt-oss-20b.MXFP4.gguf

Those commands will help me to figure out whether node-llama-cpp's calculations are correct on CUDA with unified memory.

[hassanhabib]

@giladgd No worker crashed and nothing like it. I grepped the full vitest run for worker crashed, SIGSEGV, SIGABRT, core dumped, CUDA error, out of memory and unhandled rejections, and all of them are zero. The 27 failures were assertion mismatches only, so your read is right.

Both inspect measure runs completed cleanly. gemma-4-31B took 2716s, gpt-oss-20b took 340s.

The short version

Model weight estimates are good. Context estimates are consistently under, and the error grows the fewer layers are on the GPU.

Median error of estimated versus actual context VRAM, grouped by how much of the model was offloaded:

layers on GPU	gemma-4-31B Q4_K_M	gpt-oss-20b MXFP4
80 to 100%	-8.4% (n=116)	-2.4% (n=53)
50 to 80%	-22.7% (n=162)	-6.5% (n=63)
20 to 50%	-44.4% (n=139)	-11.6% (n=71)
0 to 20%	-57.7% (n=107)	-16.1% (n=36)

Model VRAM for comparison, across all rows:

	median	p25	p75
gemma-4-31B	-3.5%	-4.6%	-2.1%
gpt-oss-20b	-1.6%	-2.8%	-0.3%

Every context row on gemma is negative, from -0.8% at the tightest to -78.6% at the loosest. On gpt-oss the range is -20.0% to +2.9%.

A hypothesis, offered as a hypothesis

I have not read the estimation code, so treat this as a direction rather than a diagnosis. The error being monotonic in offload fraction, and present in both models at different magnitudes, looks like context memory for non-offloaded layers being attributed to CPU RAM rather than to VRAM. On a discrete GPU that attribution is correct and invisible. On this part there is one coherent pool, so vram.total, ram.total and unifiedSize are all 121.69GB and the measurement reads it all as VRAM. The more layers stay on the CPU side, the more context lands in the bucket the VRAM estimate does not count, which would produce exactly this gradient.

If that is what is happening, the fix is probably about which pool context is charged to when unifiedSize > 0, rather than about the size calculation itself.

Why the direction of the error matters here

Underestimating is the dangerous direction on a unified memory machine, and I can give you a concrete example from today. While these measurements were running, the pool reached 99.58% and the kernel OOM killer fired. It did not kill the GPU process in isolation, because there is no separate VRAM to run out of. It killed the largest resident process on the machine and took the desktop session with it.

On a discrete card, overshooting gives you a CUDA allocation failure that a program can catch. Here it gives you a dead process chosen by the kernel. That seems like an argument for erring high on unified parts, though you will have a better sense than I do of what that costs elsewhere.

Sample rows

gemma-4-31B, 26 of 61 layers:

| Context | 26 M | 98816 | 7.94GB | 7.33GB | 623.64MB (08.30%) | 13.25GB | 18.39GB | -5.14GB (-27.93%) | 99.58% (121.17GB/121.69GB)
| Context | 26 M | 47104 | 7.94GB | 7.33GB | 623.64MB (08.30%) |  6.64GB | 10.52GB | -3.88GB (-36.90%) | 92.99% (113.16GB/121.69GB)

gpt-oss-20b, 25 of 25 layers:

| Context | 25 M | 131072 | 10.69GB | 11.42GB | -744.33MB (-6.37%) | 19.33GB | 20.01GB | -699.24MB (-3.41%) |
| Context | 25 M |  82688 | 10.69GB | 11.42GB | -744.33MB (-6.37%) | 12.21GB | 12.31GB | -100.3MB  (-0.80%) |

One outlier worth a look: a gemma model row at 31 layers reports -74.04% on model VRAM, well outside the -2% to -5% the rest of them sit in.

Raw output

The full tables are 587 rows for gemma and 249 for gpt-oss. Happy to paste them here, put them in a gist, or pull out any slice you want. Tell me which is easiest for you.

The checkout and all the test models are still on the machine, so re-runs are quick now.