NERVA gets the most from the hardware you already own, keeps the model exact, and turns one goal into a built, verified project.
NERVA is the accuracy engine for open models, built by Dacely. It makes the open model you already run answer better, and it turns a single goal into a built, verified project. Tier one results, on the GPU you already own.
Two ideas sit under everything NERVA does. The first is that it uses the maximum compute a system can give, whatever the configuration is. The auto tuner reads the machine and gets the best performance out of it from the first run, with no flags to set. The second is that the model stays exact. NERVA does not quantize, prune, or drop context to look fast, and a greedy decode is byte for byte identical to a reference decode. What changes is the execution machine around the weights, and what the model earns the right to say.
Today NERVA runs as a Rust and CUDA inference runtime with an OpenAI compatible server, and it already beats a rented endpoint out of the box on hardware you own. Its largest part is still in progress: general problem solving, taking one goal and working it the way a team works a product, all the way to a built and verified result.
NERVA is written in Rust and CUDA, and runs on Linux and Windows.
NERVA is a well engineered inference server that competes by being disciplined about memory, not by inventing new math. The v1 job is to use the best proven techniques as well as they can be used: paged KV, flash attention, continuous batching, byte exact decoding. Where NERVA is faster it is because it drags the weights through memory fewer times, not because it found a faster kernel or a better complexity class. The wins are engineering wins.
That is the point. Prove the current techniques work, end to end, on real hardware, before reaching for anything new. Innovation earns its place after the ground under it is solid, not before. A server that is byte exact, fast on the card you already own, and free of clever tricks that only work in a benchmark is the foundation everything else gets built on.
NERVA is tuned to run a tier one model on a card you already bought, next to whatever else that card is doing, and it still beats the rented endpoint out of the box. When a session starts, the auto tuner probes the card and every architecture from sm_75 up picks its own fastest kernel, so the machine gives all the compute it has with no manual tuning. A card from 2018 is a first class target. On a Turing GPU, NERVA re encodes the resident weights in place so the work lands on the tensor cores instead of falling off the memory roofline.
Here is batched decode throughput in tokens per second on one RTX 2080 Ti, against vLLM 0.24 on the same card.
| Batch size | NERVA | vLLM 0.24 | Gain |
|---|---|---|---|
| 32 | 3,342 | 3,027 | 10% |
| 64 | 4,982 | 4,483 | 11% |
| 128 | 6,711 | 6,095 | 10% |
The output is byte exact against NERVA's own pure decode path. On multiple choice scoring the gap is larger: about twice the throughput on MMLU, 240 iterations per second against 118, at bit identical 76.84% accuracy. That win comes from doing fewer forward passes, not from a faster kernel. A memory guard reads the real free memory and caps the context to what fits, so NERVA refuses cleanly rather than crashing a card that is already in use.
NERVA runs the dense Qwen3.6-27B (a GDN hybrid, 55 GB of weights) split tensor parallel across eight RTX 2080 Ti — 88 GB of Turing silicon that no single modern card matches on price. It re encodes each shard's weights to fp16 tensor cores, fans the eight way ring out to one host thread per GPU, and packs concurrent requests into one mixed decode step. Against vLLM running the same model fp16 on the same eight cards:
| Mode | NERVA | vLLM | Gain |
|---|---|---|---|
| Single stream | 29 tok/s | 6.6 tok/s | 4.4x |
| Batched decode, 32 streams | ~300 tok/s | 115 tok/s (at 64 streams) | ~2.6x |
System: 8x NVIDIA RTX 2080 Ti (sm_75 Turing, 11 GB each, 88 GB total), CUDA 13.1, driver
580.173.02, x86_64 Linux. Model Qwen3.6-27B (Qwen3_5ForConditionalGeneration, dense
GDN hybrid, 55 GB weights) loaded fp16, tensor parallel across all eight cards, greedy
(--temperature 0). The batched row is 32 concurrent completions of 64 tokens each
(NERVA_TP_REDUCE=fp16); its number is steady state decode throughput and the vLLM figure
is that engine's 64 way aggregate on the same eight cards.
Greedy output is byte for byte identical to vLLM. NERVA already beats vLLM's 64 way batch at half the concurrency, and it holds while every 11 GB card stays inside its budget — a place vLLM's default sizing overflows. The single stream number matters as much as the batch: a GDN hybrid decodes token by token, and NERVA is 4x faster there than the rented endpoint on the same silicon.
The longer the prompt, the more the prefill wins. Here is single stream prefill throughput in tokens per second for Qwen3 8B (bf16, YaRN) on one RTX 5090, against vLLM 0.23 on the same card, at greedy parity. Both stream the same weights; the difference is the attention kernel and how often the weights are re streamed.
| Prompt tokens | NERVA | vLLM 0.23 | Winner |
|---|---|---|---|
| 40k | 7,710 | 7,783 | vLLM 1% |
| 48k | 7,229 | 7,106 | NERVA 2% |
| 56k | 6,644 | 6,495 | NERVA 2% |
The curves cross near 44k: below it vLLM is a touch faster, above it NERVA takes the lead and holds a couple of percent through the deep band. Two things cause the flip. First, on a Blackwell card vLLM runs FlashAttention 2, whose cost per prefill grows with the square of the context, so its throughput slides; NERVA runs a cuDNN flash kernel that holds a flatter curve on the same hardware. Second, NERVA keeps its prefill working set wide at long context. A decode side scratch buffer that the cuDNN attention path never reads used to grow to gigabytes and crowd out the prefill scratch, shrinking it eight fold and forcing NERVA to re stream all sixteen gigabytes of weights many more times per prompt. Reclaiming that dead buffer keeps the prefill scratch wide, so the weights are streamed once per eight thousand prompt tokens instead of once per thousand. Both engines keep decaying past 56k and stay close, but vLLM hits its memory wall near 88k on a 32 GB card while NERVA, with fp8 KV, keeps going.
cargo build --release -p nerva
# One prompt, streamed to your terminal. NERVA picks the layout that fits on its own.
./target/release/nerva -m qwen3-8b -p "Explain the CAP theorem."
# OpenAI compatible server, continuous batching on by default.
./target/release/nerva serve -m qwen3-8b --port 8000
# Call it. Any OpenAI SDK works too, with base URL http://127.0.0.1:8000/v1.
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen3-8b","messages":[{"role":"user","content":"Write a haiku about GPUs."}]}'The value of -m is a local checkpoint directory, a built in alias such as qwen3-8b,
or a HuggingFace repo id in your local cache. In serve, the OpenAI model field of
every request must equal the exact string you passed to -m.
The commands run, generate, chat, and ask are aliases for the generation path.
By default NERVA applies the model's own chat template. Add --thinking to force the
reasoning branch, --raw to send the prompt with no chat template as a plain
completion, or --chat to force the chat template without thinking. Sampling is set
with --temperature, --top-p, --top-k, and --seed, and --temperature 0 --top-p 1
is greedy and deterministic.
How a model spans the GPUs depends on its class, and auto picks it for you.
| Model class | Example | Auto strategy | Serve |
|---|---|---|---|
| Dense, fits one card | Qwen3-1.7B, Qwen3-4B | single, or dp across cards | yes |
| Dense, too big for one card | Qwen3-8B, 14B, 32B | tensor parallel | yes |
| GDN hybrid | Qwen3.5, Qwen3.6-27B | tensor parallel, GDN fast path | yes |
| MoE | qwen3_next, DeepSeek-V4 | pipeline, generation only today | no, run with -m |
Tensor parallel is byte exact and is the fast path for models too big for one card. An
MoE runs the pipeline path, which is a slow single stream fallback, and serve does not
accept it yet, so those models are generation only through -m.
| Flag | Meaning |
|---|---|
| -m, --model | Checkpoint path, HuggingFace repo id, or alias. |
| -p, --prompt | Prompt text. -p @file.txt reads a file. --prompt-ids "1,2,3" feeds raw token ids. |
| -o, --output | Max new tokens, for example 256 or 16k. The default fills the model context window. |
| -c, --context | Context budget, for example 32k or 1m. The default is the model max position embeddings. |
| --temperature, --top-p, --top-k | Sampling. Set --temperature 0 --top-p 1 for greedy. |
| --seed | Deterministic sampling seed. |
| --thinking, --raw, --chat | Force the reasoning branch, send no chat template, or force the chat template. |
| --parallel | Multi GPU strategy: single, dp, pipeline, tp, or auto. --tp N sets the TP shard count. |
| --tp-replicas | Replica groups for manual serve TP. Requires nerva serve --parallel tp; otherwise replicas fill the selected GPUs. |
| --fp8, --no-fp8 | Force the native fp8 tensor core GEMM on or off. |
The default is auto, so you normally pass nothing. NERVA reads the model shape and the free memory and picks the strategy that fits and runs fastest.
| Strategy | What it does |
|---|---|
| single | One GPU. |
| dp | One full replica per GPU, requests round robined. Auto picked when the model fits one card. |
| tp | Each layer sharded across a GPU group with an NCCL all-reduce. Byte exact. The fast path for models too big for one card. |
| pipeline | Layers split into stages across GPUs. Runs any model too big to tensor shard, such as an MoE, at single stream speed. |
# Force a strategy and GPU set.
./target/release/nerva serve -m qwen3-8b --parallel tp --tp 2 --tp-replicas 2 --gpus 0,1,2,3
./target/release/nerva serve -m qwen3.6-27b --parallel tp --tp 8When a model is too big for one box, STRATA splits it pipeline-parallel across machines:
each box partial-loads a contiguous range of layers and passes activations to the next over
the Seam data plane. It is opt in — pass --mesh and nothing about the single-box path
changes. One box is the master (it discovers the fleet, plans the layer placement, and
drives the request); the others are nodes that follow the master's plan and serve their
stage. --mesh-shard i/n is this box's stage identity (0/2 = first half of a two-box
pipeline). Boxes on the same subnet auto discover; across subnets each box beacons to a
peer's control address with --seed-peers. --cluster-token is a shared secret so stray
boxes cannot join.
| Flag | Meaning |
|---|---|
--mesh |
Join / form the mesh. The gate for everything below; without it, single box. |
--mesh-role master|node |
This box's role. Default node. Exactly one master. |
--mesh-shard i/n |
This box's pipeline stage i of n total stages. Default 0/1. |
--mesh-bind addr:port |
Control-plane bind. Give the master a fixed port so nodes can reach it. Default 0.0.0.0:0 (ephemeral). |
--seed-peers a:p,b:p |
Control addresses to beacon to (needed across subnets). |
--mesh-data addr:port |
Advertised data-plane (Seam) address. Default derived from --mesh-bind. |
--cluster-token N |
Optional shared secret; boxes must match to join. |
Two boxes, one model split across them (same model path on both; each loads only its half):
# Box A — master, owns stage 0 (early layers), control plane on a fixed port.
./target/release/nerva -m qwen3.6-27b -p "hey how are you?" \
--mesh --mesh-role master --mesh-shard 0/2 \
--mesh-bind 10.0.0.1:9099 --seed-peers 10.0.0.2:9099 --cluster-token 4242
# Box B — node, owns stage 1 (later layers), beacons to the master.
./target/release/nerva -m qwen3.6-27b -p "hey how are you?" \
--mesh --mesh-role node --mesh-shard 1/2 \
--mesh-bind 10.0.0.2:9099 --seed-peers 10.0.0.1:9099 --cluster-token 4242Each box waits until the whole fleet is connected — all n boxes named by
--mesh-shard i/n — before it plans placement or starts generating, so nothing runs on a
partial view (a box logs waiting for fleet: k/n box(es) connected... until the last one
joins). The wait is bounded (~2 min); if a box never shows up the others warn and proceed on
whatever connected. Once complete, each box prints the fleet table (the mesh log) and the layer
range it owns, and the master drives the generation across the Seam. Three or more boxes use
--mesh-shard 0/3, 1/3, 2/3, and so on — one master, the rest nodes.
Multimodal checkouts work too: Qwen3.6-27B is a Qwen3_5ForConditionalGeneration VL model
that nests its decoder config (including the layer count) under text_config — the mesh reads
it there, so the 64-layer split is planned exactly as for a flat text-only config.
serve takes the same --mesh* flags — same roles, shards, and control/data addresses behind
the OpenAI HTTP API:
# Box A — master, stage 0, serves the HTTP API on :8000.
./target/release/nerva serve -m qwen3.6-27b \
--host 0.0.0.0 --port 8000 \
--mesh --mesh-role master --mesh-shard 0/2 \
--mesh-bind 10.0.0.1:9099 --seed-peers 10.0.0.2:9099 --cluster-token 4242
# Box B — node, stage 1, beacons to the master.
./target/release/nerva serve -m qwen3.6-27b \
--mesh --mesh-role node --mesh-shard 1/2 \
--mesh-bind 10.0.0.2:9099 --seed-peers 10.0.0.1:9099 --cluster-token 4242Point requests at the master (http://10.0.0.1:8000/v1/chat/completions). Note: on serve
the mesh currently comes up and plans/wires the placement, but the HTTP request loop is not yet
driven across the Seam — cross-box distributed generation is wired on the -p run path above.
The server speaks the OpenAI API. It implements /v1/chat/completions,
/v1/completions with echo and logprobs for scoring, and /v1/responses, with
streaming and per request sampling overrides. Point any OpenAI SDK at
http://127.0.0.1:8000/v1. Set the bind address, port, and auth on launch, and set the
default context with -c. The default serve context is 8192 tokens. Reasoning models
stream their thinking trace in the reasoning_content field, and the final answer is
in content.
On a single GPU the server auto selects the density path (a shared paged KV pool with
continuous batching) for a dense model at serving scale, and falls back to the per
request fork pool for the cases density does not help. Long prompt prefill runs on the
tensor cores automatically, arch selected: cuDNN fused flash on Ampere and newer, a
hand wmma kernel on Turing, with a scalar fallback. No flag is needed. A single 4k token
prompt reaches first token in about 0.5s. --kv-cache-dtype fp8 switches the KV cache to
lossy e4m3, which roughly doubles the context a given card holds and keeps the tensor
core prefill (fp8 is dequantized into the flash kernel on the fly). It is opt in because
it is lossy; auto (the default) is byte exact 16 bit KV.
# Bind address, port, bearer token auth, and default context.
./target/release/nerva serve -m qwen3-8b --host 0.0.0.0 --port 8080 --api-key secret -c 32k
MODEL=qwen3-8b # must equal the -m value
BASE=http://127.0.0.1:8000
# Chat completion.
curl $BASE/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Hi\"}],\"max_tokens\":64}"
# Text completion, greedy.
curl $BASE/v1/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"prompt\":\"2 + 2 =\",\"max_tokens\":8,\"temperature\":0}"
# Per request sampling override. The JSON body wins over the server defaults.
curl $BASE/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Surprise me\"}],\"temperature\":0.9,\"seed\":42}"
# Streaming with Server Sent Events.
curl -N $BASE/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Count to 5\"}],\"stream\":true}"A flagship is what NERVA serves the open model as. What makes a flagship is not the weights — the same open checkpoint sits under every tier — but the engine that runs it, the accuracy algorithms that check and settle its work, and the roster of experts it carries. An expert is an agent personality: a system prompt, the tools it may call, and the sampling it enforces. Experts own the operating point; a higher tier does more by carrying more, and more specialized, experts, not by retuning the same one. The raw checkpoint id is never advertised — only the flagship's Greek name.
Pick one with --flagship; the default is athena. The three tiers are:
- hestia — the foundational tier, named for the goddess of the hearth: the core expert roster every deployment starts from.
- athena — the balanced, all-round tier (the default), named for the goddess of wisdom and skilled craft. The general engineering roster.
- apollo — the full tier, named for the god of truth and precision: the complete roster, including the specialized and deep experts, for maximum accuracy.
# Serve the open model as the apollo flagship (advertised as "apollo").
./target/release/nerva serve -m qwen3-8b --flagship apollo--model-name still overrides the advertised name. Expert selection over the wire is a
reserved field, so today a turn uses the flagship's default expert (every tier ships the
general expert for now); the rosters widen as more experts land.
Continuous batching is on by default, and the auto tuner sizes the batch from your hardware. Just send concurrent requests. The knobs below are for pinning and for A/B measurement, and are rarely needed.
| Knob | Effect |
|---|---|
| --kv-cache-dtype auto|fp8 | KV cache element type. auto is byte exact 16 bit; fp8 is lossy e4m3, ~1.9x context. Also on the -m run path. |
| --serve-path auto|density|fork | Single GPU execution strategy. auto lets the tuner choose; density or fork force it. |
| --max-batch N | Cap the continuous batch width. Auto sized by default. |
| --kv-pool-fraction F | Fraction of free memory given to the KV pool. |
| --no-batch | Single stream path. Required for logprob scoring. |
| --api-key SECRET / --api-key-file PATH | Bearer token auth supplied directly or through a bounded single-line file. |
| --v2-authority-seed-file PATH | Enable signed /v2 directives using a file containing a 32-byte hex seed. Missing or malformed files fail startup. |
| --v2-budget-ceiling N | Set the /v2 per-grant ceiling. Requires an authority seed file; defaults to 1,000,000 when omitted. |
| --gpus 0,1 | Restrict replica GPU ordinals; omitted means all fitting devices. |
NERVA builds on Linux (Ubuntu x86_64 or aarch64) and Windows (x86_64, MSVC), and needs CUDA 12.x or 13.x.
cargo build --release -p nervaNative builds ask the selected nvcc for its supported GPU codes, compile SASS for every
supported architecture from SM75 through SM120, and retain PTX for the highest generic SM.
Select the toolkit with CUDA_HOME, CUDA_PATH, or CUDACXX. cuDNN is optional and enables fused
flash attention for prefill: run ./setup/linux/setup.sh, then build with NERVA_CUDNN=1.
NERVA compiles and runs natively on Windows x86_64 with the MSVC toolchain (the full
Rust runtime, the OpenAI server, and the native CUDA engine). You need the CUDA
Toolkit, Visual Studio 2022 (or the Build Tools) with the C++ workload, Rust's
x86_64-pc-windows-msvc toolchain, and LLVM (for libclang). build.rs discovers
MSVC automatically, so no Developer Command Prompt is needed.
# Install prerequisites (winget), then stage cuDNN + set env, then build.
powershell -ExecutionPolicy Bypass -File .\setup\windows\install.ps1 # optional toolchain bootstrap
powershell -ExecutionPolicy Bypass -File .\setup\windows\setup.ps1 # cuDNN + LIBCLANG_PATH + checks
cargo build --release -p nerva
.\target\release\nerva.exe -m qwen3-8b -p "Explain the CAP theorem."Multi-GPU tensor-parallel is byte-exact on Windows; NCCL is Linux-only, so TP
all-reduce uses the byte-exact host-staged fallback there. A few Linux-kernel fast
paths (abstract-namespace intra-node sockets, O_DIRECT cold-shard streaming) fall
back cleanly. See docs/WINDOWS.md for the full guide, the
setup-script details, and troubleshooting.
CPU-only development checks can explicitly use the debug/test-only stubs:
cargo check --workspace --features nerva-cuda/cuda-stubs
cargo test -p nerva-cuda --features cuda-stubs --test build_policy_contractcuda-stubs is rejected in release builds and is never enabled by workspace dependencies.
GPU correctness unit tests are not a stub-emulation suite and still require native CUDA.
NERVA is source available under the Business Source License 1.1, with the SPDX identifier BUSL-1.1. It is free for personal and non commercial use, and free to copy, modify, and use non productively. Any business or commercial use requires a commercial license. For commercial licensing, write to business@dacely.com. On the Change Date of 2030-07-10, each released version converts to the Apache License 2.0. The full terms are in LICENSE.
NERVA links against and ships alongside third party components that stay under their own licenses. Their terms and the required attributions are recorded in NOTICE, THIRD-PARTY-NOTICES.md, and the LICENSES directory.
The name NERVA, and the project logos and trademarks, are reserved and are not granted by the license. The CUDA kernels, the auto tuner, and the implementation methods in this source are original proprietary works of the Licensor, and all rights not expressly granted are reserved.
Copyright (c) 2026 BlobMaster41.