[amrit110]

Upgrades vllm from 0.22.0 to 0.32.0 and fixes what the new version needs, including the vec-inf metrics command, which was reading metric names vllm no longer exports. Tested end to end on Bon Echo A40 nodes (single GPU, tensor parallel 2, and two nodes with pipeline parallel 2), both from a container built the same way as vllm.Dockerfile and from a venv built from the lock.

Supersedes #328 (0.24).

Dependencies

• uv.lock relocked from main's lock with uv lock --upgrade-package vllm==0.32.0 (uv 0.13.5, lock revision 5). vllm 0.32.0 brings torch 2.13.0 (CUDA 13.0 wheels), torchvision 0.28.0, transformers 5.19.0, tokenizers 0.23.3, flashinfer-python 0.7.0.post1, xgrammar 0.2.8, triton 3.7.1, nccl 2.29.7, compressed-tensors 0.17.0.
• vllm 0.32 requires Python 3.11 or newer. The vllm extra is now vllm[audio,bench]==0.32.0; python_version >= '3.11', so the vec-inf client keeps supporting Python 3.10 and the extra simply skips vllm there.
• torchcodec moves from >=0.9,<0.10 to >=0.14 (vllm 0.32 requires it). The lock picks 0.17.0.
• The sglang extra still resolves in the same lock (sglang 0.5.12.post1, torch 2.11.0). Because the vllm fork now needs newer shared packages, uv also moved some sglang packages on Python 3.11+ (for example protobuf 7.36.1, compressed-tensors 0.17.0, huggingface-hub 1.33.0, pyarrow 26.0.0, wandb 0.31.0). I served a model with the locked sglang set on an A40 and scraped its metrics, see Testing.
• Dropped sgl_kernel>=0.3.0 from the sglang extra. sglang pins its own sglang-kernel, and the old sgl-kernel 0.3.21 installs the same sgl_kernel module built for a different torch, so importing it failed with undefined symbol: _ZN3c104cuda29c10_cuda_check_implementation.... This was already broken on main.
• The vllm extra is vllm[audio,bench]>=0.32,<0.33; python_version >= '3.11' (lock still resolves 0.32.0; only the specifier line changed in the lock).
• sglang.Dockerfile keeps its CUDA 12.8.1 base. Its build-time import check now also imports datasets, pyarrow, transformers and protobuf, like the vllm image.
• vllm.Dockerfile base moves from nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04 to nvidia/cuda:13.0.3-cudnn-devel-ubuntu24.04. torch is built for CUDA 13.0, and vllm 0.32 has FlashInfer compile its top-p/top-k sampler at startup with the system nvcc. With a mismatched toolkit the compile fails ("CUDA compiler and CUDA toolkit headers are incompatible" with nvcc 13.3 and 13.0 headers; "Could not find nvcc" with no toolkit). This matches vllm's own image (CUDA 13.0.3). CUDA 13 needs driver 580+; Bon Echo A40 and A100 nodes run 580.178.04. TORCH_CUDA_ARCH_LIST (7.5 to 9.0) is still valid for CUDA 13. sglang.Dockerfile is unchanged.
• docker.yml: the backend version lookup now anchors on ^name = "vllm"$. The new python marker made the old grep match dependency lines too and write several lines to GITHUB_OUTPUT, failing the job.

Code

• Metrics (_client_vars.py, client/_helper.py, cli/_helper.py):
  • KEY_METRICS now uses the names vllm 0.32 exports (checked against vllm/v1/metrics/loggers.py and live output): vllm:kv_cache_usage_perc, vllm:num_preemptions_total, vllm:prefix_cache_queries_total / vllm:prefix_cache_hits_total (hit rate is computed from them), plus queue time count. vllm:gpu_cache_usage_perc, vllm:cpu_cache_usage_perc, vllm:num_requests_swapped and the *_prefix_cache_hit_rate gauges no longer exist (the V0 engine metrics were already gone in 0.22).
  • Samples are now aggregated over label sets instead of the last line winning. This matters for vllm:request_success_total, which has one series per finished_reason: before, "Successful Requests" showed the last series (repetition, usually 0). It now sums stop, length and repetition, and leaves out abort and error. Data parallel engines are summed; KV cache usage is averaged.
  • SGLang metrics were never parsed and SGLang only serves /metrics with --enable-metrics. The SGLang launch templates now pass --enable-metrics, and KEY_METRICS maps the SGLang names (sglang:num_running_reqs, sglang:num_queue_reqs, sglang:token_usage, sglang:cache_hit_rate, sglang:num_retracted_requests_total, token and latency counters).
  • Removed the prefix caching check that read a top-level enable_prefix_caching key from the job JSON (engine args live under engine_args, so it was always false). Prefix caching is on by default in both engines; the row shows whenever the server reports it.
  • The metrics table rows are now: throughput, running, waiting, KV Cache Usage, Preemptions, Prefix Cache Hit Rate, Avg Request Latency, Avg Queue Time, token totals, Successful Requests. The parser also handles quoted label values with spaces and escaped quotes, and exposition timestamps.
• Engine argument aliases:
  • SGLang --ep-size mapped to a non-existent --expert-parallel-expert-size; it now maps to --expert-parallel-size (SGLang accepts --expert-parallel-size, --ep-size and --ep).
  • vllm -O mapped to --compilation-config (-O3 became --compilation-config 3, which vllm rejects). vllm itself expands -O3 to --optimization-level 3, so vec-inf now does the same and accepts -O3, -O=3 and -O 3.
  • Added the other vllm 0.32 short aliases: -ep, -sc, -cc, -ac, -dpm, -asc. Boolean short flags like -ep are now expanded too, and only the first = splits flag and value.
• Engine argument values:
  • --vllm-args/--sglang-args split only on commas outside {} and [] (quotes are tracked only inside them), so -sc={"method":"ngram","num_speculative_tokens":3} stays one argument. Behaviour change: unbalanced {, [, } or ] (or an unclosed quote inside them) in engine args now raise a ValueError at launch instead of silently swallowing the following arguments. Quotes outside braces are ordinary characters, so --x=it's,-tp=2 gives two arguments. Flag and value can be separated by = or a space. Added -dc (--diffusion-config).
  • Values are quoted in the generated scripts. A value starting with { or [ is one single-quoted word, so JSON reaches the engine intact (before, bash stripped the inner quotes). Other values are split into words on whitespace (a leading quote groups words, quotes inside a word such as it's stay literal) and each word is quoted on its own, so flags that take several values (--cudagraph-capture-sizes 1 2 4 8, --served-model-name a b, --device-ids, --ignore-patterns, ...) keep working in both the space and = forms. One word with spaces must be quoted (--chat-template='/p q/t.jinja').
  • Nothing in models.yaml, Bon Echo's cached configs or the templates relies on shell expansion inside a value. A word with a real variable reference ($NAME, ${...}, not $$) or an escaped \$ is double-quoted with every other backslash escaped, so --download-dir=$SLURM_TMPDIR/x still expands, a\$b gives a$b and $X\"q stays one word; a leading ~/ stays unquoted; any other $ (price$5, $$, JSON) stays literal.
• SGLang metrics: scheduler gauges and counters can be reported by several TP/PP/expert ranks of one data parallel group (with PP, or --enable-metrics-for-all-schedulers). For each metric and dp_rank only the lowest reporting rank is counted, then DP groups are summed; with --enable-dp-attention a group's reporting rank can have tp_rank != 0, which this handles. sglang:num_requests_total is reported as finished_requests_total ("Finished Requests" in the CLI) because it includes aborted requests.
• Multi-node containers (_slurm_templates.py, _slurm_script_generator.py): multi-node vllm runs Ray and vllm serve in separate apptainer instances. Ray 2.56 (same version as on main) finds the local raylet by scanning the process table, so with --containall (private PID namespace) vllm serve failed with "No node info found matching attributes". Multi-node vllm containers now use --contain --cleanenv; single node and SGLang keep --containall.
  • $NCCL_ENV_ARG (set by the RDMA check) is now passed to these containers, so NCCL_IB_* reaches NCCL under --cleanenv. This is a behaviour change: on RoCE links NCCL now actually gets NCCL_IB_GID_INDEX=3 (InfiniBand 0). $NCCL_ENV_ARG comes before the user's --env values, so vec-inf launch ... --env NCCL_IB_GID_INDEX=2 overrides the default (apptainer keeps the last value for a repeated key). When the host has no ibv_devices, the script no longer forces NCCL_IB_DISABLE=1; NCCL picks the transport itself. (I could not run the per-partition ibv_devices check, see Not verified, so I took the option that is safe either way.) The template also dropped the $ in export NCCL_IB_GID_INDEX={NCCL_IB_GID_INDEX:-3}, so the value was literal text; it is now a real parameter expansion. Verified on Bon Echo: NCCL logs NCCL_IB_GID_INDEX set by environment to 3 and uses RoCE between nodes.
  • The Ray head port is now picked from 20000-65535, and the server port from 20000-55535, both above Ray's worker range. A port in Ray's worker range (10002-19999) made ray start on the workers fail ("worker_ports is trying to use a port number ..."), which happened once in testing.
• Server port is now picked from 20000-55535 for both engines. SGLang 0.5.12 binds a gRPC port at the server port + 10000 and exits when that is above 65535, so roughly one SGLang launch in seven failed. Already true on main.
• Status check (client/_utils.py): warning and info log lines are no longer treated as a failed server. Ray logs [... W ...] rpc_client.h: Failed to connect to the observability pubsub during multi-node startup, which made vec-inf status report FAILED for a server that came up fine. Accepted trade-off from review: a genuinely fatal message logged at WARNING level is no longer caught by the status check (it still shows once the process exits).

Config

• models.yaml: added --trust-remote-code for Phi-3-vision-128k-instruct, Phi-3.5-vision-instruct and Molmo-7B-D-0924. vllm rejects their configs without it ("contains custom code which must be executed"); this is the same on 0.22.
• No model in models.yaml, and none of the 113 models in Bon Echo's cached configs, uses an architecture removed between 0.22 and 0.32.

Docs

• README and docs index link and badge now point to vllm 0.32.0 (they said 0.19.0 and 0.15.0), and the engine args link (404) is fixed.
• User guide: supported GPUs and drivers on Bon Echo, the Python API metric key changes (below), SGLang metric semantics (recent cache hit rate, finished requests, per-rank counting), quoting of engine arguments, new metrics table from a live 0.32 server with notes on what each row means, the alias list, the chat response example (reasoning instead of reasoning_content), and a "Notes on the vLLM 0.32 upgrade" section.

Python API: metric key changes

VecInfClient.get_metrics() returns different keys:

Old key	New key
gpu_cache_usage	kv_cache_usage
cpu_cache_usage	removed (vLLM has no separate CPU cache gauge)
requests_swapped	removed (vLLM no longer swaps; closest signal is preemptions_total)
gpu_prefix_cache_hit_rate	prefix_cache_hit_rate (plus prefix_cache_hits, prefix_cache_queries)
cpu_prefix_cache_hit_rate	removed
successful_requests_total for SGLang	finished_requests_total (vLLM keeps successful_requests_total)

New keys: preemptions_total, queue_time_count, avg_queue_time. The enabled_prefix_caching attribute of PerformanceMetricsCollector and MetricsResponseFormatter is gone.

Supported GPUs

CUDA 13 needs driver 580+ and compute capability 7.5+. On Bon Echo (sinfo): A100 (a100_b1-b5), A40 (a40_b1-b5, interac) and Quadro RTX 6000 (rtx6000_b1-b5, interac) all run driver 580.178.04. vllm 0.32 served models on A40 and RTX 6000 (Turing falls back from bfloat16 to float16 and uses Triton attention). A few drained gpu:nvidia nodes in the a40 partitions could not be checked.

Behaviour changes in vllm 0.22 to 0.32 that cluster users may notice

• Python 3.11+ required (0.32). Image is torch 2.13 / CUDA 13.0; driver 580+ required.
• Removed architectures: Aquila, Arctic, AyaVision, Baichuan, Bamba, Chameleon, Cheers, Dots1, ERNIE (embedding and classification), Fairseq2Llama, FireRedLID, Fuyu, GritLM, Grok-1, HCXVision, InternLMForCausalLM alias, JAIS, Mantis, MiniMax-Text-01/M1/VL-01, Mono-InternVL, MPT, MusicFlamingo, Ouro, Persimmon, Plamo2, PrithviGeoSpatialMAE, first generation Qwen and Qwen-VL, Tarsier/Tarsier2, TeleChat, Xverse, and the RWForCausalLM and StableLMEpochForCausalLM aliases (checked with the real 0.22 and 0.32 model registries).
• GGUF and bitsandbytes need out-of-tree plugins that are not in the image. GPTQ act-order (g_idx) is ignored (0.30).
• Model Runner V2 is the default for all models (0.29). max_num_batched_tokens default 16384, was 8192 (0.28). Shared memory tensor arena on by default (0.32), and vllm fails fast if /dev/shm is too small (0.27).
• No CORS origins allowed by default; use --allowed-origins (0.32). max_tokens is no longer reserved from the context window (0.32).
• Chat responses use reasoning, not reasoning_content (0.22/0.28); an empty tool_calls list is omitted (0.24).
• vllm no longer sets CUDA_VISIBLE_DEVICES and adds --device-ids (0.24). Slurm's allocation is still respected (verified with TP=2 and two nodes).
• Removed flags now fail to parse: --max-num-partial-prefills, --max-long-partial-prefills (0.27), --calculate-kv-scales, --override-attention-dtype (0.28). --disable-cascade-attn still parses but only warns (0.32). --mamba-cache-mode all is rejected (0.32).
• python -m vllm.entrypoints.openai.api_server is deprecated in favour of vllm serve (0.29); vec-inf already uses vllm serve.
• Prometheus: no core metric was renamed or removed between 0.22 and 0.32 (checked in source); new ones include vllm:request_num_preemptions, vllm:prompt_tokens_cached_by_source_total, vllm:admission_rejections_total.
• No vllm serve flag that vec-inf generates or documents was removed; the full option diff (parser dump of both versions) shows only the four removals above.

Testing

• Unit tests: 292 pass on Python 3.10, 3.11 and 3.12, both with CI's exact commands (uv 0.5.21, uv sync --dev --prerelease=allow, then uv sync --no-dev and the import test) and against the locked set (uv sync --frozen --dev with uv 0.13.5). New tests cover JSON-valued, multi-value (--cudagraph-capture-sizes, --served-model-name, space and = forms) and space-separated engine args (generated script checked with bash -n and the words echoed back through bash), the quoting rules for $, \$, ~/ and price$5, unbalanced splitter input, SGLang DP-attention rank labels, shell expansion for $ and ~/ values, NCCL env and GID expansion in multi-node scripts, port ranges, SGLang per-rank aggregation, metric parsing (a real /metrics capture from vllm 0.32.0 serving Llama-3.2-1B-Instruct on two A40 nodes, a real SGLang 0.5.12.post1 capture, data parallel aggregation, finish reasons, label escaping), the alias fixes, the Ray executor and container flags for multi-node, --enable-metrics for SGLang, and bash -n on generated scripts.
• pre-commit (ruff, ruff format, mypy, pytest) passes.
• Generated commands vs vllm's real parser: for every vllm model in models.yaml, as configured and as a forced two-node variant (192 command lines), the vllm serve command from SlurmScriptGenerator was parsed by vllm 0.32.0's own vllm serve parser and turned into AsyncEngineArgs.from_cli_args. All 192 pass (Linux arm64 container, locked deps, no GPU, CUDA platform stubbed). On a Bon Echo A40 node, create_engine_config() was also run against each model's real config.json, for both vllm 0.32.0 and 0.22.0: the results are identical between versions (154 pass). The remaining cases are only the expected world-size check on a 1-GPU validation node, PP on embedding models in the synthetic two-node variant, the three remote-code VLMs (fixed in this PR, rechecked: pass), and Phi-3-medium-128k-instruct, whose config fails the same way on 0.22 (rope_parameters missing original_max_position_embeddings under transformers 5).
• GPU end to end on Bon Echo (A40, driver 580.178.04), with vec-inf launch, status, metrics and shutdown from this branch:
  • Container: an apptainer image built from the same base image and the same locked requirements as vllm.Dockerfile (the pr-330 image is in a private registry I cannot read). Qwen2.5-0.5B-Instruct on 1 GPU, Qwen3-0.6B with --tensor-parallel-size=2, and Llama-3.2-1B-Instruct on 2 nodes with -pp=2 all reached READY, answered /v1/chat/completions and /v1/completions, and vec-inf metrics showed correct values (for example 48 running requests and a 95% prefix cache hit rate under a 48-request burst, and successful requests matching the number sent).
  • Venv: the same three cases from a venv built from the lock (needs VLLM_USE_FLASHINFER_SAMPLER=0 or a CUDA 13.0 toolkit, see the upgrade notes).
  • SGLang image: built the sglang.Dockerfile equivalent (CUDA 12.8.1 base, locked sglang set, new canary passes: sglang 0.5.12.post1, torch 2.11.0+cu130). vec-inf launch Qwen2.5-0.5B-Instruct --engine sglang on an A40 reached READY with --enable-metrics, chat and completions worked, and vec-inf metrics showed the new rows (Finished Requests, recent prefix cache hit rate). The first launch hit the gRPC port bug above (fixed). In one run the first chat request after a cold start returned an error response while the server stayed up; after a restart with warm kernel caches every request succeeded. I could not reproduce it.
  • JSON engine args end to end: --vllm-args '-sc={"method":"ngram","num_speculative_tokens":3,"prompt_lookup_max":4},-cc={"cudagraph_capture_sizes":[1,2,4,8]},-tp 1' on an RTX 6000. vllm logged the parsed speculative and compilation configs and reported ngram drafts in /metrics.
  • Two-node PP=2 rerun on RTX 6000 nodes after the NCCL changes: READY, chat and metrics fine, NCCL used RoCE with NCCL_IB_GID_INDEX=3.
  • RTX 6000 single GPU: READY, chat and metrics fine.
• Docker: both images build in this PR and pass the import canary (vllm image prints torch CUDA 13.0).

Not verified

• The last round of changes (multi-value quoting, SGLang DP-attention ranks, the ibv_devices fallback, the 20000 port floor) is covered by unit tests only. Non-interactive SSH to Bon Echo stopped working (it now asks for keyboard-interactive auth), so I could not rerun GPU jobs or check ibv_devices on the a40, a100 and rtx6000 nodes. Earlier in this PR, the 2-node run on rtx6000 nodes found mlx5_0 RoCE via NCCL, and the login node has ibv_devices.

• The actual pr-330 image on a GPU (private registry). The tested image uses the same base, locked requirements and install steps, except Python 3.12 comes from Ubuntu's package instead of a source build.

• A100 serving (only the driver version was checked), and the drained gpu:nvidia nodes.

• Large models, quantized models and VLM serving; those were checked through vllm's config validation only.

• The actual pr-330 SGLang image (private registry); the tested one is the equivalent apptainer build. SGLang multi-node was not run.

To test with the PR image

# Convert the PR image (needs Artifact Registry access)
apptainer build vector-inference-vllm_pr-330.sif docker://<REGION>-docker.pkg.dev/<PROJECT_ID>/<REPOSITORY>/vector-inference-vllm:pr-330
# Point vec-inf at it: copy environment.yaml into a config dir and set vllm_image_path
export VEC_INF_CONFIG_DIR=/path/to/config
vec-inf launch Qwen2.5-0.5B-Instruct --partition a40_b1 --resource-type a40 --time 00:45:00
vec-inf launch Qwen3-0.6B --partition a40_b1 --resource-type a40 --gpus-per-node 2 --vllm-args '--tensor-parallel-size=2' --time 00:45:00
vec-inf launch Llama-3.2-1B-Instruct --partition a40_b1 --resource-type a40 --num-nodes 2 --gpus-per-node 1 --vllm-args '-pp=2' --time 00:45:00
vec-inf status <job_id>; vec-inf metrics <job_id>; vec-inf shutdown <job_id>

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.61078% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.64%. Comparing base (fb32b73) to head (e05225a).

Files with missing lines	Patch %	Lines
vec_inf/client/_helper.py	95.00%	5 Missing ⚠️
vec_inf/client/_slurm_script_generator.py	92.45%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #330      +/-   ##
==========================================
+ Coverage   89.98%   90.64%   +0.66%     
==========================================
  Files          14       14              
  Lines        1468     1593     +125     
==========================================
+ Hits         1321     1444     +123     
- Misses        147      149       +2     

Files with missing lines	Coverage Δ
vec_inf/cli/_helper.py	95.45% <100.00%> (-0.03%)	⬇️
vec_inf/client/_client_vars.py	100.00% <100.00%> (ø)
vec_inf/client/_slurm_templates.py	100.00% <100.00%> (ø)
vec_inf/client/_utils.py	72.63% <100.00%> (+0.58%)	⬆️
vec_inf/client/_slurm_script_generator.py	96.61% <92.45%> (+0.44%)	⬆️
vec_inf/client/_helper.py	93.76% <95.00%> (+0.77%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.