Upgrade vllm to 0.32.0

.github/workflows/docker.yml

@@ -59,7 +59,11 @@ jobs:
       - name: Extract backend version
         id: backend-version
         run: |
-          VERSION=$(grep -A 1 "name = \"${{ matrix.backend }}\"" uv.lock | grep version | cut -d '"' -f 2)
+          # Anchor on the package header line. Unanchored, the pattern also
+          # matches dependency entries such as `{ name = "vllm", marker =
+          # "python_full_version >= '3.11' ..." }`, whose marker contains
+          # "version", and the step writes several lines to GITHUB_OUTPUT.
+          VERSION=$(grep -A 1 "^name = \"${{ matrix.backend }}\"$" uv.lock | grep '^version = ' | head -n 1 | cut -d '"' -f 2)
           echo "version=$VERSION" >> $GITHUB_OUTPUT
 
       - name: Compute image base path

README.md

@@ -7,18 +7,20 @@
 [![code checks](https://github.com/VectorInstitute/vector-inference/actions/workflows/code_checks.yml/badge.svg)](https://github.com/VectorInstitute/vector-inference/actions/workflows/code_checks.yml)
 [![docs](https://github.com/VectorInstitute/vector-inference/actions/workflows/docs.yml/badge.svg)](https://github.com/VectorInstitute/vector-inference/actions/workflows/docs.yml)
 [![codecov](https://codecov.io/github/VectorInstitute/vector-inference/branch/main/graph/badge.svg?token=NI88QSIGAC)](https://app.codecov.io/github/VectorInstitute/vector-inference/tree/main)
-[![vLLM](https://img.shields.io/badge/vLLM-0.19.0-blue)](https://docs.vllm.ai/en/v0.19.0/)
+[![vLLM](https://img.shields.io/badge/vLLM-0.32.0-blue)](https://docs.vllm.ai/en/v0.32.0/)
 [![SGLang](https://img.shields.io/badge/SGLang-0.5.9-blue)](https://docs.sglang.io/index.html)
 ![GitHub License](https://img.shields.io/github/license/VectorInstitute/vector-inference)
 
-This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using open-source inference engines ([vLLM](https://docs.vllm.ai/en/v0.19.0/), [SGLang](https://docs.sglang.io/index.html)). **This package runs natively on the Vector Institute cluster environments**. To adapt to other environments, follow the instructions in [Installation](#installation).
+This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using open-source inference engines ([vLLM](https://docs.vllm.ai/en/v0.32.0/), [SGLang](https://docs.sglang.io/index.html)). **This package runs natively on the Vector Institute cluster environments**. To adapt to other environments, follow the instructions in [Installation](#installation).
 
 ## Installation
 If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, run the following to install package:
 
 ```bash
 pip install vec-inf
 ```
+The `vec-inf` client supports Python 3.10 and newer. The optional `vllm` extra (`pip install "vec-inf[vllm]"`) installs vLLM 0.32, which needs Python 3.11 or newer: on Python 3.10 the extra installs its other packages but not vLLM.
+
 Otherwise, we recommend using the provided [`vllm.Dockerfile`](vllm.Dockerfile) and [`sglang.Dockerfile`](sglang.Dockerfile) to set up your own environment with the package. The built images are published to **Google Artifact Registry** at:
 
 ```

docs/index.md

@@ -1,6 +1,6 @@
 # Vector Inference: Easy inference on Slurm clusters
 
-This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using open-source inference engines ([vLLM](https://docs.vllm.ai/en/v0.15.0/), [SGLang](https://docs.sglang.io/index.html)). **This package runs natively on the Vector Institute cluster environments**. To adapt to other environments, follow the instructions in [Installation](#installation).
+This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using open-source inference engines ([vLLM](https://docs.vllm.ai/en/v0.32.0/), [SGLang](https://docs.sglang.io/index.html)). **This package runs natively on the Vector Institute cluster environments**. To adapt to other environments, follow the instructions in [Installation](#installation).
 
 
 ## Installation

docs/user_guide.md

@@ -76,9 +76,11 @@ vec-inf launch Qwen2.5-3B-Instruct \
 
 For the full list of inference engine arguments, you can find them here:
 
-* [vLLM: `vllm serve` Arguments](https://docs.vllm.ai/en/stable/serving/engine_args.html)
+* [vLLM: `vllm serve` Arguments](https://docs.vllm.ai/en/v0.32.0/configuration/engine_args.html)
 * [SGLang: Server Arguments](https://docs.sglang.io/advanced_features/server_arguments.html)
 
+Short aliases are expanded to the long argument names before launch. For vLLM these are the aliases `vllm serve` itself accepts, for example `-tp`, `-pp`, `-dp`, `-ep` (`--enable-expert-parallel`), `-cc` (`--compilation-config`), `-q` (`--quantization`) and `-O3` (`--optimization-level 3`). For SGLang, `--tp`, `--pp`, `--dp` and `--ep`/`--ep-size` map to `--tensor-parallel-size`, `--pipeline-parallel-size`, `--data-parallel-size` and `--expert-parallel-size`.
+
 #### Custom models
 
 You can also launch your own custom model as long as the model architecture is supported by the underlying inference engine, and make sure to follow the instructions below:
@@ -241,23 +243,33 @@ vec-inf metrics 15373800
 And you will see the performance metrics streamed to your console, note that the metrics are updated with a 2-second interval.
 
 ```
-┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
-┃ Metric                  ┃ Value           ┃
-┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
-│ Prompt Throughput       │ 10.9 tokens/s   │
-│ Generation Throughput   │ 34.2 tokens/s   │
-│ Requests Running        │ 1 reqs          │
-│ Requests Waiting        │ 0 reqs          │
-│ Requests Swapped        │ 0 reqs          │
-│ GPU Cache Usage         │ 0.1%            │
-│ CPU Cache Usage         │ 0.0%            │
-│ Avg Request Latency     │ 2.6 s           │
-│ Total Prompt Tokens     │ 441 tokens      │
-│ Total Generation Tokens │ 1748 tokens     │
-│ Successful Requests     │ 14 reqs         │
-└─────────────────────────┴─────────────────┘
+┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
+┃ Metric                  ┃ Value         ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
+│ Prompt Throughput       │ 16.9 tokens/s │
+│ Generation Throughput   │ 50.8 tokens/s │
+│ Requests Running        │ 0 reqs        │
+│ Requests Waiting        │ 0 reqs        │
+│ KV Cache Usage          │ 0.0%          │
+│ Preemptions             │ 0             │
+│ Prefix Cache Hit Rate   │ 0.0%          │
+│ Avg Request Latency     │ 0.1 s         │
+│ Avg Queue Time          │ 0.00 s        │
+│ Total Prompt Tokens     │ 69 tokens     │
+│ Total Generation Tokens │ 208 tokens    │
+│ Successful Requests     │ 8 reqs        │
+└─────────────────────────┴───────────────┘
 ```
 
+**Notes**
+
+* The metrics come from the server's Prometheus `/metrics` endpoint. Counters are summed over all engines (for example with `--data-parallel-size`), and KV cache usage is averaged over them.
+* KV cache usage is the share of the GPU KV cache in use. vLLM 0.32 no longer reports separate GPU and CPU cache usage or swapped requests, so those rows were replaced by **KV Cache Usage** and **Preemptions**.
+* The prefix cache hit rate shows whenever the server reports it (prefix caching is on by default in vLLM and SGLang). For vLLM it is the share of prompt tokens served from the prefix cache since the server started. For SGLang it is SGLang's own `cache_hit_rate` gauge, which covers recent batches only.
+* For vLLM, **Successful Requests** counts requests that finished normally; requests aborted by the client or failed with an error are not counted. SGLang does not report how requests ended, so for SGLang servers the row is **Finished Requests** and includes aborted requests.
+* SGLang can report scheduler metrics from several tensor, pipeline and expert parallel ranks of the same data parallel group. Only the lowest reporting rank of each group is counted, so running and waiting requests are not multiplied by the number of ranks; data parallel groups are added up.
+* SGLang servers launched by `vec-inf` are started with `--enable-metrics`, which SGLang needs before it serves `/metrics`.
+
 ### `shutdown` command
 
 Finally, when you're finished using a model, you can shut it down by providing the Slurm job ID:
@@ -347,8 +359,7 @@ Once the inference server is ready, you can start sending in inference requests.
                 "content":"Arrr, I be Captain Chatbeard, the scurviest chatbot on the seven seas! Ye be wantin' to know me identity, eh? Well, matey, I be a swashbucklin' AI, here to provide ye with answers and swappin' tales, savvy?",
                 "role":"assistant",
                 "function_call":null,
-                "tool_calls":[],
-                "reasoning_content":null
+                "reasoning":null
             },
             "stop_reason":null
         }
@@ -377,6 +388,50 @@ ssh -L 8081:10.1.1.29:8081 username@v.vectorinstitute.ai -N
 ```
 The example provided above is for the Vector Killarney cluster, change the variables accordingly for your environment. The IP address for the compute nodes on Killarney follow `10.1.1.XX` pattern, where `XX` is the GPU number (`kn029` -> `29` in this example). Similarly, for Bon Echo it's `172.17.8.XX`, where `XX` is from `gpuXX`.
 
+## Notes on the vLLM 0.32 upgrade
+
+The vLLM image moved from vLLM 0.22.0 to 0.32.0. Things you may notice:
+
+* **Python, CUDA and drivers**: vLLM 0.32 needs Python 3.11 or newer, so the `vllm` extra skips vLLM on Python 3.10. The image uses torch 2.13 built for CUDA 13.0 on a CUDA 13.0.3 base image, which needs NVIDIA driver 580 or newer on the GPU nodes.
+* **Your own environment (`--venv`)**: FlashInfer compiles some kernels the first time a server starts and needs a CUDA 13.0 toolkit whose `nvcc` and headers match (set `CUDA_HOME`). Without one, startup fails with "Could not find nvcc"; pass `--env VLLM_USE_FLASHINFER_SAMPLER=0` to avoid the compile. The container image already has the toolkit.
+* **Removed model architectures** (since 0.22): Aquila, Arctic, AyaVision, Baichuan, Bamba, Chameleon, Cheers, Dots1, ERNIE (embedding and classification), Fairseq2Llama, FireRedLID, Fuyu, GritLM, Grok-1, HCXVision, the `InternLMForCausalLM` alias, JAIS (`JAISLMHeadModel`), Mantis, MiniMax-Text-01, MiniMax-M1 and MiniMax-VL-01, Mono-InternVL (`InternLM2VEForCausalLM`), MPT, MusicFlamingo, Ouro, Persimmon, Plamo2, PrithviGeoSpatialMAE, first generation Qwen and Qwen-VL (`QWenLMHeadModel`), Tarsier and Tarsier2, TeleChat, Xverse, and the `RWForCausalLM` and `StableLMEpochForCausalLM` aliases. None of the models in the default `models.yaml` use them.
+* **Plugins**: GGUF and bitsandbytes checkpoints now need out-of-tree vLLM plugins that are not in the image. GPTQ act-order (`g_idx`) is ignored.
+* **Defaults**: Model Runner V2 is used for all models, `--max-num-batched-tokens` defaults to 16384 (was 8192), and the server allows no CORS origins unless you pass `--allowed-origins`. `max_tokens` is now only an upper bound and is no longer reserved from the context window.
+* **API responses**: chat responses use `reasoning` instead of `reasoning_content`, and an empty `tool_calls` list is left out.
+* **Flags**: `-O3` means `--optimization-level 3`. The removed flags `--max-num-partial-prefills`, `--max-long-partial-prefills`, `--calculate-kv-scales` and `--override-attention-dtype` now fail to parse, and `--disable-cascade-attn` only prints a warning.
+* **`metrics` command**: see the [`metrics` command](#metrics-command) section for the new rows.
+* **Python API metric keys**: `VecInfClient.get_metrics()` returns different keys, because vLLM no longer exports the old metrics:
+
+    | Old key | New key |
+    | ------- | ------- |
+    | `gpu_cache_usage` | `kv_cache_usage` |
+    | `cpu_cache_usage` | removed, vLLM has no separate CPU cache gauge |
+    | `requests_swapped` | removed, vLLM no longer swaps; the closest signal is `preemptions_total` |
+    | `gpu_prefix_cache_hit_rate` | `prefix_cache_hit_rate` (also `prefix_cache_hits` and `prefix_cache_queries`) |
+    | `cpu_prefix_cache_hit_rate` | removed |
+    | `successful_requests_total` (SGLang) | `finished_requests_total`; vLLM keeps `successful_requests_total` |
+
+    New keys: `preemptions_total`, `queue_time_count` and `avg_queue_time`. The `enabled_prefix_caching` attribute of `PerformanceMetricsCollector` and of the CLI's `MetricsResponseFormatter` is gone.
+* **Engine arguments** (`--vllm-args`, `--sglang-args` and `*_args` in `models.yaml`):
+    * Commas inside `{}` or `[]` (and inside quoted strings within them) no longer split arguments, so JSON values such as `--vllm-args '-sc={"method":"ngram","num_speculative_tokens":3}'` reach the engine unchanged. Unbalanced braces, brackets or quotes are reported as an error.
+    * A flag and its value can be separated by `=` or a space: `-tp=2` and `-tp 2` are the same.
+    * Values are quoted in the generated Slurm script. A value starting with `{` or `[` is passed as one word. Other values are split into words, so flags that take several values work in both forms, for example `--cudagraph-capture-sizes=1 2 4 8` or `--served-model-name a b`. To pass one word that contains spaces, quote it: `--chat-template='/path/with space/t.jinja'`.
+    * A word that references a variable (`$NAME` or `${NAME}`) is still expanded by the shell, `\$` gives a literal dollar sign, and a leading `~/` still expands to your home directory. Any other `$` (for example `price$5` or `$$`) is passed literally. Quotes only group words when they start a word; apostrophes inside a word (`it's`) are kept as they are.
+* **Multi-node networking**: multi-node vLLM containers now receive the NCCL settings from the launch script's RDMA check. On nodes with RDMA tools, RoCE links get `NCCL_IB_GID_INDEX=3` and InfiniBand links `0`. To use another index, pass it at launch, for example `vec-inf launch ... --env NCCL_IB_GID_INDEX=2`; your `--env` values come after the script's defaults in the container command, so they take precedence. Without `ibv_devices` on the host the script no longer forces `NCCL_IB_DISABLE=1`; NCCL picks the transport itself and falls back to TCP sockets when there is no usable device.
+* **Ports**: the server port is now picked between 20000 and 55535, and the Ray head port between 20000 and 65535.
+
+### Supported GPUs
+
+The vLLM image uses CUDA 13.0, which needs NVIDIA driver 580 or newer and a GPU with compute capability 7.5 or higher (Turing or newer). On the Vector **Bon Echo** cluster all GPU partitions qualify: every GPU type runs driver 580.178.04 (checked October 2026), and vLLM 0.32 was tested serving models on A40 and RTX 6000 nodes.
+
+| Partitions | GPU | Compute capability |
+| ---------- | --- | ------------------ |
+| `a100_b1` to `a100_b5` | A100 80GB | 8.0 |
+| `a40_b1` to `a40_b5`, `interac` | A40 | 8.6 |
+| `rtx6000_b1` to `rtx6000_b5`, `interac` | Quadro RTX 6000 | 7.5 |
+
+Turing GPUs (RTX 6000, T4) have no bfloat16 support, so vLLM loads bfloat16 models in float16 there. Older GPUs (Volta V100 and earlier) are not supported by CUDA 13.
+
 ## Python API Usage
 
 You can also use the `vec_inf` Python API to launch and manage inference servers.

pyproject.toml

@@ -50,18 +50,22 @@ inference = [
 [project.optional-dependencies]
 # vLLM inference backend (conflicts with sglang due to dependency version conflicts)
 # Install with: uv sync --extra vllm --group inference
+# vLLM 0.32 requires Python 3.11 or newer, so the extra skips vllm on Python 3.10
+# (the vec-inf client itself still supports 3.10). vLLM 0.32 needs torchcodec
+# 0.14 or newer for video decoding.
 vllm = [
-    "vllm>=0.11.2",
+    "vllm[audio,bench]>=0.32,<0.33; python_version >= '3.11'",
     "ray[default]>=2.51.0",
-    "vllm[audio]",
-    "vllm[bench]",
-    "torchcodec>=0.9.0,<0.10.0",
+    "torchcodec>=0.14.0",
 ]
 # SGLang inference backend (conflicts with vllm due to dependency version conflicts)
 # Install with: uv sync --extra sglang --group inference
 sglang = [
+    # sglang pins its own kernel package (sglang-kernel). Do not add the older
+    # sgl-kernel distribution here: both install the sgl_kernel module, and the
+    # old one is built for a different torch, so importing it fails with an
+    # undefined c10 symbol.
     "sglang>=0.5.5",
-    "sgl_kernel>=0.3.0",
     "orjson>=3.11.0",
     "torchao>=0.9.0",
 ]

sglang.Dockerfile

@@ -68,8 +68,11 @@ RUN uv export --frozen --no-emit-project --no-hashes \
     rm -rf /root/.cache/uv /tmp/*
 
 # Build-time canary: fail the build if the locked deps cannot be imported.
-RUN python3.12 -c "import sglang, torch; \
-    print('sglang', sglang.__version__, '/ torch', torch.__version__)"
+RUN python3.12 -c "import sglang, datasets, pyarrow, transformers, torch; \
+    from google import protobuf; \
+    print('sglang', sglang.__version__, '/ datasets', datasets.__version__, \
+          '/ pyarrow', pyarrow.__version__, '/ protobuf', protobuf.__version__, \
+          '/ torch', torch.__version__, '/ torch CUDA', torch.version.cuda)"
 
 # Install a single, system NCCL (from NVIDIA CUDA repo in base image)
 RUN apt-get update && apt-get install -y --allow-change-held-packages\

tests/vec_inf/cli/test_cli.py

@@ -306,7 +306,7 @@ def test_metrics_command_server_ready(runner):
         mock_response.metrics = {
             "prompt_tokens_per_sec": 100.0,
             "generation_tokens_per_sec": 500.0,
-            "gpu_cache_usage": 0.5,
+            "kv_cache_usage": 0.5,
         }
         mock_client.get_metrics.return_value = mock_response