vLLM Router

A high-performance and light-weight request forwarding system for vLLM large scale deployments, providing advanced load balancing methods and prefill/decode disaggregation support.

Key Features

• Core Architecture: Request routing framework and async processing patterns
• Load Balancing: Multiple algorithms (cache-aware, power of two, consistent hashing, random, round robin)
• Prefill-Decode Disaggregation: Specialized routing for separated processing phases
• Service Discovery: Kubernetes-native worker management and health monitoring
• Enterprise Features: Circuit breakers, retry logic, metrics collection

Quick Start

Prerequisites

Rust and Cargo:

# Install rustup (Rust installer and version manager)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Follow the installation prompts, then reload your shell
source $HOME/.cargo/env

# Verify installation
rustc --version
cargo --version

Python with pip installed

Installation & Basic Usage

Rust Binary

# Build Rust components
cargo build --release

Python Package

Install from PyPI

pip install vllm-router                                                                                                                                                        ```

To build from source:
```bash    
pip install setuptools-rust wheel build
python -m build
pip install dist/*.whl

# Rebuild & reinstall in one step during development
python -m build && pip install --force-reinstall dist/*.whl

Usage Examples

Standard Data Parallelism Routing

# Launch router with data parallelism (8 replicas per worker URL)
# When data-parallel-size > 1, the router automatically creates DP-aware workers
./target/release/vllm-router \
    --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy consistent_hash \
    --intra-node-data-parallel-size 8

# Alternative: using cargo run
cargo run --release -- \
    --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy consistent_hash \
    --intra-node-data-parallel-size 8

# Alternative: using python launcher
vllm-router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy consistent_hash \
    --intra-node-data-parallel-size 8

Prefill-Decode Disaggregation

# When vLLM runs the NIXL connector, prefill/decode URLs are required.
# See a working example in scripts/llama3.1/ folder.
cargo run --release -- \
    --policy consistent_hash \
    --vllm-pd-disaggregation \
    --prefill http://127.0.0.1:8081 \
    --prefill http://127.0.0.1:8082 \
    --decode http://127.0.0.1:8083 \
    --decode http://127.0.0.1:8084 \
    --decode http://127.0.0.1:8085 \
    --decode http://127.0.0.1:8086 \
    --host 127.0.0.1 \
    --port 8090 \
    --intra-node-data-parallel-size 1 \


# When vLLM runs the NCCL connector, ZMQ based discovery is supported.
# See a working example in scripts/install.sh
cargo run --release -- \
    --policy consistent_hash \
    --vllm-pd-disaggregation \
    --vllm-discovery-address 0.0.0.0:30001 \
    --host 0.0.0.0 \
    --port 10001 \
    --prefill-policy consistent_hash \
    --decode-policy consistent_hash

# When vLLM runs the Mooncake connector, pass --kv-connector mooncake.
# The router queries each prefill node's Mooncake bootstrap server at startup
# to learn engine_id per DP rank, and injects transfer_id / remote_bootstrap_addr /
# remote_engine_id into each request's kv_transfer_params for P/D coordination.
cargo run --release -- \
    --policy consistent_hash \
    --vllm-pd-disaggregation \
    --kv-connector mooncake \
    --prefill http://127.0.0.1:8081 \
    --prefill http://127.0.0.1:8082 \
    --decode http://127.0.0.1:8083 \
    --decode http://127.0.0.1:8084 \
    --host 127.0.0.1 \
    --port 8090 \
    --intra-node-data-parallel-size 1

Configuration

Authentication

Enable bearer-token validation by listing validation URLs (comma-separated) in .env via API_KEY_VALIDATION_URLS or passing --api-key-validation-urls. When set, all HTTP endpoints require Authorization: Bearer <token> and tokens are validated with HTTP 200 responses.

# .env
API_KEY_VALIDATION_URLS=https://codebase.helmholtz.cloud/api/v4/user

# CLI override
vllm-router --api-key-validation-urls https://codebase.helmholtz.cloud/api/v4/user

Metrics

Prometheus metrics endpoint available at 127.0.0.1:29000 by default.

# Custom metrics configuration
vllm-router \
    --worker-urls http://localhost:8080 http://localhost:8081 \
    --prometheus-host 0.0.0.0 \
    --prometheus-port 9000

Retries and Circuit Breakers

Retry Configuration

Retries are enabled by default with exponential backoff and jitter:

vllm-router \
  --worker-urls http://localhost:8080 http://localhost:8081 \
  --retry-max-retries 3 \
  --retry-initial-backoff-ms 100 \
  --retry-max-backoff-ms 10000 \
  --retry-backoff-multiplier 2.0 \
  --retry-jitter-factor 0.1

Circuit Breaker Configuration

Circuit breakers protect workers and provide automatic recovery:

vllm-router \
  --worker-urls http://localhost:8080 http://localhost:8081 \
  --cb-failure-threshold 5 \
  --cb-success-threshold 2 \
  --cb-timeout-duration-secs 30 \
  --cb-window-duration-secs 60

Circuit Breaker State Machine:

• Closed → Open after N consecutive failures (failure-threshold)
• Open → HalfOpen after timeout (timeout-duration-secs)
• HalfOpen → Closed after M consecutive successes (success-threshold)

Retry Policy: Retries on HTTP status codes 408/429/500/502/503/504, with backoff/jitter between attempts.

Request ID Tracking

Track requests across distributed systems with configurable headers:

# Use custom request ID headers
vllm-router \
    --worker-urls http://localhost:8080 \
    --request-id-headers x-trace-id x-request-id

Default headers: x-request-id, x-correlation-id, x-trace-id, request-id

Load Balancing Policies

The router supports multiple load balancing policies:

Policy	Description	Session Affinity	Use Case
round_robin	Sequential distribution across workers	No	General purpose, even distribution
random	Uniform random selection	No	Simple deployments
consistent_hash	Routes same session/user to same worker	Yes	Multi-turn chat, KV cache reuse
sticky_least_loaded	Assigns new sessions to the worker with the fewest active sessions	Yes	Multi-turn workloads with explicit session lifetimes
power_of_two	Picks least loaded of two random workers	No	Load-sensitive workloads
cache_aware	Optimizes for prefix cache hits	Yes	Repeated prompts, few-shot

# Example: Using consistent_hash with HTTP header for session affinity
curl -X POST http://router:8000/v1/chat/completions \
  -H "X-Session-ID: my-session-123" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3", "messages": [{"role": "user", "content": "Hello!"}]}'

For detailed configuration options, hash key priorities, and usage examples, see Load Balancing Documentation.

Sticky least-loaded sessions

Use --policy sticky_least_loaded to balance active sessions, not concurrent requests, token counts, or GPU utilization. New sessions reserve a worker atomically; ties use rendezvous hashing. Existing sessions keep their healthy worker, even if session counts later become uneven. Unavailable workers cause reassignment on the next request. Prefill and decode pools track sessions separately.

Session IDs prefer stable headers (x-session-id, x-user-id, x-tenant-id, and x-correlation-id), then the JSON fields session_params.session_id, user, session_id, and user_id. Per-request x-request-id and x-trace-id headers remain available to consistent_hash but do not reserve sticky sessions. Requests without an explicit session identifier do not reserve a session.

Release a session after its final request completes:

curl -X POST 'http://router:8000/finish_session?session_id=my-session-123'

This endpoint uses the router's configured API-key validation. It releases the ID across default, per-model, prefill, and decode policies. Repeated or unknown IDs are harmless. It does not cancel generation; use globally unique IDs and finish only after all requests for that session have completed. Models using the shared default policy must include the model in their session ID (for example, model-a:conversation-123), or use separate per-model policy instances. Reusing one ID across disjoint model pools reassigns its single reservation between pools.

Idle sessions expire after two hours. Set VLLM_ROUTER_SLL_SESSION_EXPIRATION_IN_S on the router to override the TTL; expired entries are swept on routing requests at most once per minute. Revisited sessions are checked for expiry before refreshing their affinity. State is local to one router process and is lost on restart. Multiple routers need consistent ingress routing and session release on each router that tracked the session. The policy keeps one entry per active session and serializes assignment under a mutex, so callers should release sessions promptly and choose a TTL appropriate for their workload.

Each policy instance retains at most 100,000 sessions and accepts session IDs up to 256 bytes. Override these limits with VLLM_ROUTER_SLL_MAX_SESSIONS and VLLM_ROUTER_SLL_MAX_SESSION_ID_BYTES. Requests that introduce a session beyond either limit continue to route least-loaded without affinity; existing tracked sessions remain sticky.

By default every pinned session counts toward its worker's load until it is finished or expires. Clients that never call /finish_session therefore leave finished sessions counting as load for the whole TTL, and new sessions are placed by a stale picture. VLLM_ROUTER_SLL_ACTIVE_WINDOW_IN_S separates placement load from affinity retention. It defaults to the session expiration, preserving existing placement behavior. A session counts as load while a request is in flight or its last request started or finished within the active window. Idle sessions stop counting in the request-triggered expiration sweep (at most once per minute), but stay pinned until expiration. A returning session reactivates on its healthy worker; its cached prefix can be reused if the engine has not evicted it.

The regular router tracks request lifetimes, including streamed responses and transparent proxy requests. PD routing does not track in-flight session requests; keep the active window at its default there. In PD mode the idle clock is based on routing decisions, so a shorter window can uncount an executing request.

This improves placement of new sessions; it does not redistribute existing sessions after a worker recovers. /finish_session remains optional early cleanup.

For long agent sessions (100k+ token contexts, turns minutes to hours apart) whose clients do not finish sessions, a starting point is:

VLLM_ROUTER_SLL_SESSION_EXPIRATION_IN_S=14400  # keep affinity for 4 h of idle time
VLLM_ROUTER_SLL_ACTIVE_WINDOW_IN_S=1800        # count as load for 30 min (15 min is also reasonable)

The router exports vllm_router_sll_active_sessions{worker} (sessions counted as load) and vllm_router_sll_pinned_sessions{worker} (all pinned sessions).

Optional bounded session rebalancing

Set VLLM_ROUTER_SLL_REBALANCE_RATIO=1.5 to allow voluntary moves on the regular router. The default 0 disables moves; enabled ratios must be finite and at least 1. Selection paths without request reservations (including PD) never rebalance.

On a session's next request, it may move to the least-loaded healthy worker if it has no outstanding requests and its current worker exceeds the destination's active-session count by both the ratio (strictly) and at least four sessions. The moving session is excluded from the comparison. Selection, reactivation, and reservation happen under the same lock, so concurrent requests cannot voluntarily move a session after another request has selected its worker.

All policy instances in one router process share a limit of one voluntary move per 30 seconds, with no accumulated burst allowance. A moved session has a five-minute cooldown. These are conservative bounds, not a KV-cost estimate; a moved long-context session may require a costly cold prefill, and prefills can overlap if they last longer than 30 seconds. Independent router processes do not share the budget. Failed-worker reassignment bypasses both limits.

Moves occur only when sessions make requests: there is no background migration or guaranteed recovery time. The policy still counts sessions equally, regardless of context length or DEP group. Measure prefix-cache hits, per-rank KV pressure, latency and throughput before enabling this on large-context workloads. vllm_router_sll_rebalances_total{from,to} counts voluntary moves only.

Advanced Features

Kubernetes Service Discovery

Automatic worker discovery and management in Kubernetes environments.

Basic Service Discovery

vllm-router \
    --service-discovery \
    --selector app=vllm-worker role=inference \
    --service-discovery-namespace default

Command Line Arguments Reference

Service Discovery

• --service-discovery: Enable Kubernetes service discovery
• --service-discovery-port: Port for worker URLs (default: 8000)
• --service-discovery-namespace: Kubernetes namespace to watch
• --selector: Label selectors for regular mode (format: key1=value1 key2=value2)

Development

Troubleshooting

VSCode Rust Analyzer Issues: Set rust-analyzer.linkedProjects to the absolute path of Cargo.toml:

{
  "rust-analyzer.linkedProjects": ["/workspaces/vllm/vllm-router/Cargo.toml"]
}

CI/CD Pipeline

The continuous integration pipeline includes comprehensive testing, benchmarking, and publishing:

Build & Test

1. Build Wheels: Uses cibuildwheel for manylinux x86_64 packages
2. Build Source Distribution: Creates source distribution for pip fallback
3. Rust HTTP Server Benchmarking: Performance testing of router overhead
4. Basic Inference Testing: End-to-end validation through the router
5. PD Disaggregation Testing: Benchmark and sanity checks for prefill-decode load balancing

Releasing

To cut a new release with pre-built wheels (amd64 + arm64):

./scripts/release.sh 0.1.13

This bumps the version in pyproject.toml, commits, tags, and pushes. GitHub Actions then builds and attaches wheels to the release.

You can also trigger a wheel build manually from the Actions tab without tagging.

Acknowledgement

This project is a fork of SGLang Model Gateway, and we would like to explicitly acknowledge and thank the original authors for their work. At this stage, our fork includes only minimal changes to preserve the existing interface and ensure compatibility with vLLM. We anticipate further divergence as we pursue the roadmap we have in mind, which is the reason for creating the fork.