From a9a34d145d542f7c7a019e99f481b7b326a0ed5e Mon Sep 17 00:00:00 2001 From: Itay Zitvar Date: Thu, 3 Sep 2026 10:57:44 -0700 Subject: [PATCH 1/2] docs(readme): restructure around three integration paths Signed-off-by: Itay Zitvar --- README.md | 193 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 135 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index c6537b7e3..0e2c01512 100644 --- a/README.md +++ b/README.md @@ -8,35 +8,30 @@ **[Get started →](#get-started)** -```mermaid -flowchart LR - A["Claude Code · Codex CLI
OpenAI / Anthropic SDK clients"] - SY["Switchyard
routing algorithm + protocol translation"] - E["Efficient model
GLM, Qwen, your own vLLM"] - C["Capable model
Opus, GPT, NVIDIA NIM"] - - A -->|"unchanged native API"| SY - SY -->|"routine turns"| E - SY -->|"hard turns"| C -``` - -It has three modes: +![Accuracy versus total cost on Terminal-Bench 2.1. Switchyard's staged, escalation, and classifier routes reach 71-76% accuracy for 13-30% less than the Opus 4.8 baseline, while single fixed models stay below 56%.](assets/benchmark-accuracy-vs-cost.svg) -**1. A Rust proxy** — run it in front of the agent you already use: -```bash -cargo install --locked switchyard-server -switchyard-server --config routes.toml --port 4000 -``` +## What is Switchyard -**2. An embeddable Rust library** — call the same algorithms from a gateway you already own: +**1. An embeddable Rust library** `switchyard-libsy` picks the model for you. Drops into whatever gateway or agent runtime you already have. ```bash cargo add --git https://github.com/NVIDIA-NeMo/Switchyard.git --tag v0.2.0 \ switchyard-libsy switchyard-protocol ``` -**3. A NeMo Relay plugin** — load the same `routes.toml` into a NeMo Relay deployment you already run: +```mermaid +flowchart LR + subgraph R["Your LLM gateway / harness"] + P["Switchyard"] + end + P--> M["Efficient model"] + P--> N["Capable model"] + P--> O[etc.] + G[You] -->|"request"| P +``` + +**2. A NeMo Relay plugin** — load a `routes.toml` file into a NeMo Relay deployment you already run: ```toml [[plugins.dynamic]] @@ -46,9 +41,38 @@ manifest = "./plugins/switchyard/relay-plugin.toml" switchyard_config_path = "/etc/switchyard/routes.toml" ``` -Relay runs any routing algorithm `switchyard-runner` supports while Switchyard owns provider HTTP dispatch. See [`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md). +```mermaid +flowchart LR + subgraph R["NeMo Relay"] + P["Switchyard plugin"] + end + P--> M["Efficient model"] + P--> N["Capable model"] + P--> O[etc.] + G[You] -->|"request"| P +``` + + +**3. A standalone Rust proxy** — when you want a server in front of an agent +rather than code in your own stack: + +```bash +cargo install --locked switchyard-server +switchyard-server --config routes.toml --port 4000 +``` + +Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at the proxy. +Every request keeps its native API format; Switchyard decides per turn which +model serves it. -Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at it. Every request keeps its native API format; Switchyard decides per turn which model serves it. +```mermaid +flowchart LR + P["Switchyard
standalone proxy"] + P--> M["Efficient model"] + P--> N["Capable model"] + P--> O[etc.] + G[You] -->|"unchanged native API"| P +``` ## Maturity @@ -62,27 +86,83 @@ Switchyard is pre-alpha software that is evolving rapidly. The API and algorithm > - switchyard-runner: Alpha. Evolving rapidly. > - switchyard-server: Demo server, not for production use. -## Why Switchyard +## Get Started -Every Switchyard route below pairs the same Opus 4.8 capable tier with a cheaper -efficient tier. The baseline is Opus 4.8 serving every turn. +Three paths, one per mode above. Each is self-contained: start at step 1, stop +when you reach the result named under the heading. -![Accuracy versus total cost on Terminal-Bench 2.1. Switchyard's staged, escalation, and classifier routes reach 71-76% accuracy for 13-30% less than the Opus 4.8 baseline, while single fixed models stay below 56%.](assets/benchmark-accuracy-vs-cost.svg) +### Path 1 — Embed the Library -| Configuration | Accuracy | Total cost | vs. Opus 4.8 baseline | -|---|---:|---:|---| -| Opus 4.8 baseline | 76.0% | $98.06 | — | -| **[Escalation](#routing-algorithms)** | 75.7% | $85.00 | 99.6% of accuracy, 13.3% cheaper | -| **[Stage](#routing-algorithms)** | 72.7% | $68.19 | 95.7% of accuracy, 30.5% cheaper | -| **[Capability](#routing-algorithms)** | 71.2% | $79.32 | 93.7% of accuracy, 19.1% cheaper | -| Kimi K2.6 alone | 55.8% | $76.28 | | -| GLM 5.2 alone | 52.4% | $16.47 | | -| DeepSeek V4 Pro alone | 48.7% | $96.92 | | -| Ultra 3 alone | 39.0% | $29.66 | | +**Recommended.** You finish with your own service picking a model per request +and still making every model call itself. -## Get Started +**1. Add the crates to your service's `Cargo.toml`.** + +```toml +[dependencies] +async-trait = "0.1" +futures = "0.3" +switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } +switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } +tokio = { version = "1", features = ["macros", "rt"] } +``` -Install [Rust with Cargo](https://rust-lang.org/tools/install/), then: +**2. Construct an algorithm.** `StageRouter`, `LlmTaskClassifier`, `Random`, or +`Passthrough` — the same set the proxy exposes as route types. See +[Routing Algorithms](#routing-algorithms) for which to pick. + +**3. Drive its step stream.** `Algorithm::run_stream` yields `Step` items. Each +`Step::CallModel` is a classifier or judge call — you serve it over your own +transport. The stream ends with `Step::Done` carrying a `RoutingOutcome`: the +selected model, ordered fallbacks, the rewritten request, and a response if +routing already produced one. + +**4. Make the answer call** from that outcome, with your own HTTP client, +retries, and credentials. + +**Shortcut for step 4:** add `switchyard-llm-client` and call its `run`. It +drives the stream, makes the answer call, and handles retries and fallback over +HTTP. + +Type reference: [`switchyard-libsy`](crates/libsy/README.md) and +[`switchyard-protocol`](crates/protocol/README.md). + +### Path 2 — Load the NeMo Relay Plugin + +You finish with an existing NeMo Relay deployment routing through Switchyard. +Requires NeMo Relay `>=0.8.1,<0.9.0`. + +**1. Build the plugin bundle.** + +```bash +python crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py +``` + +**2. Write the Switchyard deployment** to `/etc/switchyard/routes.toml` — the +same version-1 TOML the proxy uses. Copy the file from step 2 of Path 3 below. + +**3. Point Relay at the generated manifest.** Use exactly one deployment +source: a path, as here, or the config nested under `switchyard_config`. + +```toml +[[plugins.dynamic]] +manifest = "./plugins/switchyard/relay-plugin.toml" + +[plugins.dynamic.config] +priority = 0 +switchyard_config_path = "/etc/switchyard/routes.toml" +``` + +**4. Restart Relay.** It now runs any algorithm `switchyard-runner` supports, +while Switchyard owns provider HTTP dispatch. + +Details: [`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md) +and the [server configuration guide](crates/switchyard-server/CONFIGURATION.md). + +### Path 3 — Run the Standalone Proxy + +You finish with a server on `localhost:4000` that any OpenAI or Anthropic client +can call. Needs [Rust with Cargo](https://rust-lang.org/tools/install/). **1. Install the server.** @@ -121,7 +201,10 @@ confidence_threshold = 0.5 TOML ``` -**3. Run it.** `--dry-run` loads the config, prints `server OK:` and the model +Every key is documented in the +[server configuration guide](crates/switchyard-server/CONFIGURATION.md). + +**3. Start it.** `--dry-run` loads the config, prints `server OK:` and the model IDs it exposes, then exits without starting the server. ```bash @@ -143,7 +226,7 @@ The same route also answers on `/v1/messages` (Anthropic Messages) and what, and `/metrics` exposes Prometheus counters for requests, errors, latency, tokens, and routing overhead. -### Point a Coding Agent at It +**5. Point a coding agent at it.** ```bash export ANTHROPIC_BASE_URL="http://localhost:4000" @@ -157,22 +240,6 @@ Codex CLI and other OpenAI clients use the OpenAI variables instead: export OPENAI_BASE_URL="http://localhost:4000/v1" ``` -### Embed It in Your Own Gateway - -`switchyard-libsy` never calls a model: an algorithm returns the target it chose -and hands the call back to you, so it drops into a proxy, gateway, or agent -runtime you already run. Pair it with `switchyard-llm-client` to have the calls -made for you. - -```toml -[dependencies] -switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } -switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } -``` - -See [Getting Started](docs/getting_started.md#library-path) for setup, or the -[`switchyard-libsy`](crates/libsy/README.md) crate docs. - ## Routing Algorithms Most use an LLM as a judge. All of them pick between an **efficient** model and a @@ -196,7 +263,6 @@ the common route shape and self-hosted targets. ## Documentation -- **[Getting Started](docs/getting_started.md)**: complete standalone server walkthrough - **[Core Concepts](docs/core_concepts.md)**: LLM clients, targets, routes, model IDs, and routing algorithms - **[Routing Overview](docs/routing_algorithms/overview.md)**: choose and configure a routing algorithm - **[TOML Schema](docs/reference/toml_schema.md)**: every configuration key @@ -209,7 +275,18 @@ the common route shape and self-hosted targets. ## Benchmark Provenance -The numbers in [Why Switchyard](#why-switchyard) are the v0.2.0 Terminal-Bench 2.1 +| Configuration | Accuracy | Total cost | vs. Opus 4.8 baseline | +|---|---:|---:|---| +| Opus 4.8 baseline | 76.0% | $98.06 | — | +| **[Escalation](#routing-algorithms)** | 75.7% | $85.00 | 99.6% of accuracy, 13.3% cheaper | +| **[Stage](#routing-algorithms)** | 72.7% | $68.19 | 95.7% of accuracy, 30.5% cheaper | +| **[Capability](#routing-algorithms)** | 71.2% | $79.32 | 93.7% of accuracy, 19.1% cheaper | +| Kimi K2.6 alone | 55.8% | $76.28 | | +| GLM 5.2 alone | 52.4% | $16.47 | | +| DeepSeek V4 Pro alone | 48.7% | $96.92 | | +| Ultra 3 alone | 39.0% | $29.66 | | + +These are the v0.2.0 Terminal-Bench 2.1 results from [Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/). Those runs used NVIDIA-internal inference endpoints, so absolute solve rates may shift on another serving stack; the routing parameters are the ones that ran. From 93250840e71e2d704e7f1338bf48dfffc0f40ca0 Mon Sep 17 00:00:00 2001 From: Itay Zitvar Date: Thu, 3 Sep 2026 14:12:05 -0700 Subject: [PATCH 2/2] docs(readme): use-case framing, components table, Python embed path Signed-off-by: Itay Zitvar --- README.md | 181 ++++++++++++++++---------- assets/benchmark-accuracy-vs-cost.svg | 6 +- 2 files changed, 116 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index 0e2c01512..efd341ee7 100644 --- a/README.md +++ b/README.md @@ -10,51 +10,61 @@ ![Accuracy versus total cost on Terminal-Bench 2.1. Switchyard's staged, escalation, and classifier routes reach 71-76% accuracy for 13-30% less than the Opus 4.8 baseline, while single fixed models stay below 56%.](assets/benchmark-accuracy-vs-cost.svg) +_\*Total cost based on average ISP token cost_ ## What is Switchyard -**1. An embeddable Rust library** `switchyard-libsy` picks the model for you. Drops into whatever gateway or agent runtime you already have. +Switchyard picks which model serves each LLM call. -```bash -cargo add --git https://github.com/NVIDIA-NeMo/Switchyard.git --tag v0.2.0 \ - switchyard-libsy switchyard-protocol -``` +### Use Switchyard + +Switchyard runs inside gateways you may already have. + +- **NeMo Relay** — a native plugin. Load a `routes.toml` into a Relay deployment + you already run. [Setup →](#path-1--load-the-nemo-relay-plugin) +- **LiteLLM** — a routing plugin for LiteLLM's `Router` and proxy. + [`examples/litellm`](examples/litellm/README.md) +- **More integrations** coming soon. ```mermaid flowchart LR - subgraph R["Your LLM gateway / harness"] + subgraph R["LiteLLM · NeMo Relay"] P["Switchyard"] end P--> M["Efficient model"] P--> N["Capable model"] P--> O[etc.] G[You] -->|"request"| P + style P fill:#76B900,stroke:#5A8F00,color:#000 ``` -**2. A NeMo Relay plugin** — load a `routes.toml` file into a NeMo Relay deployment you already run: +### Integrate Switchyard into your gateway or harness -```toml -[[plugins.dynamic]] -manifest = "./plugins/switchyard/relay-plugin.toml" +Embed the routing algorithms in your own. Switchyard picks the model; your +harness makes the call, so your transport, retries, and credentials stay +untouched. -[plugins.dynamic.config] -switchyard_config_path = "/etc/switchyard/routes.toml" -``` +- Install: `pip install nemo-switchyard` +- Then follow [Path 2 — Embed the Library](#path-2--embed-the-library): + construct an algorithm, drive its step stream, make the answer call. +- Also available for Rust as `switchyard-libsy`; Path 2 has the `Cargo.toml` + block. ```mermaid flowchart LR - subgraph R["NeMo Relay"] - P["Switchyard plugin"] + subgraph R["Your LLM gateway / harness"] + P["Switchyard"] end P--> M["Efficient model"] P--> N["Capable model"] P--> O[etc.] - G[You] -->|"request"| P + G["Your users"] -->|"request"| P + style P fill:#76B900,stroke:#5A8F00,color:#000 ``` +### Run Switchyard as a standalone proxy -**3. A standalone Rust proxy** — when you want a server in front of an agent -rather than code in your own stack: +A server in front of an agent, when you have no gateway to put Switchyard in: ```bash cargo install --locked switchyard-server @@ -62,8 +72,7 @@ switchyard-server --config routes.toml --port 4000 ``` Point Claude Code, Codex CLI, or any OpenAI/Anthropic SDK client at the proxy. -Every request keeps its native API format; Switchyard decides per turn which -model serves it. +Switchyard decides per turn which model serves it. ```mermaid flowchart LR @@ -72,62 +81,27 @@ flowchart LR P--> N["Capable model"] P--> O[etc.] G[You] -->|"unchanged native API"| P + style P fill:#76B900,stroke:#5A8F00,color:#000 ``` -## Maturity +## Components -Switchyard is pre-alpha software that is evolving rapidly. The API and algorithms are expected to change significantly before we reach v1.0. +Pre-1.0 software. APIs, configuration, and routing behavior can change between +releases — pin the version you integrate. -> [!WARNING] -> Switchyard is a very young project showcasing active research. Component maturity levels: -> -> - libsy: Beta. Ready for trial integration. -> - switchyard-llm-client: Alpha. May change significantly. -> - switchyard-runner: Alpha. Evolving rapidly. -> - switchyard-server: Demo server, not for production use. +| Component | Stability | Use it for | Guidance | +|---|---|---|---| +| `switchyard-libsy` | **Beta** | Routing embedded in your own gateway or harness. You own model calls, credentials, and retries. | Trial integrations. API will change before v1.0. | +| `switchyard-llm-client` | **Alpha** | HTTP model calls and protocol translation alongside libsy. | Experiments and pilots. | +| `switchyard-runner` | **Alpha** | Running configured routes inside another runtime, such as NeMo Relay. | Integration work and supervised pilots. | +| `switchyard-server` | **Demo** | A standalone OpenAI- and Anthropic-compatible proxy. | Demos and evaluation only. Not for production. | ## Get Started -Three paths, one per mode above. Each is self-contained: start at step 1, stop -when you reach the result named under the heading. - -### Path 1 — Embed the Library - -**Recommended.** You finish with your own service picking a model per request -and still making every model call itself. - -**1. Add the crates to your service's `Cargo.toml`.** - -```toml -[dependencies] -async-trait = "0.1" -futures = "0.3" -switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } -switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } -tokio = { version = "1", features = ["macros", "rt"] } -``` - -**2. Construct an algorithm.** `StageRouter`, `LlmTaskClassifier`, `Random`, or -`Passthrough` — the same set the proxy exposes as route types. See -[Routing Algorithms](#routing-algorithms) for which to pick. - -**3. Drive its step stream.** `Algorithm::run_stream` yields `Step` items. Each -`Step::CallModel` is a classifier or judge call — you serve it over your own -transport. The stream ends with `Step::Done` carrying a `RoutingOutcome`: the -selected model, ordered fallbacks, the rewritten request, and a response if -routing already produced one. - -**4. Make the answer call** from that outcome, with your own HTTP client, -retries, and credentials. +Three paths, in the same order as above. Each is self-contained: start at +step 1, stop when you reach the result named under the heading. -**Shortcut for step 4:** add `switchyard-llm-client` and call its `run`. It -drives the stream, makes the answer call, and handles retries and fallback over -HTTP. - -Type reference: [`switchyard-libsy`](crates/libsy/README.md) and -[`switchyard-protocol`](crates/protocol/README.md). - -### Path 2 — Load the NeMo Relay Plugin +### Path 1 — Load the NeMo Relay Plugin You finish with an existing NeMo Relay deployment routing through Switchyard. Requires NeMo Relay `>=0.8.1,<0.9.0`. @@ -159,6 +133,77 @@ while Switchyard owns provider HTTP dispatch. Details: [`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md) and the [server configuration guide](crates/switchyard-server/CONFIGURATION.md). +### Path 2 — Embed the Library + +You finish with your own harness picking a model per request and still making +every model call itself. Shown in Python; the Rust API has the same shape. + +**1. Install.** + +```bash +pip install nemo-switchyard +``` + +For Rust, add the crates to your `Cargo.toml` instead: + +```toml +[dependencies] +async-trait = "0.1" +futures = "0.3" +switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } +switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git", tag = "v0.2.0" } +tokio = { version = "1", features = ["macros", "rt"] } +``` + +**2. Construct an algorithm.** Target names are whatever your harness calls its +models. This is the stage router from the benchmark; `random`, +`llm_task_classifier`, and `llm_classifier` are built the same way. + +```python +from switchyard.libsy import LlmResponse, Step +from switchyard.libsy.algorithms import stage_router + +algorithm = stage_router( + "capable", + "efficient", + picker="efficient_first", + confidence_threshold=0.5, +) +``` + +**3. Drive it.** `run_stream` takes an OpenAI-style request dict and yields +steps. A `CallModel` step is a classifier or judge call — make it with your own +client and hand the result back. `Done` carries the pick. + +```python +async def route(request: dict, clients: dict) -> tuple[str, dict]: + async for step in algorithm.run_stream(request): + match step: + case Step.CallModel(call): + target = call.models[0] + try: + response = await clients[target].call({**call.request, "model": target}) + except Exception as error: + call.fail(error) + else: + call.respond(LlmResponse.Agg(response)) + case Step.Done(outcome): + return outcome.selected_model_ids[0], outcome.request +``` + +`clients` is your existing per-model client map. `call.models` lists fallbacks +in order; `outcome.request` is the request to send, which may carry a rewrite +the algorithm applied. + +**4. Make the answer call** with the returned model and request, using your own +HTTP client, retries, and credentials. If `outcome.response` is set, routing +already produced the answer and you can return it directly. + +Type reference: [`switchyard-libsy`](crates/libsy/README.md) and +[`switchyard-protocol`](crates/protocol/README.md). In Rust the loop is +`Algorithm::run_stream` yielding `Step::CallModel` and `Step::Done`, with +`switchyard-llm-client`'s `run` available to drive it for you. + ### Path 3 — Run the Standalone Proxy You finish with a server on `localhost:4000` that any OpenAI or Anthropic client diff --git a/assets/benchmark-accuracy-vs-cost.svg b/assets/benchmark-accuracy-vs-cost.svg index 08d7ccd4d..cc162c1c0 100644 --- a/assets/benchmark-accuracy-vs-cost.svg +++ b/assets/benchmark-accuracy-vs-cost.svg @@ -73,12 +73,12 @@ svg { GLM 5.2 Ultra 3 -SY: Staged +Switchyard: Staged 72.7% · $68.19 Kimi K2.6 -SY: Classifier +Switchyard: Classifier 71.2% · $79.32 -SY: Escalation +Switchyard: Escalation 75.7% · $85.00 DeepSeek V4 Pro Opus 4.8 baseline