[Feature] Endpoint presets

[peterschmidt85]

Change

Inference optimization for a specific model and hardware requires the use of agents. This PR
introduces a dstack primitive called endpoint presets that automates and streamlines the use of
agents to build optimized inference configurations and reuse them.

flowchart TD
    config["Endpoint configuration<br/>(e.g. endpoint.dstack.yml)"]
    create["dstack endpoint preset create<br/>-f endpoint.dstack.yml"]

    config --> create --> discovery

    subgraph agent["Agent"]
        discovery["Inspect model, fleets,<br/>backends, and offers"]
        task["Prototype using<br/>dstack tasks"]
        service["Deploy and verify<br/>a dstack service"]
        benchmark["Benchmark<br/>the service"]
        discovery --> task --> service --> benchmark
    end

    preset["Save preset<br/>~/.dstack/presets"]
    apply["dstack endpoint preset apply<br/>-f endpoint.dstack.yml"]
    select["Select a matching preset<br/>for current offers"]
    deploy["Show the service plan<br/>and submit the service"]

    benchmark --> preset
    config --> apply
    preset --> apply
    apply --> select --> deploy

Loading

How it works

• preset create starts the local claude CLI on the CLI host with the bundled dstack skills.
  The agent uses the regular dstack CLI against the configured server and project. Its tasks and
  final service run on the configured backends.
• The CLI combines the verified service run with the agent's model, context-length, and benchmark
  report. It saves the resulting YAML preset under ~/.dstack/presets.
• Presets contain the base and selected model, service configuration, context length, exact
  validation resources, and benchmark. The server continues to manage fleets, offers, tasks, and
  services through its existing APIs; this PR adds no endpoint-specific server API or database
  state.
• preset apply reads local presets, applies the endpoint and profile constraints, selects a
  matching preset for the current offers, and submits its service through the existing service
  configurator. It does not run an agent.

Define an endpoint

type: endpoint
name: qwen25-7b

model:
  base: Qwen/Qwen2.5-7B-Instruct

env:
  - HF_TOKEN

model.base allows the agent to select a compatible model variant. An exact model repo can be
specified with model: <repo>. Placement properties such as fleets, backends, regions, resources,
spot policy, and maximum price constrain both preset creation and reuse.

Create a preset

dstack endpoint preset create -f endpoint.dstack.yml

The endpoint name can come from the configuration or --name. Creation uses either an existing
local claude login or an Anthropic API key.

List presets

$ dstack endpoint preset
 MODEL                     GPU                    CONTEXT  BENCHMARK
 Qwen/Qwen2.5-7B-Instruct
    preset=8f3a12c4        nvidia:16GB..24GB:1..  32K      concurrency=1 464 tok/s TTFT 312ms

Presets are grouped by base model. Each row shows the preset ID, GPU requirements, context length,
and benchmark. dstack endpoint preset is the short form of dstack endpoint preset list.

Apply a preset

$ dstack endpoint preset apply -f endpoint.dstack.yml
 Model          Qwen/Qwen2.5-7B-Instruct (base)
 Preset         8f3a12c4 (context=32K, concurrency=1 464 tok/s TTFT 312ms)

 #  BACKEND            RESOURCES                      INSTANCE TYPE     PRICE
 1  runpod (CA-MTL-1)  cpu=9 mem=50GB disk=200GB      NVIDIA RTX A5000  $0.27
                       gpu=A5000:24GB:1
 2  runpod (CA-MTL-1)  cpu=9 mem=50GB disk=200GB      NVIDIA RTX A5000  $0.27
                       gpu=A5000:24GB:1 (spot)
 3  runpod (US-IL-1)   cpu=12 mem=25GB disk=200GB     NVIDIA RTX A5000  $0.27
                       gpu=A5000:24GB:1
    ...
 Shown 3 of 4 offers, $0.27max

Submit the run qwen25-7b? [y/n]: y

Without an explicit preset in the endpoint configuration or --preset, dstack selects a matching
preset based on the current offers.

Delete a preset

dstack endpoint preset delete --preset 8f3a12c4

See the Endpoints concept page
for the complete UX and configuration reference.

AI assistance

AI assistance was used for implementation, tests, documentation, and E2E investigation.

Experimental

The endpoints feature is experimental and may change.

Roadmap

1. Support multiple trials, allowing the agent to improve benchmark results based on previous trials.
2. Allow the endpoint configuration to define custom agent instructions, such as target metrics and the experimentation approach.
3. Control which experiments the agent may perform, including modifying serving framework code and generating custom kernels.