[Feature] Endpoint preset trials#4049
Draft
peterschmidt85 wants to merge 7 commits into
Draft
Conversation
Sequential experimental trials for endpoint preset creation: the agent searches serving configurations across hardware, frameworks, and variants in `dstack` tasks, records reproducible trial records (trials.jsonl), promotes the best to a verified service, and saves the preset. Verified across nine live e2e sessions (Qwen2.5-0.5B, Qwen3-32B on H100, Qwen3-32B under $1/hr — best result 533 tok/s on a single RTX 5090 via NVFP4 + EAGLE3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ycle list - Preset ID minted at session start and stable through run names, the session directory, and the final preset; sessions survive Ctrl+C, hard kills, and host restarts and continue with --resume <preset id> - Agent deaths without a submitted report retry in-process; the retry budget refills only when the failed attempt made progress - Durable workspace under the session directory with a /tmp symlink alias (Unix socket path limit); byte-offset persistence prevents duplicated output on resume; stale running manifests downgrade to interrupted - Flat ~/.dstack/presets/<id>/ store holding preset.yaml next to session internals; lazy migration from the models--* layout; delete archives to .archive/ instead of destroying - dstack endpoint preset list shows in-flight and interrupted sessions with colored statuses, trial progress after the status, best-trial benchmark and GPU, -w watch mode, and -v details - Top-level base:/repo: configuration shorthand; nested model: deprecated unless model.name is set; --base/--repo filters on list and delete Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A watchdog process holds a pipe from the CLI and kills the agent's process tree the moment the pipe snaps — covering deaths the CLI cannot react to (SIGKILL included), which previously left an orphaned agent able to keep submitting GPU runs. Graceful stops disarm the watchdog. - POSIX kills the agent's process group; Windows kills the process tree - Presets and sessions sort newest-first within each base-model group, matching dstack ps Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sion A final report echoing a known secret (e.g. the bearer token in benchmark.command) was rejected wholesale, discarding an otherwise verified session at its very last step. Scrub known secret values from the report structure before validation; the bearer check still rejects unknown leaked tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ish list
- Configuration `type: preset`; flat `dstack preset {list,get,create,apply,delete}`
command tree; user-facing strings follow (internal names are a follow-up)
- `apply` selects among repeatable `--id` candidates (given order,
capacity-aware) replacing the machine-local `preset:` configuration field
- The bearer-token report guard only rejects credential-shaped values;
prose such as "auth via bearer header from env" failed two live sessions
- Preset rows show `success (n/m)` from the completed creation session;
trial count is the record count again (one long-lived task commonly hosts
several trials, so distinct task names undercount)
- List polish: dimmed base group rows and trial progress, green `success`
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- BENCHMARK renders identically with and without -v; the verbose-only detail dump is gone (full data remains in `preset get --json`) - -v adds exactly two things: a `ctx=` benchmark prefix (replacing the CONTEXT column) and a dimmed indented `repo=` on the preset's own row - `apply` shows the selected preset with the same `ctx= con= tok/s TTFT` string Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `prompt:` in the configuration, inline or as a file `path:` relative to the configuration file; validated and capped - The agent contract gains the text via template directives in system_prompt.md (`<!--?prompt:...-->`); without a prompt the rendered contract stays byte-identical. The default no-patching limits gain an escape clause the prompt can invoke explicitly - The resolved prompt is pinned per session and kept across resumes, with a warning when the configuration changes meanwhile - The list shows `verifying` once a running session's trial budget is spent and the final service is being deployed and verified Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
dstack preset— agent-driven search for the best model serving configuration. A headless agent (Claude Code) runs sequential trials on real fleets — hardware, framework, quantization, engine parameters — benchmarks each, deploys the best as a verifieddstackservice, and saves the result as a reusable preset. Purely client-side: works against any existing dstack server.Key changes
type: presetconfiguration:base:orrepo:,max_trials,concurrency,context_length,fleets,env, andprompt:— custom agent instructions (inline or a filepath:) that can reshape objectives and explicitly extend the default limitsdstack presetCLI:create(--resume ID),list— a live dashboard of presets and in-flight sessions,apply(repeatable--id),get,deleteHow to test
Install from this branch:
Prerequisites: any dstack server with a GPU-capable backend (the feature is client-side, so an existing deployment works as is), at least one fleet, and an authenticated
claudeCLI.Interrupt a
createwith Ctrl+C and continue it withdstack preset create -f preset.dstack.yml --resume <ID>.🤖 Generated with Claude Code