Multi-backend GPU research orchestration — the platform layer I built to run a self-funded, multi-project ML research program (knowledge distillation, RL post-training, agent evaluation) across five compute providers on a ~US$5,000 total budget.
When every run costs your own money, the platform has to answer three questions before any GPU spins up: what will this cost, what will we learn, and what happens when it fails? This codebase is the answer: contract-typed backends, preflight gates, automated postmortems, and a file-driven experiment lifecycle that survives across sessions and machines.
rig/
├── backend/ # One Protocol, five implementations
│ ├── protocol.py # Backend interface: search → launch → poll → collect → destroy
│ ├── modal_backend.py # Serverless (per-second billing)
│ ├── vast.py # SSH-based GPU rental (cheapest raw compute)
│ ├── spartan.py # SLURM HPC (sbatch / squeue / filesystem collect)
│ └── registry.py # Backend dispatch
├── trial/ # The unit of spend
│ ├── schema.py # TrialConfig / TrialResult / Context / Execution contracts
│ ├── preflight.py # Pre-launch gates: credentials, mounts, cost estimates
│ ├── postmortem.py # Auto-capture on completion/failure: log tail, W&B summary,
│ │ # cost actuals, failure classification — never blocks, never fails
│ └── ops_friction.py # Infrastructure failures become written memos automatically
├── failure/ # Taxonomy: OOM / NCCL / disk / NaN / SSH / unknown
├── artifact/ # Lifecycle promotion: RAW → CANDIDATE → BENCHMARKED → DEPLOYABLE → DEPLOYED
├── tracking/ # W&B middleware + query CLI
└── pipeline/ # Study / experiment schema contracts
Backends are a Protocol, not a base class. Five providers with nothing in common — serverless Python calls, SSH+SCP loops, SLURM queues — share only a contract: search_instances → launch → poll → collect_files → destroy. The platform never SSHs or creates instances directly; training scripts are opaque commands the backend executes. Adding a provider means implementing six methods, not inheriting behaviour you then fight.
Failures are data. Every run ends in a postmortem: exit code, classified failure mode (OOM / NCCL timeout / disk full / NaN loss / SSH drop), log tail, and cost actuals — captured best-effort so the capture path itself can never break a run. Infrastructure friction auto-generates written memos. The result: a 30+ entry operational gotcha log instead of re-learning the same Modal volume race three times.
Cost is a first-class field. ResourceSpec carries max_dph and max_hours; every backend implements estimate_cost(); preflight refuses launches that can't state a price. This discipline is why a full distillation research program — including a pipeline rebuild that cut training cost from $22 to $3.85 per 1k steps (15× per-token throughput) — fit in a ~$5k budget.
State lives in files, not in heads. Experiments are driven by YAML/Markdown artifacts (study.yaml → tracker.yaml → analysis.md → decision.yaml, see templates/) so any session — human or agent — can pick up exactly where the last one stopped.
pytest tests -q # 102 passed
Contract tests verify every dataclass, enum, protocol, and schema version; behavioural tests cover backend dispatch, preflight gating, postmortem classification, and artifact lifecycle transitions.
Built and operated solo, 2025–2026, driving:
- Knowledge-distillation training on 4×H100 / 4×B200 (FSDP2, bf16) at 98–99% GPU utilisation, 15.5k tok/s sustained
- A 30k-sample / 29 GB teacher-logprob corpus generated at 1.37 samples/s (SGLang, trtllm_mha backend: +8.6% throughput, −49% cold-start vs flashinfer)
- 30+ experiment cycles across four concurrent projects, with every run's cost, config, and failure mode on record
The project-specific research layers (datasets, training recipes, competitive strategy) are private; this repository is the reusable platform underneath them.
MIT