Repository navigation
studio: the fine-tuning cockpit with Ctrl agent; brings in the synthesizer (#14) and SDFT/SDPO (#13) - #19
Open
patel-lyzr wants to merge 27 commits into
Open
studio: the fine-tuning cockpit with Ctrl agent; brings in the synthesizer (#14) and SDFT/SDPO (#13)#19patel-lyzr wants to merge 27 commits into
patel-lyzr wants to merge 27 commits into
Conversation
… unit
Three fixes the synthesizer needs underneath it, each standing on its own.
traces._span_call hardcoded tools=None, so a Trajectory rebuilt from spans
never carried the tool definitions the spans were already reporting — and
to_dataset therefore never emitted a "tools" key. Read them from
gen_ai.request.tools, or OpenInference's indexed llm.tools.{i}.tool.json_schema,
wrapping bare function schemas.
more_plus.split_units built a unit's BM25 surrogate from grp[0] alone, so when
a unit holds several phrasings of one fact, rows 2..k contributed nothing to
routing — an expert reachable only by the wording that happened to come first.
Join the whole group.
Neither backend passes a dataset's tool schemas to the training chat template,
while chat() does pass them at inference. Whether trl can carry a tools column
is version-dependent and unverified here, so say it out loud rather than let
the skew pass silently — the convention this repo already holds for config a
backend can't honor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t yet
capture() records a live agent, traces reads one that already ran, and
Dataset.from_* loads what you already have. All three need the traffic to
exist. This makes it: describe the task, point at a document, or amplify a few
real episodes, and a teacher writes the rest.
Two orthogonal axes again, mirroring backends × methods. Seeds decide where
scenarios come from (task / document / episodes); modes decide what gets
written per scenario (conversation / preference / paraphrases). Any seed
composes with any mode. Everything converges on Trajectory — the same type
capture and traces produce — and emit.py renders that into the shape the
consumer takes, chosen from the method's *spec* and never its name.
Generation is taxonomy first, instances second. A teacher asked for variety in
one breath rewrites one example n times; a teacher asked to fill a named slot
in a scenario tree doesn't. That structure, not prompt wording, is the
anti-mode-collapse mechanism.
Two consequences of the architecture worth naming:
- trajectory-GRPO learns from the spread between good and bad attempts, so
the judge scores but does not filter for that format — gating it would
throw away exactly the signal it needs.
- an outcome is accepted or rejected whole. MoRE+ groups rows by fixed size,
so dropping one row of a paraphrase unit would misalign every unit after
it; since a conversation outcome is one row, that is the same rule
everywhere.
Nothing is dropped silently: SynthReport reconciles exactly (generated == kept
+ invalid + duplicate + low-scoring + surplus) and report.balanced asserts it.
Teachers are duck-typed .chat(), so a frontier model, a local slm.load() model
and — later, for SDPO — the student itself are interchangeable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`shadowlm synth` with --dry-run, so you can see the resolved output shape before spending a teacher call on it. POST/GET /v1/synth follows the model-download pattern — a background thread polled for status — rather than the training queue, since synthesis is teacher calls and would otherwise hold the one training slot for the whole run. The finished rows land in DatasetStore like any other dataset. Teacher API keys are used for the run and never written to disk, unlike the HF token, which is deliberately persisted. Datasets gains a Synthesize tab beside Upload and Hugging Face: task, optional grounding document, target method, and a teacher that is either an OpenAI-compatible endpoint or a model already on this machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ples synthesize_from_task.py is the cold start — no agent has run, so there is nothing to capture and no traces to read. synthesize_from_doc.py is the signature path: facts pulled from a document, each phrased several ways, because MoRE+ routes on the question side and a fact asked about one way is an expert nobody can reach. CLAUDE.md gains the synth/ section and, while there, the traces.py and eval.py entries the architecture notes had drifted past. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything below was found by running the synthesizer against a real model.
A scripted fake answers the prompt you meant to write; a real one answers the
prompt you actually wrote.
Tool episodes failed 6/6. Three causes, all mine:
- The OpenAI wire shape is four levels deep — messages > tool_calls >
function > arguments — and the teacher miscounted the closing braces,
emitting `"arguments":{"invoice_id":"5678"}}}]}}` and losing the whole
episode to a JSON error. It also invented call ids it then failed to match.
So ask for a shallow {"call": {name, args}} / {"role":"tool","result":…}
pair and build the protocol in _wire_tool_calls, where the nesting and the
ids are right by construction. The teacher writes the semantics; we write
the wire format.
- A reply cut off by the token budget left the outer object unclosed, so the
object scan returned the *first message* instead: a dict that parses,
carries no conversation, and reads as "the teacher said nothing useful".
_parse_conversation now requires the shape, turning a silent dead end into
a retry.
- The judge scored only first-user-message against last-assistant-message, so
an answer citing what a tool returned looked unsupported and was gated out.
It now sees the whole lead-up — which is also the fix for any multi-turn
episode, not just tool ones.
Offered a "tool" role with no tools defined, the teacher invented tool turns to
narrate its own reasoning ("Classifying urgency: urgent"). Don't offer the role
when there are no tools, and say why.
per_scenario now defaults by shape rather than to 4. GRPO and the MoRE methods
want depth — several attempts to compare, several phrasings to route by —
but four conversations about one scenario are four rewrites of one example.
Measured on the same workload: 4 scenarios → 7, wasted retries 5 → 1.
Also cap the turn count in the prompt: max_seq_length defaults to 2048, so a
sprawling episode would have its final assistant turn — the thing being trained
— truncated away at training time.
Tool episodes now land 6/6 with correct wire form, the OTLP round-trip is
lossless on real generated data, and document→MoRE+ produces exactly k rows per
fact with every unit routable by its keyword-style phrasing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ld ask Leading the question prompt with "TASK: <what the assistant does>" made the teacher perform the task instead of writing the customer's message. The "question" came back as an assistant reply, "chosen" then answered that reply in almost the same words, and the pair collapsed into three near-identical strings that teach a model nothing. Both sides passed the not-equal check by differing somewhere past the first two hundred characters. So state the writing job first and demote the task to parenthetical context, and reject a pair whose question and chosen answer overlap heavily — the signature of the teacher having swapped roles again. The reason this survived until a live run: no fake-teacher test exercised the preference path at all, so renaming the prompt broke nothing visible. Covered now, including the student-as-rejected pairing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ask for JSON
A pass over the feature with fresh eyes, plus a live run to check each fix.
Robustness — the expensive failure was losing a run's paid-for rows:
- One teacher exception anywhere used to propagate through the thread pool
and kill the whole run at whatever it cost. Now a failed job counts as
invalid and the run continues; a circuit breaker trips after four
consecutive failures so a dead endpoint stops the run early *with* its
rows instead of burning a retry cycle per remaining job. Generation and
judging get separate breakers — a generation failure must not cancel the
scoring of rows already generated.
- serve.start_synth loaded a local teacher (and resolved an HF episodes
dataset) on the HTTP request thread; a cold 7B load held the POST for
minutes. Both now happen on the background thread the run owns.
Accounting — two places the funnel quietly lied:
- A GRPO group whose attempts all scored the same is dropped at emit (no
spread, no signal), but report.kept still claimed its rows. New
rejected_flat bucket; report.balanced now covers it. A live run promptly
produced one: "4 flat-group", reconciled.
- mean_score skipped rows scoring 0.0 (`if t.reward`), biasing the mean up
exactly where low scores are legitimate. Now averages judge_score metrics.
Coverage and cost:
- Document seeds ignored `avoid`: a top-up round re-extracted and
re-planned the same facts, dedup rejected them all, and the run stalled
without ever reading past where round one stopped. Facts are now cached
per chunk and skipped once planned, so round two advances into the
document.
- The CLI's --per-scenario default of 4 silently overrode the shape-aware
default (breadth for SFT, depth for GRPO/MoRE). It defaults to unset now,
and --dry-run prints the resolved value.
Quality:
- Student-as-rejected pairs are now judged on both sides; a pair where the
student scores no worse than the teacher is dropped — it would train the
DPO gap backwards. Teacher-corrupted rejects stay unjudged: flawed by
construction, scoring them doubles judge cost for nothing.
- OpenAI-compatible teachers ask for the server's JSON mode on
JSON-shaped prompts, with one plain retry (remembered) for servers that
reject response_format. Live effect: tool episodes went from 1 invalid +
1 repaired to 0 and 0.
- The paraphrase prompt now actually uses the style its metadata was
already claiming; serve treats min_score=0 as "no gate", same as the CLI;
to_groups' error named a parameter that no longer exists.
Without it wc -l undercounts by one — the report says 6 rows, the file reports 5 — and anything appending to the file corrupts the final row.
serve, dev and demo run $(VENV)/bin/shadowlm; on a fresh clone that is just a missing file, and make reports it as one. Point at make install / install-torch instead, and mention that the server also runs straight from any environment that already has the package. check and gpu-test only need the interpreter, so they now bootstrap the venv through the existing rule rather than failing the same way.
The wheel ships the compiled UI, so the built bundle is part of the feature, not a local artifact.
An empty key was accepted silently and only surfaced as a raw provider 401 once the run was already underway — a message that says nothing about where the key was supposed to go. Check it when the teacher is built instead, and name the three places it can come from. Only api.openai.com insists: vLLM and Ollama serve the same API without auth, so a custom base_url stays keyless.
…ument The studio sat at 0 for an entire run. on_progress fired once per round, and round one does all the work — so the bar had nothing to show until everything was already done, then jumped to 100%. Generation was parallel, but ThreadPoolExecutor.map hands back the whole batch at once, so there was no incremental signal even internally. run_jobs now collects through as_completed while writing results into their submitted slot: callers still get submission order (MoRE+ units are consecutive rows and must stay that way) and get a tick per finished job. on_progress becomes (done, total, phase) — planning, generating, judging, kept — and fires from every phase including the leaf planning. The studio bar tracks the live phase counter while a batch runs and the real kept count once a round lands; the server keeps logging one line per round rather than one per job, which would bury the report. Document fact extraction ran one chunk at a time, so a long document spent its whole extraction latency up front with the teacher idle. It now goes through run_jobs like everything else. Default frontier parallelism 4 → 8. Measured on the handbook document: longest silent stretch 7.1s → 5.2s, and that remainder is one indivisible extraction call, now shown as a labelled pulsing phase rather than a dead bar.
Two new methods on both local backends (mlx + torch), bringing the count to 15. Both make the model its own teacher: instead of cross-entropy on golden tokens or a scalar reward per rollout, the student's on-policy samples are pushed toward the same model reading richer context. - sdft (arXiv 2601.19897): on-policy learning from demonstrations. Each chat row's golden response is placed in-context for the teacher pass; the student's sampled completion is distilled toward the demonstration-conditioned distributions. Learns the task like SFT with far less catastrophic forgetting. Knobs: sdft_alpha (forward KL ↔ reverse KL ↔ JSD), completion length, temperature, teacher template. - sdpo (arXiv 2601.20802): RL via self-distillation on the grpo data surface. Rollout groups are scored by reward_fns — a fn may return (score, feedback) pairs — then each rollout is distilled toward the self-teacher reading a successful sibling solution and/or the textual feedback. Dense per-token advantages; all-fail groups still teach. The teacher trails the student as an EMA over adapter weights. Shared prompt-building and group-scoring helpers live in shadowlm/sdft.py and shadowlm/sdpo.py (pure stdlib) so both backends and the CPU tests use one implementation. Ships CPU tests, GPU matrix cells, mlx/torch/remote examples, and Train-page params (grouped under Preference · RL · distillation; no held-out eval, like grpo). The remote backend's reward_fns error now covers sdpo. pyproject: pin mlx-lm<0.30 and mlx-lm-lora==1.0.0 — newer releases require transformers 5.x / mlx-lm 0.30.6+, past our transformers<4.57 ceiling. Frontend static bundle rebuilt.
# Conflicts: # README.md # shadowlm/models.py
Three things stood between this and production, all of them about a feature whose entire cost model is "call a paid API in a loop". You could not see what a run cost. The provider reports usage on every response and we discarded it. Tokens now come straight from that block into the report — prompt and completion split out, and deliberately no price table, because a hardcoded rate card goes stale and then lies about money. A teacher that reports no usage (a local model) reads as zero rather than a guess. You could not stop one. A 5,000-row run started by mistake had to be killed with the server. synthesize() takes should_stop= and token_budget=; the studio grows a cancel button and the CLI a --token-budget flag. Both keep the rows already produced. They gate generation only, and that distinction is the whole design: rows that exist have already been paid for, so leaving them unscored fails them at the judge gate and wastes everything spent making them. Scoring what was produced costs a fraction of generating it. The test suite pins this — a generation outage must still let judging finish. The budget is honestly a throttle rather than a ceiling. Measured live, a 1,552-token budget landed at 8,587: in-flight parallel calls plus that deliberate scoring. Said plainly in the docstring and the CLI help rather than dressed up as a cap. Studio runs were memory-only and unbounded — lost on restart, growing forever. They now persist under work_root/synth/ like training jobs, with the log lines per run and the runs themselves both capped, and a record still marked running at startup is reported stopped, since its thread is gone. Two smaller things a real provider will hand you: 429s now honour the server's own Retry-After instead of our backoff curve, and a 400 naming max_completion_tokens switches the parameter and retries, so reasoning models work without configuration.
Only the docs collided, and only where upstream and this branch each added a row describing an inlet — trace ingestion and synthesis are both real, so both stay. Everything else merged clean; the suite is green at 293 tests. Worth noting for the review: upstream independently fixed the worker's backend_factory bug the same way this branch's sibling fix did, so fix/worker-honors-backend-factory is now redundant and can be dropped. Adopted from upstream on the way through: the login throttle, owner-only settings/token files, the request body cap and the tar member vetting all now sit under the synth routes too, which read their bodies through the same capped helper.
Resolving the README conflict spliced the cost-controls sentence into the middle of the coverage paragraph, leaving one block that ran from scenario diversity through accounting, budgets, output shapes and OTLP without a break. Three paragraphs: what makes the data good, what shape it comes out in, and what it costs to make.
Business and Research mode need the fine-tuning loop on the server: - projects (one fine-tune for a job) with their earlier versions kept as history - evaluations as background jobs scoring several targets on the same questions, including the user's frontier model as a baseline and as the judge - chat as a background task, since a held request dies at the proxy's 100 s - the frontier model setting (any OpenAI-compatible API; key stays server-side) - agent capture: /v1/capture/<id> passes calls through to the frontier model and records them; capture.reconstruct turns them into episodes, then a dataset - OpenTelemetry trace import as a dataset - deployments serving a fine-tune at /openai/v1 under its own hashed key
…rontend Making a model happens in one cockpit: Ctrl agent's conversation proposes each next step as an approvable action, beside a live map of the loop (data, fine-tune, evaluate, deploy) and an inspector for each station. Versions are compared on evidence; the start flow takes examples written, uploaded, from a dataset, captured from an agent or imported as traces. Two modes over the same objects (Business walks a project; Research keeps the object pages plus Evaluate and Deployments). The frontend moves to feature folders, one typed router over the URL hash, and a react-query data layer. Phones get a rail-by-default shell and a single-scroller cockpit.
Brings Pradipta's synthesizer — describe a task, ground it in a document, or amplify real episodes, and a teacher model writes the training data — onto the restructured studio: - its Datasets 'Synthesize' dialog is ported from pages/Datasets.tsx to features/datasets/Datasets.tsx in the console design system - synthesis can teach with the frontier model saved in settings (teacher kind 'frontier'), so its key is never pasted twice - the shipped UI is rebuilt
Brings Ghanshyam's self-distillation methods (sdft: on-policy learning from demonstrations; sdpo: RL via self-distillation) to 15 methods, onto the restructured studio: - its Train-page params land in features/train/Train.tsx; the held-out eval switch is hidden for grpo/sdft/sdpo, in the console design system - pyproject keeps main's packaging (mlx-lm-lora stays in the [preference] extra); mlx-lm-lora 1.0.0 is compatible with mlx-lm<0.30, so moving it into the base install is a separate decision - test_divergence_matches_trl_gkd skips when the installed trl no longer has GKDTrainer (removed in trl 1.x) instead of failing - SDFT / SDPO join the studio's method labels; the shipped UI is rebuilt
So the cockpit can use the synthesizer: - POST /v1/synth takes a project_id: the project links the run while it writes, and the result becomes its data when the run lands (an unknown project is a 422) - include_seed keeps the examples it amplified, so the next version trains on old and new together - a project remembers what its current version trained on (run_dataset_id), and history records that rather than whatever the data became since
Brings the synthesizer and the self-distillation methods into the loop:
- start a model from a description or a document: 'Generate them' on the Data
station; the frontier model writes the examples in the cockpit, and the plan
follows when they land
- Ctrl agent offers to write more examples when a version isn't ready on thin
data ('write 40 more examples' works too), shows them arriving on the Data
station, then proposes the next version on all of them
- the fine-tune card's method choice offers LoRA, SDFT (keeps more of the
base model's general skills) and MoRE; 'use SDFT' picks it
- the live decision stays at the bottom of the thread; '1 example', not '1 examples'
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Making a model now happens in one cockpit. Ctrl agent, on the right, proposes each next step as an action you approve. Beside it is a live map of the loop (Data → Fine-tune → Evaluate → Deploy), with an inspector for each station. The studio has two modes over the same objects: Business mode walks a project; Research mode keeps the object pages, plus Evaluate and Deployments.
This PR also brings in #14 (data synthesizer, @pradipta-lyzr) and #13 (SDFT + SDPO, @ghanshyam-lyzr). Both are wired into the new UX, and merging this closes both.
The cockpit
/v1/capture/<id>), or imported as OpenTelemetry traces./openai/v1under a hashed key for each deployment.The two PRs, used in the cockpit
Synthesizer (#14)
POST /v1/synthtakesproject_id. The project links to the run, and the result becomes the project's data.include_seedkeeps the examples the run amplified.SDFT / SDPO (#13)
Notes for review
mlx-lm-lora==1.0.0into the base install. This branch keepsmain's packaging ([preference]extra,>=2.0). 1.0.0 does install alongsidemlx-lm<0.30, so making it a base dependency is a separate call.test_divergence_matches_trl_gkdnow skips when the installed trl has noGKDTrainer(it was removed in trl 1.x).make checkis clean.