[rycerzes]

Summary

The experimental, eval-first env-only prototype agreed in #1382: batch jobs on HF Jobs, an evaluator (dev_evaluate vs score) and a per-rollout budget. The agent post-trains a model by submitting scripts. The env server's broker runs them on HF Jobs with its own token. A grader-only job scores the submitted checkpoint on a hidden split, and that score is the reward. No core changes.

cc @burtenshaw @sergiopaniego

Type of Change

• New environment
• Environment documentation (RFC 014 is separate in RFC 014: Environment Services (design only) #1508)

What's in it

• Broker:
  • Holds the only token; agent jobs start with secrets={}.
  • Each job reserves its worst case (price/min × timeout) and settles at the minutes it ran.
  • Jobs end with the episode; a cancellation HF doesn't confirm marks the job orphaned and blocks the next reset.
• No network in agent jobs. HF Jobs has no egress control, so a launcher installs the agent's packages (PyPI wheels only), then applies an unprivileged seccomp filter that refuses non-AF_UNIX sockets, and drops privileges. Inputs are read-only mounts: the pinned base model and the public datasets the agent asks for.
• Evaluator:
  • Only safetensors weights come from the checkpoint; everything else comes from the base model.
  • A failed score job ends the episode unscored (reward=None), which RFC 013's update leaves out.
• service_spend in every observation, as a cost for RFC 013 trainers, never a sampling weight.
• Two T0 tasks that check the plumbing, and examples/posttrain_env_agent.py.

Compared with huggingface_hub

• Uses run_uv_job as is, including its jobs-artifacts bucket and /data folder.
• Uses Volume mounts for inputs, and list_jobs_hardware / inspect_job for prices and durations.
• Never runs hf jobs uv run, whose script header can request HF_TOKEN from the caller (source).
• The one gap is egress: run_job only has network_group. A native no-network option would replace the launcher.

Known gaps

• The server has no auth; keep it on a private network.
• Bucket writes aren't capped.
• The step and time limits apply on step, not on direct /mcp calls.

All three are in the README.

RFC Status

• RFC 014 is split into design-only PR RFC 014: Environment Services (design only) #1508 (origin/discussion RFC: Environment Services — brokered, budgeted access to external services (training, GPU jobs, evaluators, simulators) #1382), still In Review. This env PR has no RFC file changes and no core changes. Does not depend on RFC 009.

Test Plan

• PYTHONPATH=src:envs uv run pytest tests/envs/test_posttrain_environment.py -v: 58 pass against a fake HF Jobs; one Linux-only seccomp test also passes in a container. The full suite passes apart from three timing tests that fail the same way on main locally.

• Live runs on HF Jobs, with no network in agent jobs (full table in the RFC):

  Task	Driver	Reward	Agent job spend
  t0_artifact	script	1.0	$0.0002
  t0_arithmetic_format	scripted SFT, t4-small	0.93	$0.041
  t0_arithmetic_format	Qwen3.8 via HF Inference Providers	1.0	$0.12

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

Maintainer review follow-up

Addresses #1500 (review):

• RFC-only RFC 014: Environment Services (design only) #1508 preserves @rycerzes's nine RFC revisions, author/date and source hashes, author credit and prototype results. It adds the proposed provider contract and second-backend/native-network prerequisites before step 2. No new backend, core manifest or reserved prefix is implemented or adopted. None-reward integration remains contingent on RFC 013.
• README, client docstring and RFC example use step(CallToolAction(...)), including the final reward and observation service_spend. README explains direct MCP bypasses episode step/time limits, eval-first use, blocking submit (2700 s), timeout overhead, forwarding settled spend as costs, and spend final versus episode done.
• Experimental labels in README/catalog; SKIP_HF_DEPLOYMENT excludes automatic Spaces releases. No Space or collection entries are added. Local entry point binds 127.0.0.1; Docker keeps its explicit 0.0.0.0 command.
• Direct-MCP episode-limit enforcement is tracked in Enforce episode step and time limits for direct MCP tool calls #1507, separate from request-security Opt-in request security for env servers (rate limiting, IP lists, payload screening) #1360.
• Normal main merge, no force push. Private broker, blocking submit, orphan/reset guard and network policy are retained. The checkpoint unexpected-keys guard is unchanged: the reported normal-save failure remains unsubstantiated, and the author's exact-loader HF checks are recorded in Add experimental posttrain_env prototype (RFC 014) #1500 (comment). Score-file transport remains a follow-up.

Validation: 95 passed, 1 Linux-only skip across posttrain and deployment-script tests; the README/client examples execute through server serialization and client parsing. RFC example also executed offline. No real HF Jobs or spend. Changed Python files pass lint/format/import sorting; environment doc synchronization and whitespace checks pass. Full-repository lint still reports existing main formatting/import-sort failures. Full local pytest selection: 3958 passed, 150 skipped, 79 deselected, 3 failed (259 s). Failures are in unchanged main files: two harness-process tests raised macOS os.killpg PermissionError during the full run, but passed in isolation on main; the Harbor occupied-port test failed identically on main (expected "already in use", observed startup timeout). These results do not establish a regression from this change. Full lint has 47 pre-existing main formatting findings plus the two known usort files; changed files pass. The test hook itself lacks timeout on this host, so the same pytest selection ran directly. CI was pending at the post-push check; no monitoring requested. Latest-main changes are merged.

---

Note

High Risk
Introduces a credential-holding broker that can spend real HF Jobs budget if mis-deployed; job isolation relies on seccomp rather than platform egress controls, and the server has no built-in auth.

Overview
Adds posttrain_env, an experimental MCP environment (RFC 014 prototype) where agents post-train models via brokered Hugging Face Jobs—the server holds HF_TOKEN, enforces a reserve-then-settle per-rollout budget, and uses a hidden-split grader job as the episode reward.

Agent jobs run inside job_launcher.py: PyPI wheels only, read-only model/dataset mounts, then seccomp blocks non-Unix sockets before the user script runs. MCP tools cover jobs_submit / status / wait / logs / cancel, dev_evaluate (public dev split), and submit (final score + episode end). Observations include service_spend for trainer cost accounting.

Also ships two T0 tasks, PostTrainEnv client, examples/posttrain_env_agent.py, site/docs updates, SKIP_HF_DEPLOYMENT (private deploy only), and a large fake-HF-Jobs test suite (~2k lines).

^{Reviewed by Cursor Bugbot for commit 15d7d02. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[burtenshaw]

Thanks for this. I checked out 8429a25 and ran the checks locally. Findings below; the RFC design itself is for maintainers to decide, so I list open questions rather than a verdict on it.

Non-blocking

• server/eval_scripts/t0_arithmetic_format_eval.py:80 rejects missing and mismatched keys but not unexpected_keys. Transformers logs unexpected key names, which the agent controls, and broker.py:538 takes the score from any log line starting with POSTTRAIN_SCORE . The real score line is printed last, so I could not build an exploit, but rejecting unexpected keys (or writing the result to a file in the score job's /data instead of logs) would close it.
• rfcs/014-environment-services.md:8 links to huggingface/OpenEnv; the repo now lives at openenv/OpenEnv (the redirect works).
• Decision 1 in the RFC cites only TRL's AsyncGRPOTrainer for staleness. A second framework, or a framework-neutral phrasing, would keep with the neutrality rule.

Open design questions

1. Is the env server as broker (credential in the env process) acceptable for v1, given the server has no auth and must stay on a private network until RFC 009 lands?
2. submit blocks one WebSocket step for up to eval_timeout*60 + 900 + 240 s. Is a long blocking step the right contract, or should scoring be async with a status poll (#1026)?
3. Should an orphaned job block the next reset with an error, or should the orchestrator get a typed state it can handle without a failed reset?
4. Reward is None on a failed score job but 0.0 on step or time limits. Is that the split the trainers in RFC 013 expect?
5. Step and time limits are enforced only on step, not on direct /mcp calls. Is that acceptable for v1, or should the limits move into the tool path?
6. The no-network policy depends on a seccomp launcher because HF Jobs has no egress control. Should the RFC treat a native HF Jobs no-network option as a precondition for step 2, and should arbitrary PyPI wheels and public datasets stay allowed given the contamination risk the RFC names?
7. The service manifest lives in tasks.py for now. Is the proposed manifest shape and jobs_ reserved prefix the one core should adopt in step 2?

[rycerzes]

Thanks @burtenshaw! The three non-blocking points are fixed in eb21148. The scope is the slice agreed with Sergio in #1382, and the open questions map to the RFC's plan:

1. Fine for v1: the token never reaches the agent, and a Resource Group spend limit caps spend. Credential custody moves to openenvd in step 3.
2. I'd keep it blocking for v1. reset returns score_timeout_s, so clients can size their timeout.
3. orphaned_jobs is already in the final observation. A typed state belongs with the cancel-failure cleanup in step 2.
4. Yes, on purpose. None = our failure, left out by RFC 013; 0.0 = the policy hit a limit.
5. OK for v1 while the server is private. Enforcing limits on /mcp too belongs in core, so step 2.
6. Yes, a precondition for step 2. Feature request: [Jobs] Option to run a job without outbound network access huggingface/huggingface_hub#5261. Wheels and datasets stay allowed for now, behind the denylist.
7. Yes, that's step 2: the manifest and the jobs_ prefix go into core in a follow-up PR, using tasks.py as the starting shape.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[sergiopaniego]

thanks @rycerzes, great work! the scope matches what we agreed in #1382: batch jobs, dev_evaluate vs score and the per-rollout budget, with the other profiles moved to future work. the live runs table in the RFC is really useful too.

I tested it end to end on HF Jobs (my own namespace, at 83202a0):

• unit tests: 84 passed, 1 skipped (the Linux-only seccomp one).
• t0_artifact, scripted: reward 1.0, $0.000167.
• isolation probes inside an agent job: no *TOKEN* in the env, uid 10000, AF_INET/AF_INET6/AF_NETLINK/AF_PACKET/AF_VSOCK sockets, a raw socket syscall and io_uring_setup all get EACCES, DNS fails, no Unix sockets in the container, pid 1's environ isn't readable and /base_model is read-only. Nice job on the launcher.
• t0_arithmetic_format, scripted SFT on t4-small: 310 s, $0.040, dev 0.92, hidden split 0.92, close to your numbers. The ledger matched: $0.20 reserved, $0.040 settled, and the score job wasn't charged.
• the example LLM agent (Qwen3.8 via Inference Providers): 19 turns, 3 agent jobs (the first failed on its own tokenizer bug and it recovered from the logs), reward 1.0, $0.37 of jobs (gpu_s 1283), no orphaned jobs and final: true. The final observation came through step with the reward and service_spend as expected.

Split the RFC from the env

The goal of the prototype was to have a real, working example of the design, and it does that really well. Since the RFC is a design decision and the env is an implementation, I'd land them as two PRs, like RFC 005 did:

• RFC PR: only rfcs/014-environment-services.md and the rfcs/README.md entry, with the ../envs/posttrain_env/ link pointing to the env PR. The "Prototype results" table can stay, since it's the evidence that the design works. No core changes needed: the core types (manifest, ledger, jobs) stay as step 2, in their own PR. Happy to approve this one as soon as it's up.
• Env PR (this one): posttrain_env as an experimental env. I'd merge it once:
  • the README "Client" section, the PostTrainEnv docstring and the RFC example use env.step(CallToolAction(...)) instead of env.call_tool(...). call_tool goes through /mcp, so it skips max_steps and the time limit (I checked: three call_tools leave step_count at 0) and never returns the final observation with the reward and service_spend. examples/posttrain_env_agent.py already does it right.
  • the README says this is eval-first today (see below).
  • it's labelled as experimental in the README and the docs env list, and stays out of the Spaces release and the collections.
  • small one: server binds 0.0.0.0 by default while the README says to keep it on localhost. Defaulting to 127.0.0.1 outside Docker would match.

Then we keep iterating on it in main with smaller PRs: the first real task (something closer to PostTrainBench, where the hidden split can't be enumerated), reading the score from a file, and so on.

Provider neutrality (before step 2)

The three rules, the ledger, service_spend, the job states and the dev_evaluate / score split aren't HF-specific, which is great. The v1 mechanics are, though: run_uv_job, artifacts as folders in the jobs-artifacts bucket, per-minute billing from list_jobs_hardware, the Resource Group spend cap, and the seccomp launcher, which only exists because HF Jobs has no egress control (Modal, for example, seems to have native network blocking for sandboxes).

Since step 2 moves these types into core with a single backend, it'd be easy for the abstraction to end up shaped like HF Jobs. Two suggestions:

• add a short section to the RFC with the minimal contract a batch-job provider has to offer (submit, inspect, cancel, logs, price, artifacts), with HF Jobs as its first implementation.
• sketch a second backend before step 2. A local Docker one would be the cheapest and would also give us end-to-end tests that spend nothing. Modal would be a good check that the abstraction isn't HF-shaped.

Training / HF side

• README: say this is eval-first today. RFC decision 1 says post-training episodes are multi-hour and fall outside the staleness window of async GRPO trainers, so their near-term use is evaluation. The env README doesn't say that, and someone will try to plug posttrain_env straight into a trainer's rollout loop. A short "Using it with a trainer" note would help: submit blocks for up to score_timeout_s (2700 s for t0_arithmetic_format), so the client's tool/message timeout has to be sized from that value, and metadata["service_spend"] is what to forward as costs.
• Score from a file instead of logs. +1 to @burtenshaw's suggestion. Evaluator jobs already get a read-write /data folder in the jobs-artifacts bucket, and the broker already lists that bucket for artifacts. If the evaluator wrote /data/result.json and the broker read it from there, the log parsing (and anything the agent can make the loader print) would be out of the picture. Fine as a follow-up.
• No-network flag on HF Jobs. Thanks for opening huggingface/huggingface_hub#5261, let's follow it there.
• reward=None on a failed score job. Could the RFC say explicitly that this depends on RFC 013 (#1256), so it follows whatever 013 settles on?

On the open design questions

1. Broker inside the env server: fine for v1, documented as private-network only.
2. Blocking submit: fine for v1. It keeps the reward inside the env, and score_timeout_s lets clients size their timeout.
3. Orphaned job blocks reset: fine for v1. Failing loudly beats billing silently. The typed state can come with step 2.
4. None vs 0.0: agree with the split. A grader failure shouldn't penalize the policy.
5. Limits only on step: fine for v1 while the server is private, once the docs point clients to step (see above). Since enforcing them on /mcp is a core issue rather than a posttrain_env one, could you open an issue so it doesn't get lost?
6. Native no-network flag as a precondition for step 2: agree. Open wheels and datasets are fine for T0, but a real task with public test data will need a contamination check or an allowlist.
7. Manifest and jobs_ prefix in core: not yet. I'd wait for the provider contract and a second backend (see above) before adopting them.

thanks again, really solid work 🚀

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 834c8c2. Configure here.}