Repository navigation
Rebuild the test suite as black-box scenarios - #700
Draft
jdchawla29 wants to merge 250 commits into
Draft
jdchawla29 wants to merge 250 commits into
jdchawla29 wants to merge 250 commits into
Conversation
Tests lived in a tests/ package inside each hud subpackage, so the wheel shipped them and the sdist shipped the Harbor fixtures too. They now live in a top-level tests/ package that mirrors the hud layout: tests for hud/cli/ in tests/cli/, tests for top-level modules in tests/ itself. Paths that assumed the old depth now resolve from the module under test, pytest collects tests/, ty checks it alongside hud, and the build and coverage excludes for nested test directories are gone.
Tests drive the SDK through its public boundaries and fake only what lies outside it. tests/harness provides those fakes: one local server standing in for the HUD API, telemetry, gateway, runtime and training services; scripted model providers speaking Chat Completions, Responses, Anthropic Messages and Gemini; the real hud CLI in a subprocess; a docker executable on PATH; and a VNC screen. Every test starts from an isolated configuration set through environment variables, with service URLs pointed at a closed port so nothing reaches a real deployment. AGENTS.md, CONTRIBUTING.md and a new REVIEW.md state the rules, and scripts/check_tests.py enforces them in CI. Tests needing Docker, a Linux sandbox, or a real HUD account are marked e2e and run in their own CI lanes.
The exact pin failed every other uv version outright, including the ones container images ship. The root lockfile is not committed, so the exact pin bought nothing beyond choosing CI's uv; a floor keeps the minimum the workflows rely on without blocking newer releases.
Replace the server, manifest and argument tests, which reached into _TaskFactory.manifest_entry and the framing helpers, with one served environment spoken to in raw JSON-RPC. Every reply frame is compared with a golden copy: the tasks.list manifest, each grading and argument shape, the error replies for malformed requests, and how cancel, bye, grade, restart and disconnect close a template. Sessions and tunnels move to the same raw wire, so no test patches StreamWriter or reads a private endpoint; the workspace-binding session rows move to the sandbox lane.
tasks.start read its args as `params.get("args") or {}`, so an empty
list, zero or an empty string slipped past the object check as no args
and surfaced as a confusing signature error (or ran a no-arg template).
Only an absent or null args now means none.
tasks.start caught KeyError around the whole start, so a template that raised KeyError before its prompt (a missing dict key, an env lookup) was answered with "unknown task" naming the template that exists. The template id is now checked before starting, and anything the template raises reaches the peer as the error it is.
The tracker tests constructed FileTracker and patched its private size caps. Serve a tracking workspace over a fixture tree instead and compare each reply on the wire: the manifest without build output, VCS metadata, ignored files or secrets; agent edits with redacted secret placeholders and the oversized-file placeholder; the setup layer, advance, and a flush that captures deliverables, skips secret-shaped ones and flags a capture cap. An unknown method gets an error and the connection keeps serving.
Collapse the loader tests into one table over a fixture tree holding every source shape a project has, each row resolving a reference to an environment name or the error a user sees. Add the serving process to the same tree: python -m hud.environment.server and hud serve announce their port, answer hello, and run the environment's shutdown hook when stopped; a reference that does not load exits with the load error.
load_environment raised FileNotFoundError for a missing source only when the reference contained a slash. A bare file reference such as `hud serve envv.py` fell through to a module import and failed with "No module named 'envv'". Any .py reference that is not on disk is now reported as a missing environment source.
Only python -m hud.environment.server turned SIGTERM into a clean stop; hud serve called serve() directly, so a SIGTERM (docker stop on the harbor image, whose CMD is hud serve) killed the process without running the environment's shutdown hooks or stopping its daemons. serve() now owns the SIGTERM handling for every caller and returns normally when terminated.
Rewrite the direct-control tests as one table of rollouts: each row records what the sim was commanded, every tool result as the model reads it, and the reward. Add the yaw-only move at pitch +-pi/2, which ported the private orientation test. Telemetry is read from the span file instead of patching queue_span and swapping the TraceRecorder class, and the mid-move end comes from cancelling the session the way a failed agent does. The harness steps() takes an optional schema so robot telemetry, which uses its own span schema, is readable the same way.
The slot tests mocked RobotEndpoint._call, seeded _claims and the bridge registry, and drove the private tick loop with fake sockets. Serve a stub bridge instead and claim slots over its control channel, through an env template driving RobotEndpoint, and from RobotClient connections. The rows cover a full batch, mismatched batch args, cancel/bye/grade/drop each freeing a slot exactly once (with a failing release retried, and drained on stop), a silent slot timed out beside an active peer, an undialed claim holding the batch, and no stepping after a terminal observation.
The regression tests packed and applied spawn state through private helpers, hand-built GymBridge internals to check action reshaping, and patched the recorder's report function. Spawn a declared bridge and read back its contract, slots and metadata from the child, and check that a sim exiting before its port fails start. Serve env.gym over a Box(2, 2) env and a **kwargs vector factory and send flat actions from a RobotClient; each env reports the shape it was stepped with. The job-id spelling rows now observe the reports on the fake platform.
The hint catalog in hud/utils/hints.py was only rendered by HUDConsole.render_exception, which nothing calls; the CLI prints errors through CliError. Exceptions no longer carry hint lists. hud.graders no longer exports the private _combine_subscores, which combine() uses internally.
One snapshot of every package's __all__ replaces the hand-kept export lists in test_init.py and test_init_module.py; each name must resolve and none may be private. hud version --json and hud --help run in a subprocess, the way users see them.
Rollouts with telemetry on run against the fake telemetry service. The scenario pins the upload path, bearer, body shape, per-run trace grouping, step order and status, the 100-span/4 MiB batch limits, the 503 retry, the no-key and uploads-off cases, flush's deadline, the exit flush, and that a dead service leaves the run and Taskset.run intact.
Runs with each span-directory configuration assert where their spans land, that an unwritable directory does not fail the run, the step span schema and timing, and what @hud.instrument records under a real trace (self dropped, defaults bound, lists truncated, exceptions, record_args/record_result off, unserializable values). The exporter and instrument tests that patched hud seams or never set a trace context are removed.
Trace's final, collect, error, numbering and reload rows now run on a rollout's own trace instead of hand-built steps; the span emission rows that patched hud.types.queue_span moved to the telemetry scenarios.
test_types.py pinned str/rich formatting trivia, pydantic defaults on MCPToolCall.annotation and a private summary helper; test_serialization pinned a helper the instrument and grader scenarios exercise; the CLI init area owns environment-name normalization.
hud set writes ~/.hud/.env, a project .env overrides it, and the process environment overrides both; the next hud command sends the winning key. Service URL defaults and HUD_FILE_TRACKING_ENABLED are observed through Settings and a served environment's manifest instead of pydantic field metadata.
…ables Combined BashGrader, combinator, custom-grader and LLM-judge grades now run inside templates of an in-process environment, asserting the run's reward, the evaluation summary tree and the evaluate step's details. The judge answers through the scripted model fake, so the requests it sends (bearer, model, criterion type, query) are pinned. Authoring mistakes fail the grade with their message or warn. The text helpers stay as one table through the public functions.
make_request(_sync) and every PlatformClient verb now talk to the fake API: the /v2 prefix, repeated query keys, the bearer, doubling backoff on 502/503, a single attempt for 4xx, typed errors for unreachable, slow and unparseable responses, and the once-per-process deprecation warning. Through the CLI, hud models list pages the catalog and API errors become JSON error documents. test_platform.py and the exception-string tests fold into this scenario.
Each agent family runs a real rollout with a provider key, the HUD key alone, or a forced gateway, against the scripted gateway and a second fake standing in for the provider's own API; the test asserts which server answered and with which credential. Bedrock profiles reach the Bedrock endpoint signed with AWS credentials, unroutable models are refused when the agent is built, gateway requests carry their run's Trace-Id, and a nested rollout names its parent trace. The tests that patched hud.utils.gateway and asyncio.sleep are removed.
Runs come from real rollouts: a trainable model behind the gateway returns token ids, so forward_backward sends inline trajectories, and plain runs train by trace id. The scenario pins the request bodies, the catalog lookup made once with the client's own key, a single optim-step POST even on 502/503/504, the custom-loss backward gradients, the no-gradient warning, and that untrainable batches are refused before any request.
hud/patches/mcp_patches.py had no tests. A scripted streamable-HTTP MCP
server shows a capability call retrying 502/503/504 and then
succeeding, resolving with a -32000 transport error once
HUD_CLIENT_TIMEOUT passes, on unparseable JSON, and on a dropped
connection instead of hanging, and accepting structured output outside
the tool's schema. A real FastMCP server shows plain results wrapped as
{"result": ...} without output validation. Every row fails with the
patches disabled.
The select rows that patched questionary to inspect its choice list become keystroke rows; the smoke tests that only checked output methods do not raise are removed.
A lossy encoder finalizes on time with a full queue and a stalled callback, and a lossless submit stops waiting once its encoder thread has died. A wrapped two-env sim reports one trace per episode: done closes the slot's trace with env success outranking the shaped reward, skips the post-reset observation, and opens a fresh trace for the next episode.
settings.span_dir is typed str | None and only a test that patched an object() into settings ever reached the TypeError branch.
ty rejected reading __name__ off a Callable and a lambda returning the Event's bool where the segment callback returns None.
The row allowed one second for completion while the escaped child slept two, so it timed out under coverage with eight workers. The child now outlives a ten-second budget by twenty seconds, which still proves completion does not wait for it.
…ning body by what it carries
…th the loader and agents
…that protect files
…duplicate deprecation check go
In hud, harness means the agent side; the test package holds fakes and the hud runner.
tests/eval/envs.py and tests/agents/support.py move into tests/support (envs, run_task, mcp_server, image); the MCP-over-HTTP setup in connect and patches uses the shared helper; a dead subprocess provider and its source string go.
Tests seed platform state (tasksets, environments, models, jobs, imports, QA checks) and check what the command left behind, instead of each file routing its own replies and reading request bodies.
… fake The platform fake now requires the user's HUD key on its API, records hosted submissions and their cancellation, and serves a submitted trace's states to its polls.
…form fake The fake now models the training service (an optimizer step saves and promotes a checkpoint) and the set of keys it accepts, so key precedence is checked by whether the platform answers rather than by reading headers. One shared 'reporting' fixture turns telemetry on and drains uploads.
Drop the patch-engine test: nothing in the SDK imports that module, so no public boundary reaches it.
…, drop unused builders
On Python 3.11 a garbage collection can run while tuple(sys.modules.items()) builds its tuple, and a finalizer that imports changes the dict mid-copy, so loading an environment failed with 'dictionary changed size during iteration'. dict.copy() runs no Python code. 3.12 defers collection to the eval loop, which is why only the 3.11 lane failed.
…rule Each case is held to what holds for any dataset (findings by task, rows that match tasks.json, environments copied whole, Compose projects that resolve, no host secret written out) and gets a scenario for the rule it exercises.
…of a whole output Drop inline-snapshot, which no test uses, and show the platform fixture in the contributing guide.
Defaults compared to themselves, a copy of every package's __all__, container accessors, constructors that store their arguments, clients returning what a scripted peer just sent, ids passed in and read back, and validation rows that restate a schema. Each was a check a user or peer could never observe failing on its own.
… test Constructor and misuse-message checks, a test that reads the shipped seccomp JSON back, --json output that prints what the platform sent, a step recorded and read back, transcript and params snapshots, and rows that prove a rule a sibling row or test already proves.
sidecar-reachability-repeated is sidecar-reachability run twice on one placement, which verifier-restore covers; the default-credential fixtures repeat env-templates' default fallback; bad-agent-group is bad-agent-user's check; two reward-files cases repeat the non-finite and unparseable rules.
Nine fixtures copied the seven-file sidecar-reachability task to change one to three files. Each now names it as its base in expected.toml and holds only the files it changes; the scenario copies the base first and the fixture over it, so every assembled task is the same bytes as before.
artifact-symlink plants one symlinked artifact and lists only it, the shape artifact-ancestor-symlink already has; the base task's collect hooks would otherwise fail first. verifier-child-user's verifier image keeps only the user and the root USER its check depends on.
The fake answers from what a test seeded and records what the code sent. It no longer decides what a cancel stops, which checkpoint is served, how QA results settle, which names conflict, or what a search or filter returns; the tests that leaned on those now check the request the SDK sent, or script the reply.
Strengthen four assertions that were weaker than the behaviour their test names: the hosted submission carries the Compose document, a deploy dry run never prompts, an unknown Harbor taskset reports not_found, and a non-subscore in a combine is refused on its own.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds the test suite around what the SDK promises at its public boundaries (the
hudCLI, the exported API, wire protocols, files, process effects), checked against the real outside world wherever one can run. Tests run in CI; locally, ruff and ty are the loop.What the suite looks like now
The real thing where it can run. Each external system gets its own lane:
tests/test_agents.py,live).tests/fixtures/Dockerfile(docker).modal).LeRobotDataset(lerobot).Fakes only where nothing real can run, or to force a fault.
tests/harnessholds:There is no fake Docker, Modal, LeRobot or provider SDK.
Fixture environments (
tests/fixtures/envs/): small envs whose reward is 1 only if the whole path worked:echo,tools,servers(two MCP servers),workspace,picture,screen;sessions,verifier,secret;faulty,brokenandstalls.What tests assert: outcomes, not transcripts. Tests check:
--jsonfields that carry meaning.Rendered text, request bodies and argv are not snapshotted. Error mapping from HTTP status to error kind is tested once, in
tests/cli/test_shell.py; each command only lists its own refusals.Policy:
scripts/check_tests.pyruns in the CI lint job. It fails a test that patches ahudname, imports a private one, or touches a private attribute. The three files that predate the rule are listed inscripts/check_tests_baseline.json, and their counts may only fall.Lanes:
ci.yml(every PR, Python 3.11 and 3.12): everything not markede2e, underpytest -n auto.e2e.yml:docker;sandbox(as root with bubblewrap, under HUD's seccomp profile);live, with real models and the real Claude Code and Codex CLIs;modal,lerobotandcookbooks.pre-release.yml: the hosted template case (deploy, run on the HUD runtime, read the trace back).Numbers
Bugs the scenarios found, fixed here
Each fix is its own commit.
LocalRuntimeuntil its 120 s ready timeout.rollout_timeoutor Ctrl-C.GeminiConfig.excluded_predefined_functionsnever reached the request.ClaudeToolSearchTool.thresholddid nothing.create_agentrequested a catalog model by the name typed rather than its id.hud serveignored SIGTERM, sodocker stopskipped shutdown hooks.hud servedeadlocked on SIGTERM while a session was still grading.KeyErrorwas reported as an unknown task.tasks.startargument was accepted as empty..pysource was not named as missing.harbor export(a missing template, an unusable slug, two slugs naming one folder) left the folders of the tasks before it on disk.task.tomlcrashed adaptation of the whole dataset.includeandextendswere reported as invalid rather than unsupported, as documented.CDPErrornever named the failed command.hud evalcould exit 0 when a run ended without a valid grade.hud evalreportedmodel: nullwhen.hud_eval.tomlset the agent's model.hud trace getshowed no tool calls.PermissionError.RobotBridge.stop()could wait forever when a client hung up as it stopped.asyncio.wait_fordropped the cancellation, so the tick loop kept running.-0.0000.hud.agents.typesdoes not export.Changes to review
hud evalexits 1 when any run ends without a valid grade, and its JSON output marks those runs.hud.environment.server.serve()no longer installs a SIGTERM handler. The entry points (hud serve,python -m hud.environment.server) useserve_until_terminated.harbor.unsupported.compose_include_extends(kindcontract) replacesharbor.invalid.composeforincludeandextends.VNC_PORT(default 5900); its capability is unchanged.docs/v6/referenceanddocs/v6/experimentalnow match the code.Known defects, not fixed here
ConnectionRefusedErrorwithout the container's output. The Docker lane asserts only failure and removal until this is fixed.gemini-3.5-flash,gemini-3.8-flash). The agent declares the oldercomputerUsetool, but the models call the newer actions (click,take_screenshot,wait). The live lane's Gemini screen cell is red until this is fixed.ValueError,ValidationErrorincluded, tousage.RobotAgent(save=True), a camera whose frames are under 32 px hangs the rollout when the episode is saved; a 2 px frame errors instead. lerobot's AV1 encoder fails on frames that small, and the dataset is always written as video.RuntimeError: dictionary changed size during iteration(CI, Python 3.11, full suite). It has not reproduced since, and the run keeps only the message, not the traceback.hud harbor importlogs the 404 from its by-name taskset lookup before creating the taskset.Not validated here
Sandboxcheck is expected red.MODAL_TOKEN_IDandMODAL_TOKEN_SECRET. Daytona has no lane, because no account was available.