Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion environments/featured/browser-use/Dockerfile.hud
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ FROM mcr.microsoft.com/playwright/python:v1.63.0-noble@sha256:72bd171a9ffc2b4b59
WORKDIR /app
COPY requirements.txt /app/
COPY .hud-sdk/ .hud-sdk/
ARG HUD_SDK_VERSION=0.6.19
ARG HUD_SDK_VERSION=0.6.20
RUN python3 -m pip install --no-cache-dir --require-hashes -r requirements.txt \
&& python3 -c "from importlib.metadata import version; import os; assert version('hud') == os.environ['HUD_SDK_VERSION']; assert version('playwright') == '1.63.0'; print('HUD_ENV_SDK_VERSION=' + version('hud'))"
COPY --chmod=644 env.py tasks.py /app/
Expand Down
20 changes: 20 additions & 0 deletions environments/featured/browser-use/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Browser Use

A visual web-research environment backed by a real Chromium browser. Agents can
inspect screenshots, navigate, click, scroll, and use the browser's developer
protocol without leaving the allowed website.
Comment thread
solvemproblr marked this conversation as resolved.

The featured task asks the agent to inspect rental photos and count the chairs
at a specific kitchen table. Grading checks both the exact answer and whether
the agent cited screenshot evidence that actually showed the kitchen.

The environment exposes:

- `cdp`: direct browser control for compatible agents
- `mcp`: screenshot and browser-interaction tools

Deploy from this directory with:

```bash
hud deploy
```
2 changes: 1 addition & 1 deletion environments/featured/browser-use/requirements.in
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
hud[agents]==0.6.19
hud[agents]==0.6.20
playwright==1.63.0
uvicorn==0.54.0
8 changes: 4 additions & 4 deletions environments/featured/browser-use/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# This file was autogenerated by uv via the following command:
# uv --no-config pip compile requirements.in --generate-hashes --universal --python-version 3.12 -o requirements.txt --offline
# uv --no-config pip compile requirements.in --generate-hashes --universal --python-version 3.12 -o requirements.txt
aiofile==3.12.3 \
--hash=sha256:5c1bcc9e929c50834608e8cc1a4cc1d7503eb60c15a535b779fd39e2f372c017 \
--hash=sha256:caa6aa746b5e47e2165f7abd741b6415e49cf4d44fddc0f61844612cc3924d41
Expand Down Expand Up @@ -618,9 +618,9 @@ httpx2-jsfetch==1.0 ; sys_platform == 'emscripten' \
--hash=sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60 \
--hash=sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32
# via httpx2
hud==0.6.19 \
--hash=sha256:3288dc887ac4489383bb15ddcc10d3f59440b4af9fbfbed22c44598dffa00c94 \
--hash=sha256:f9c45d614bb2efac228d41f23f215a1ae874bbe87eb73777ac90d994af2519f7
hud==0.6.20 \
--hash=sha256:4ffb877da4e9fd7f0c397fb60874a997824c110d3e6903fae2595f35b81a66f9 \
--hash=sha256:7f5d084c12093dfe68b4e845cd428cf29208545d57085b6bb7c8c11437ec6802
# via -r requirements.in
idna==3.20 \
--hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \
Expand Down
2 changes: 1 addition & 1 deletion environments/featured/file-analysis/Dockerfile.hud
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ WORKDIR /hud
COPY pyproject.toml uv.lock env.py ./
COPY fixtures ./fixtures
COPY .hud-sdk/ .hud-sdk/
ARG HUD_SDK_VERSION=0.6.19
ARG HUD_SDK_VERSION=0.6.20
RUN UV_PROJECT_ENVIRONMENT=/usr/local/venv uv sync --frozen --no-install-project --no-dev && chmod -R a+rX /usr/local/venv && chmod -R go-rwx /hud
ENV PATH="/usr/local/venv/bin:${PATH}" WORKSPACE_DIR=/workspace
RUN python -c "from importlib.metadata import version; import os; assert version('hud') == os.environ['HUD_SDK_VERSION']; print('HUD SDK', version('hud'))"
Expand Down
21 changes: 21 additions & 0 deletions environments/featured/file-analysis/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# File Analysis

A document-analysis environment for tasks built from prompts, uploaded files,
and weighted grading criteria. Files are staged in an isolated workspace where
the agent can inspect them with shell tools and write a final report.

The featured task is a supplier-payment reconciliation. It combines invoices,
bank payments, receiving records, and vendor terms, then grades the resulting
report for correct calculations, evidence, and recommended follow-up.

The environment provides:

- an isolated shell workspace with network access disabled
- file tracking for the agent's report
- criteria-based grading with an LLM judge

Deploy from this directory with:

```bash
hud deploy
```
2 changes: 1 addition & 1 deletion environments/featured/file-analysis/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.1.0"
description = "Argument-hints environment: prompt, data-file attachments, and rubric grading via console form hints (HUD v6)"
requires-python = ">=3.12,<3.13"
dependencies = [
"hud[agents]==0.6.19",
"hud[agents]==0.6.20",
"httpx>=0.27.0",
]

Expand Down
8 changes: 4 additions & 4 deletions environments/featured/file-analysis/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion environments/featured/flask-coding/Dockerfile.hud
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ WORKDIR /hud
COPY pyproject.toml uv.lock env.py grader.py flask.bundle ./

COPY .hud-sdk/ .hud-sdk/
ARG HUD_SDK_VERSION=0.6.19
ARG HUD_SDK_VERSION=0.6.20
RUN UV_PROJECT_ENVIRONMENT=/usr/local/venv uv sync --frozen --no-install-project \
&& /usr/local/venv/bin/python -c "from importlib.metadata import version; import os; assert version('hud') == os.environ['HUD_SDK_VERSION']; print('HUD_SDK_VERSION=' + version('hud'))" \
&& mkdir /app \
Expand Down
20 changes: 20 additions & 0 deletions environments/featured/flask-coding/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Flask Coding

A coding environment that gives the agent an isolated Flask repository and
grades its changes with hidden tests. Each task starts from a clean git
baseline, exposes only the working tree to the agent, and applies the hidden
test patch after the agent finishes.

The featured task asks the agent to add binary-file loading support to
`flask.Config.from_file` while preserving the existing text and error-handling
behavior. The score comes from targeted fail-to-pass and regression tests.

The workspace blocks network access and runs the agent as an unprivileged user.
Existing runs remain reproducible because each task pins its repository
baseline and test contract.

Deploy from this directory with:

```bash
hud deploy
```
2 changes: 1 addition & 1 deletion environments/featured/flask-coding/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ requires-python = ">=3.11,<3.13"
dependencies = [
"blinker==1.5",
"click>=8.0",
"hud==0.6.19",
"hud==0.6.20",
"itsdangerous>=2.0",
"jinja2>=3.0",
"pytest>=8.0,<9",
Expand Down
8 changes: 4 additions & 4 deletions environments/featured/flask-coding/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion environments/featured/libbet/Dockerfile.hud
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ WORKDIR /app
# the lock, and the runtime modules the wheel build needs (pyproject `only-include`).
COPY pyproject.toml uv.lock env.py tasks.py emulator.py scoring.py games_loader.py ./
COPY .hud-sdk/ .hud-sdk/
ARG HUD_SDK_VERSION=0.6.19
ARG HUD_SDK_VERSION=0.6.20
RUN uv sync --frozen --no-dev
RUN /app/.venv/bin/python -c "import importlib.metadata; import os; version = importlib.metadata.version('hud'); print('HUD_SDK_VERSION=' + version); assert version == os.environ['HUD_SDK_VERSION']"

Expand Down
19 changes: 19 additions & 0 deletions environments/featured/libbet/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Libbet

A visual Game Boy environment built from the freely licensed homebrew puzzle
game *Libbet and the Magic Floor*. Agents see the current screen and play with
MCP tools for the directional pad, buttons, waiting, and screenshots.

The emulator advances only when a tool is called, so the game stays paused
while the agent reasons. The featured task gives the agent 40 action steps to
explore the game.

Scoring rewards visually distinct progress discovered during play rather than
requiring a particular level completion. Screenshots and scoring details are
recorded in the trace.

Deploy from this directory with:

```bash
hud deploy
```
2 changes: 1 addition & 1 deletion environments/featured/libbet/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.1.0"
description = "A HUD environment template that wraps VideoGameBench Game Boy games as scored, turn-based agent evals."
requires-python = ">=3.11,<3.13"
dependencies = [
"hud==0.6.19",
"hud==0.6.20",
"pyboy>=2.7.0", # Game Boy backend
"imagehash>=4.3.1",
"pillow>=10.0.0",
Expand Down
8 changes: 4 additions & 4 deletions environments/featured/libbet/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions environments/featured/robot-gripper/.dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,4 @@ examples/
docs/
run_vla.py
*.md
!README.md
2 changes: 1 addition & 1 deletion environments/featured/robot-gripper/Dockerfile.hud
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ WORKDIR /app
COPY pyproject.toml uv.lock ./
COPY wheels/ wheels/
COPY .hud-sdk/ .hud-sdk/
ARG HUD_SDK_VERSION=0.6.19
ARG HUD_SDK_VERSION=0.6.20
RUN uv sync --frozen --no-dev
RUN /app/.venv/bin/python -c "from importlib.metadata import version; import os; assert version('hud') == os.environ['HUD_SDK_VERSION']; print('HUD SDK:', version('hud'))"

Expand Down
19 changes: 19 additions & 0 deletions environments/featured/robot-gripper/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Robot Gripper

A tabletop robotics environment powered by a Newton physics simulation. Agents
control a gripper through MCP tools, inspect rendered views and simulator state,
and receive rewards from the resulting physical state.

The environment includes tasks for opening a drawer, lifting and moving an
object, and maintaining a force-controlled grasp. The featured task asks the
agent to pick up a mug within a 40-step budget.

Grading measures continuous progress as well as task completion, so partial
movement receives a proportional score while reaching the target earns the
success bonus.

Deploy from this directory with:

```bash
hud deploy
```
2 changes: 1 addition & 1 deletion environments/featured/robot-gripper/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ requires-python = ">=3.12,<3.13"
# Eval-machine dependencies. The Newton physics engine (`newton` + `gizmo`) is
# NOT on PyPI: it ships as a pre-built wheel in wheels/ (see [tool.uv.sources]).
dependencies = [
"hud==0.6.19", # HUD SDK: Environment, capabilities, hud eval
"hud==0.6.20", # HUD SDK: Environment, capabilities, hud eval
"mujoco>=3.2.0", # rendering, model name lookups, IK
"numpy>=1.26.0",
"Pillow>=10.0.0", # base64 PNG encode/decode for render()
Expand Down
8 changes: 4 additions & 4 deletions environments/featured/robot-gripper/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading