diff --git a/CHANGELOG.md b/CHANGELOG.md index 43cd263..d76fe72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## 0.2.0a1 — 2026-10-21 + +- Subscription replay through installed Codex and Claude CLIs, with native login and no API fallback. +- Sequential trusted local workspaces, baseline/reference controls, and offline Docker verification. +- Frozen rules comparisons, durable attempt receipts, and subscription-aware cost labels. +- Explicit tool-profile and host-isolation limitations; errors stop further allowance use. + ## 0.1.0a1 — 2026-10-20 - Local failure capture, request selection and interactive Harbor task authoring. diff --git a/README.md b/README.md index c984d78..27e2c02 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ Local capture. No telemetry. Open source. Task authoring and model replay use yo **Python 3.11+ and Git.** Install the published alpha with [uv](https://docs.astral.sh/uv/): ```bash -uv tool install https://github.com/alexdevmotion/grudges/releases/download/v0.1.0a1/grudges-0.1.0a1-py3-none-any.whl +uv tool install https://github.com/alexdevmotion/grudges/releases/download/v0.2.0a1/grudges-0.2.0a1-py3-none-any.whl grudges --help ``` @@ -92,6 +92,21 @@ Building opens an interactive authoring session. **Review the reconstructed task Capture preserves your working tree, normal Git index and branches. Missing checkpoints are labeled approximate. To remove the managed hooks and unchanged managed skills: `grudges install --uninstall`. +## Replay with the subscription you already have + +Sign in with `codex login` or `claude auth login`, then run a reviewed task: + +```bash +grudges run path/to/task --runner local-codex --trust-local -k 1 +grudges run path/to/task --runner local-claude --trust-local -m sonnet -k 1 +# Compare the same task with your project rules: +grudges run path/to/task --runner local-codex --trust-local --rules ./AGENTS.md -k 3 +``` + +No API key required. Grudges invokes your installed CLI with its existing login; credentials stay with the CLI. Runs consume your plan's allowance, execute sequentially, and stop on runner errors. Docker checks the baseline and reference solution before the first model attempt, then grades each candidate offline. + +**Trusted local mode:** agents run on your host. Codex has a workspace shell; Claude uses restricted file-editing tools. Hidden tests are kept outside the working directory, but this is not a security boundary or a fair cross-agent benchmark. [Requirements and limitations](docs/usage.md#subscription-runners). + ## Replay the same task. Change one thing. Set `OPENAI_API_KEY` in your environment for Codex replay. Use model IDs available to your account; live runs incur provider charges. @@ -110,22 +125,22 @@ grudges run path/to/task -a codex \ -m YOUR_MODEL_ID --rules ./AGENTS.md -k 3 ``` -The two arms are **as captured** and **with rules**. Existing baseline rules are retained; the second copy replaces the root harness rules file. Inputs are frozen before execution. Native Codex and Claude Code replay use API keys, not subscription logins. See [supported rules layouts and behavior](docs/usage.md#try-a-pack). +The two arms are **as captured** and **with rules**. Existing baseline rules are retained; the second copy replaces the root harness rules file. Inputs are frozen before execution. The default Harbor runner uses API keys. Choose a local runner above to use your subscription. See [supported rules layouts and behavior](docs/usage.md#try-a-pack). ## Scores you can inspect | What you see | What it means | | --- | --- | -| Solved / scored | Successful attempts out of valid scored attempts; timeouts count as failures. | +| Solved / scored | Successful attempts out of valid scored attempts; Harbor agent timeouts count as failures. | | Errors & missing | Provider, grader and infrastructure errors, plus planned attempts with no receipt. | | Majority-fixed | Every planned attempt was scored and strictly more than half solved the task. | | Cost | Reported usage, with unavailable or partial totals labeled. | -Each run keeps `results.json`, the HTML card and Harbor receipts locally. A whole-task score doesn't tell you which requirement failed; inspect the verifier output. You can [set public titles and regenerate a card](docs/usage.md#read-results) before sharing it. +Each run keeps `results.json`, the HTML card and runner receipts locally. A whole-task score doesn't tell you which requirement failed; inspect the verifier output. You can [set public titles and regenerate a card](docs/usage.md#read-results) before sharing it. ## Built on tools that already work -Grudges handles capture, replay comparisons and reporting. [evals-from-sessions](https://github.com/ibragim-bad/evals-from-sessions) supplies the task-authoring workflow; [Harbor](https://github.com/harbor-framework/harbor) runs the tasks. +Grudges handles capture, replay comparisons and reporting. [evals-from-sessions](https://github.com/ibragim-bad/evals-from-sessions) supplies the task-authoring workflow; [Harbor](https://github.com/harbor-framework/harbor) runs API tasks; installed Codex and Claude CLIs run subscription tasks. The example packs include [eight explicit tasks](packs/starter/README.md) and [six intent tasks](packs/starter-intent/README.md). Their fairness notes distinguish stated requirements from inferred behavior. [Release checks](RELEASE-CHECKS.md) document what has actually been validated. diff --git a/RELEASE-CHECKS.md b/RELEASE-CHECKS.md index f69708d..b9f2c21 100644 --- a/RELEASE-CHECKS.md +++ b/RELEASE-CHECKS.md @@ -1,3 +1,20 @@ +# Local release validation — 0.2.0a1 + +21 October 2026. Linux; GitHub prerelease preparation. + +- Python 3.11 and 3.13: 78 tests passed, one optional external-receipt test skipped. +- All 14 bundled tasks passed local baseline/reference controls (28 offline Docker verifier runs): broken inputs fail and reference solutions pass through the actual local-runner mounts and UID. +- Live subscription replay passed for Codex CLI 0.163.0 (`gpt-6.1-sol`) and Claude Code (`sonnet`). Each completed a one-attempt, two-setup comparison on `starter/blur-formatting`. +- Both rules canaries were absent from baseline candidates and present only in the rules candidates; all four final task attempts solved. This verifies rule delivery and replay mechanics, not rule effectiveness or model quality. +- Restricted Claude did not automatically load root `CLAUDE.md` in direct canary tests. The runner now explicitly includes it in the prompt; unit and live tests cover that behavior. +- Review fixes include reward-file grading despite nonzero verifier exit, writable disposable verification copies, host UID permissions, paired setup ordering, subscription receipts, timing, interruption and conservative error handling. +- Desktop (1280px) and mobile (390px) cards checked in a browser; no page overflow at 390px. Subscription cost and trusted-host limitations remain visible. +- Release packaging and fresh installed-wheel checks cover both packs and CLI entry points. Source secret scan passed. Raw model transcripts and credentials are excluded from the release. + +Subscription tests consumed plan allowance. Local runners use different tool profiles and do not isolate the agent from the host; they are not equivalent to isolated Harbor evaluations. API replay remains available. Local subscription mode supports Linux and the documented source-task layout only. + +--- + # Local release validation — 0.1.0a1 20 October 2026. Linux only. Pre-publication validation record. See the GitHub release for publication status. diff --git a/RELEASING.md b/RELEASING.md index ecbf0e1..0601d07 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,12 +1,12 @@ # Release checklist -Target: `alexdevmotion/grudges`, version `0.1.0a1`. The owner authorized the initial GitHub release. Future releases require owner approval. GitHub publication does not authorize uploading to PyPI or sending launch messages. +GitHub releases require owner authorization. PyPI publication and launch messages require separate authorization. -1. Run unit tests, build wheel and sdist, and run `scripts/check_dist.py`. Install the wheel in a fresh environment outside the checkout; verify `--version`, capture and both bundled packs. -2. Run nop/oracle controls and inspect individual receipts. Check desktop and narrow-screen cards, including errors, missing attempts and more than six configurations. -3. Scan candidate files and the intended public Git history for secrets. The prepared source ZIP excludes Git history. Use it for a fresh public initial import after approval, preserving the original development repository privately. Do not push the old development history: it contains attribution trailers prohibited by the repository policy. Do not publish private research, sessions, receipts or the historical rules-trial card. -4. Review NOTICE, task fairness notes, README commands and platform limitations. Configure a private vulnerability-reporting channel and repository rules under the owner's account. -5. After approval, create/push the reviewed public history, tag the exact release commit, and attach the checked wheel/sdist with SHA-256 checksums to the release. PyPI publication requires separate account/name verification and authorization; no automatic publishing workflow is configured. -6. Test installation from the actual published artifact before announcing it. Record the public commit, artifact hashes and smoke-test results. +1. Run tests on Python 3.11 and 3.13. Build wheel and sdist; run `scripts/check_dist.py`. Install the wheel outside the checkout and verify the CLI and both bundled packs. +2. Check baseline/reference controls and individual rewards. For runner changes, test the affected live CLI, including a rules canary. Do not infer model quality from these smoke tests. +3. Check cards at desktop and mobile widths. Keep errors, missing attempts and unavailable costs visible. +4. Scan release files and the intended public Git history for secrets. Never include captures, raw sessions, credentials or private research. Preserve the original development checkout and its history. +5. Update the version, changelog, README install URL and `RELEASE-CHECKS.md`. Open a focused PR; require passing CI before merge. +6. Tag the exact merged commit. Build artifacts from that commit, attach wheel/sdist and SHA-256 checksums to a GitHub prerelease, then download and verify them in a fresh environment. -Do not claim general model rankings from the example pack or repaired-input results from historical rescoring. External alpha usage and repaired rules-trial calibration remain separate validation steps. +The public repository is `alexdevmotion/grudges`. The alpha is distributed through GitHub, not PyPI. Keep claims limited to the documented platforms, task layouts and evidence. diff --git a/SECURITY.md b/SECURITY.md index 3897dbe..831f2b7 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,4 +4,6 @@ Do not post captured sessions, secrets or exploit payloads in public issues. Rep Treat captures, Git objects, build workspaces and replay outputs as private. Deleting a working file does not delete its snapshot. Review task code and Dockerfiles before execution. Use provider API keys with appropriate limits; do not copy consumer login files into containers. Shareable task authoring and report rendering still require human review before publication. +Local subscription runners are for trusted tasks. The agent runs on the host; a fresh workspace does not isolate it from other host files, global instructions or configuration. Claude uses restricted file tools; Codex uses its native workspace sandbox. Offline Docker verification happens afterward. Keep native login files in their normal location; Grudges does not copy them into containers. + This alpha has no security support SLA. Only the latest alpha is intended for fixes. diff --git a/docs/usage.md b/docs/usage.md index 6bb565e..1923390 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -22,7 +22,28 @@ grudges run --pack starter -a codex -m YOUR_MODEL_ID -k 3 grudges run path/to/task -a codex -m YOUR_MODEL_ID --rules ./AGENTS.md ``` -Live runs cost money. Codex and Claude Code replay require provider API keys; subscription credentials are not forwarded. Each run freezes task copies and records its configuration before execution. Rules comparisons support the `environment/repo` plus `COPY repo/` Dockerfile layout and the Codex/Claude Code harnesses. Prebuilt `docker_image` tasks are rejected for rules comparisons because they would bypass the modified files. The comparison is **as captured** versus **with rules**: the baseline can already contain rules. The supplied text replaces the root harness rules file in the second copy; it does not remove other instruction files. +Harbor model runs incur API charges and require provider API keys. Subscription credentials are not forwarded into Harbor. Each run freezes task copies and records its configuration before execution. Rules comparisons support the `environment/repo` plus `COPY repo/` Dockerfile layout and the Codex/Claude Code harnesses. Prebuilt `docker_image` tasks are rejected for rules comparisons because they would bypass the modified files. The comparison is **as captured** versus **with rules**: the baseline can already contain rules. The supplied text replaces the root harness rules file in the second copy; it does not remove other instruction files. + +## Subscription runners + +Use `--runner local-codex` or `--runner local-claude` with `--trust-local`. They launch the installed official CLI on your host using its existing ChatGPT or Claude subscription login. API login is rejected, provider/key environment overrides are removed, and there is no API fallback. Grudges never reads, copies, or exports login tokens. Allowance limits still apply; cost is labeled unavailable, never $0. + +```bash +codex login # or: claude auth login +grudges run path/to/task --runner local-codex --trust-local -k 1 +grudges run path/to/task --runner local-claude --trust-local -m sonnet -k 1 +grudges run path/to/task --runner local-claude --trust-local --rules ./CLAUDE.md -k 3 +``` + +Requirements: Linux, Git, Docker, and a recent CLI. Codex must support `--ephemeral`; Claude must support `--restricted` and `--permission-prompts`. `-m` is optional and otherwise uses the CLI default; choose a model your plan supports. `--effort` forwards the CLI's effort setting. Local runs are sequential; `--agent` is for Harbor only. + +Supported tasks have `environment/repo`, an explicit `COPY repo/ /app/`, `WORKDIR /app`, `tests/test.sh`, and `solution/solve.sh`. Prebuilt images, symlinks and nested Git metadata are rejected. Baseline must fail and the reference solution must pass using the same verifier mount arrangement before any model attempt. Tasks requiring hidden build artifacts beneath `/app` may fail this gate. The verifier must work without network access and write a binary reward to `/logs/verifier/reward.txt`. + +Every attempt gets a fresh temporary source workspace. Candidates are preserved locally; verification runs in Docker with no network and a disposable writable copy. Task text referring to `/app` is mapped to the temporary working directory for the agent. Restricted Claude does not auto-load project instructions, so Grudges explicitly includes the root `CLAUDE.md` in its prompt for both baseline and rules setups; `@import` directives are not expanded and nested instruction discovery is not guaranteed. CLI versions, requested models, tool profiles and rules hashes are recorded. Claude's reported model usage is retained; Codex's requested model is not independently verified. + +Local mode is for **trusted tasks only**. Codex uses its workspace sandbox and shell; Claude uses restricted Read/Edit/Write/Glob/Grep tools and cannot run tests itself. Host tools, global instructions and CLI configuration may affect results. An agent can potentially discover verifier inputs elsewhere on the host. Do not treat these results as isolated benchmarks, compare the two tool profiles as equivalent, or use this mode for hostile tasks. Credentials remain in their normal location and are not mounted into Docker. + +Authentication, quota and incomplete-output errors stop the queue, preserve receipts, and leave remaining attempts missing. Local timeouts are unscored infrastructure errors because a timeout alone cannot distinguish model work from provider retries. No automatic retry spends additional allowance. ## Capture your own failure @@ -45,7 +66,7 @@ Task building fetches a pinned [evals-from-sessions](https://github.com/ibragim- Each run writes `results.json` and `card.html` under `~/.grudges/runs/`. `grudges card` renders the latest run, or pass a run directory. `--titles titles.json` supplies public task titles. -Cards show solved/scored attempts, errors and missing attempts. A task is majority-fixed only when all planned attempts are scored and strictly more than half solve it. Agent timeouts count as scored failures; provider, infrastructure and grader errors do not. Partial cost totals are labeled. A whole-task reward cannot identify which individual requirement failed. Inspect verifier receipts for that detail. CLI exit 2 signals setup/execution errors; a valid scored task failure returns 0. +Cards show solved/scored attempts, errors and missing attempts. A task is majority-fixed only when all planned attempts are scored and strictly more than half solve it. Harbor agent timeouts count as scored failures; local timeouts remain unscored; provider, infrastructure and grader errors do not. Partial cost totals are labeled. A whole-task reward cannot identify which individual requirement failed. Inspect verifier receipts for that detail. CLI exit 2 signals setup/execution errors; a valid scored task failure returns 0. ## Privacy and limits diff --git a/pyproject.toml b/pyproject.toml index 30f4cff..0ba402d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "grudges" -version = "0.1.0a1" +version = "0.2.0a1" description = "Turn coding-agent failures into replayable tests. Compare models and project rules." readme = "README.md" license = "Apache-2.0" diff --git a/src/grudges/card.py b/src/grudges/card.py index d186803..dde244e 100644 --- a/src/grudges/card.py +++ b/src/grudges/card.py @@ -27,7 +27,7 @@ def _tally(n_tasks, fixed, width=560, height=64): return f'' -HARNESS_NAMES = {"codex": "Codex", "claude-code": "Claude Code", "opencode": "OpenCode", "terminus-2": "Terminus"} +HARNESS_NAMES = {"local-codex": "local Codex", "local-claude": "local Claude Code", "codex": "Codex", "claude-code": "Claude Code", "opencode": "OpenCode", "terminus-2": "Terminus"} def _who(label): @@ -42,7 +42,9 @@ def _who(label): def _spend(s): parts = [] - if s.get("cost_usd") is not None: + if s.get("billing") == "subscription": + parts.append("subscription usage · USD cost unavailable") + elif s.get("cost_usd") is not None: parts.append(f"${s['cost_usd']:.2f}" + (" known cost (partial)" if s.get("cost_complete") is False else "")) else: parts.append("cost unavailable") @@ -118,6 +120,8 @@ def rank(label): subject = f"{_n(n_tasks, 'task')} my coding agent failed" footer = (f"{expected} planned attempts per task/setup. Errors excluded from scored; missing = planned but unreported. " "Majority-fixed = all planned attempts scored and more than half solved.") + if result.get("runner", "").startswith("local-"): + footer += " Trusted local CLI run; host configuration may affect results. Not equivalent to isolated Harbor runs." body = [] for i, t in enumerate(tasks + [None] * (n_tasks - len(tasks)), 1): cells = [] diff --git a/src/grudges/cli.py b/src/grudges/cli.py index e4a9775..6b37d4e 100644 --- a/src/grudges/cli.py +++ b/src/grudges/cli.py @@ -2,11 +2,12 @@ import json import os import shutil +import subprocess import sys from importlib.metadata import PackageNotFoundError, version from pathlib import Path -from . import build, card, hook, install, run, save, snapshot, store +from . import build, card, hook, install, local, run, save, snapshot, store def _default_claude_dirs(): @@ -54,7 +55,7 @@ def cmd_show(a): def cmd_run(a): - if not a.model and a.agent not in ("nop", "oracle"): + if a.runner == "harbor" and not a.model and a.agent not in ("nop", "oracle"): raise SystemExit("grudges: pass at least one -m MODEL") rules = a.rules.read_text() if a.rules else None tasks = list(a.tasks) @@ -62,14 +63,23 @@ def cmd_run(a): tasks.append(pack_path(a.pack)) if not tasks: raise SystemExit("grudges: provide task directories or --pack starter / starter-intent") - run_dir, result = run.replay(tasks, a.agent, a.model, attempts=a.attempts, effort=a.effort, - concurrency=a.concurrency, rules=rules) + if a.runner == "harbor": + run_dir, result = run.replay(tasks, a.agent, a.model, attempts=a.attempts, effort=a.effort, + concurrency=a.concurrency or 4, rules=rules) + else: + if a.concurrency not in (None, 1): + raise ValueError("subscription runners are sequential; use --concurrency 1") + if a.agent != "codex": + raise ValueError("--agent selects Harbor agents; choose --runner local-codex or local-claude") + run_dir, result = local.replay(tasks, a.runner, a.model, attempts=a.attempts, effort=a.effort, + rules=rules, trust_local=a.trust_local) for label, s in result["summary"].items(): errs = ", ".join(f"{n} {k}" for k, n in s["errors"].items()) or "no errors" print(f"{label}: fixed {s['fixed']}/{s['n_tasks']} · attempts {s['attempts_solved']}/{s['attempts_scored']} · {errs}") print(f"results: {run_dir / 'results.json'}") print(f"card: {card.write(run_dir)}") - return 2 if (result["harbor_exit_code"] or not result["records"] or + return 2 if (result.get("harbor_exit_code", 0) or result.get("interrupted") + or not result.get("complete", True) or not result["records"] or any(s["errors"] for s in result["summary"].values())) else 0 @@ -142,14 +152,16 @@ def main(argv=None): sh = sub.add_parser("show", help="print a grudge's metadata") sh.add_argument("id") - rn = sub.add_parser("run", help="replay tasks across models with Harbor") + rn = sub.add_parser("run", help="replay tasks with Harbor or a local subscription CLI") rn.add_argument("tasks", nargs="*", type=Path, help="task directories or pack directories") rn.add_argument("--pack", choices=["starter", "starter-intent"], help="run a bundled example pack") rn.add_argument("-a", "--agent", default="codex") - rn.add_argument("-m", "--model", action="append", default=[], help="repeatable; required except for nop/oracle") + rn.add_argument("-m", "--model", action="append", default=[], help="repeatable; local runners default to their CLI model") rn.add_argument("-k", "--attempts", type=positive, default=3) rn.add_argument("--effort") - rn.add_argument("-n", "--concurrency", type=positive, default=4, help="parallel trials (default 4)") + rn.add_argument("-n", "--concurrency", type=positive, default=None, help="parallel Harbor trials (default 4); local runners use 1") + rn.add_argument("--runner", choices=["harbor", "local-codex", "local-claude"], default="harbor", help="Harbor API replay or installed subscription CLI") + rn.add_argument("--trust-local", action="store_true", help="acknowledge local tasks run on your host; use reviewed tasks only") rn.add_argument("--rules", type=Path, help="compare tasks as captured against a copy with your project rules") b = sub.add_parser("build", help="turn a grudge into a Harbor task with your agent") @@ -172,7 +184,7 @@ def main(argv=None): return hook.main() try: return {"save": cmd_save, "requests": cmd_requests, "list": cmd_list, "show": cmd_show, "build": cmd_build, "run": cmd_run, "card": cmd_card, "install": cmd_install}[a.cmd](a) or 0 - except (OSError, ValueError, RuntimeError) as exc: + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as exc: print(f"grudges: {exc}", file=sys.stderr) return 2 diff --git a/src/grudges/local.py b/src/grudges/local.py new file mode 100644 index 0000000..cf12bf3 --- /dev/null +++ b/src/grudges/local.py @@ -0,0 +1,334 @@ +"""Trusted local CLI replay. Credentials stay with the installed agent. + +Agent workspaces are separate from verifier inputs, but not a host security +boundary. Docker grades the candidate offline after the CLI has exited. +""" +import hashlib +import json +import math +import os +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import time +import tomllib +import uuid +from pathlib import Path + +from . import run, store + +RUNNERS = {"local-codex": "codex", "local-claude": "claude"} + + +def environment(): + """Keep native login locations; remove API, provider and injected-token overrides.""" + allowed = {"HOME", "PATH", "USER", "LOGNAME", "SHELL", "LANG", "LC_ALL", "TERM", + "TMPDIR", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_RUNTIME_DIR", + "CODEX_HOME", "CLAUDE_CONFIG_DIR", "SSL_CERT_FILE", "SSL_CERT_DIR", + "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "http_proxy", "https_proxy", "no_proxy", + "NODE_EXTRA_CA_CERTS"} + return {k: v for k, v in os.environ.items() if k in allowed} + + + +def docker_environment(): + env = environment() + for key in ("DOCKER_HOST", "DOCKER_CONTEXT", "DOCKER_CONFIG", "DOCKER_CERT_PATH", "DOCKER_TLS_VERIFY"): + if key in os.environ: + env[key] = os.environ[key] + return env + + +def command(runner, model, effort=None): + if runner == "local-codex": + cmd = ["codex", "exec", "--json", "--ephemeral", "--sandbox", "workspace-write", + "-c", 'approval_policy="never"', "-c", 'forced_login_method="chatgpt"', + "-c", 'model_provider="openai"', "-c", 'sandbox_workspace_write.network_access=false'] + if effort: + cmd += ["-c", f"model_reasoning_effort={json.dumps(effort)}"] + if model: + cmd += ["-m", model] + return cmd + ["-"] + cmd = ["claude", "-p", "--output-format", "stream-json", "--verbose", "--restricted", + "--permission-mode", "acceptEdits", "--permission-prompts", "none", + "--tools", "Read,Edit,Write,Glob,Grep", "--strict-mcp-config", "--mcp-config", '{"mcpServers":{}}', + "--setting-sources", "", "--settings", '{"forceLoginMethod":"claudeai","disableAllHooks":true}', + "--no-session-persistence"] + if model: + cmd += ["--model", model] + if effort: + cmd += ["--effort", effort] + return cmd + + +def prompt_for(task, workspace, runner): + prompt = ("The task was exported from a Docker workspace. Any instruction referring to /app means " + "your current working directory here, not the host /app.\n\n") + if runner == "local-claude": + # Restricted Claude does not auto-load CLAUDE.md, even with project setting sources. + rules = workspace / "CLAUDE.md" + if rules.is_file(): + prompt += "Project instructions (root CLAUDE.md):\n" + rules.read_text() + "\n\n" + prompt += (task / "instruction.md").read_text() + prompt += "\n\nWork only in this repository. Implement the requested change. Hidden verification runs separately after you finish." + if runner == "local-claude": + prompt += "\nThis replay exposes file read/edit tools only; do not try to run shell commands." + return prompt + + +def authenticate(runner, env): + binary = RUNNERS[runner] + if not shutil.which(binary): + raise ValueError(f"install {binary} and sign in with your subscription first") + args = (["codex", "-c", 'forced_login_method="chatgpt"', "login", "status"] + if runner == "local-codex" else ["claude", "auth", "status", "--json"]) + p = subprocess.run(args, env=env, capture_output=True, text=True, timeout=30) + if runner == "local-codex": + valid = p.returncode == 0 and "Logged in using ChatGPT" in p.stdout + p.stderr + else: + try: + auth = json.loads(p.stdout) + valid = (p.returncode == 0 and auth.get("loggedIn") is True + and auth.get("authMethod") == "claude.ai" and auth.get("apiProvider") == "firstParty") + except (ValueError, AttributeError): + valid = False + if not valid: + raise ValueError(f"{binary} must be signed in with its subscription; no API fallback is allowed") + help_args = [binary, "exec", "--help"] if runner == "local-codex" else [binary, "--help"] + help_text = subprocess.run(help_args, env=env, capture_output=True, text=True, timeout=30).stdout + required = ("--ephemeral", "--json", "--sandbox") if runner == "local-codex" else ("--restricted", "--permission-prompts", "--no-session-persistence") + if not all(flag in help_text for flag in required): + raise ValueError(f"update {binary}: required local replay flags are unavailable") + return subprocess.run([binary, "--version"], env=env, capture_output=True, text=True, check=True, timeout=30).stdout.strip() + + +def validate_task(task): + cfg = tomllib.loads((task / "task.toml").read_text()) + dockerfile = task / "environment/Dockerfile" + if not dockerfile.is_file() or not (task / "environment/repo").is_dir(): + raise ValueError(f"{task.name}: local replay needs environment/repo and Dockerfile") + text = dockerfile.read_text() + if ((cfg.get("environment") or {}).get("docker_image") + or not re.search(r"(?im)^COPY\s+repo/\s+/app/?\s*$", text) + or re.findall(r"(?im)^WORKDIR\s+(.+?)\s*$", text) != ["/app"]): + raise ValueError(f"{task.name}: local replay supports source-built COPY repo/ /app/ and WORKDIR /app only") + for required in ("instruction.md", "tests/test.sh", "solution/solve.sh"): + if not (task / required).is_file(): + raise ValueError(f"{task.name}: missing {required}") + for p in task.rglob("*"): + if p.is_symlink() or p.name == ".git": + raise ValueError(f"{task.name}: materialize links and remove nested Git metadata before replay") + return cfg + + +def execute(cmd, cwd, env, log, timeout, prompt=None): + """Private logs; kill the whole process group on timeout or interruption.""" + with log.open("w") as out, log.with_suffix(".stderr").open("w") as err: + p = subprocess.Popen(cmd, cwd=cwd, env=env, stdin=subprocess.PIPE, + stdout=out, stderr=err, text=True, start_new_session=True) + try: + p.communicate(prompt, timeout=timeout) + except BaseException: + try: + os.killpg(p.pid, signal.SIGKILL) + except ProcessLookupError: + pass + p.wait() + raise + return p.returncode + + +def agent_result(runner, path): + if path.stat().st_size > 32 * 1024 * 1024: + raise ValueError("agent output exceeded 32 MiB") + raw = path.read_text() + if runner == "local-claude": + events = [json.loads(line) for line in raw.splitlines() if line.strip()] + init = next((e for e in events if e.get("type") == "system" and e.get("subtype") == "init"), {}) + if init.get("apiKeySource") != "none": + raise ValueError("Claude did not confirm native subscription authentication") + data = next((e for e in reversed(events) if e.get("type") == "result"), {}) + if data.get("is_error") or data.get("subtype") != "success": + return {"status": "agent_error"} + usage = data.get("usage") or {} + models = list((data.get("modelUsage") or {}).keys()) + return {"status": "completed", "resolved_model": ",".join(models) or None, + "permission_denials": len(data.get("permission_denials") or []), + "input_tokens": sum(usage.get(k, 0) for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")), + "cached_input_tokens": usage.get("cache_read_input_tokens"), "output_tokens": usage.get("output_tokens")} + events = [json.loads(line) for line in raw.splitlines() if line.strip()] + if any(e.get("type") in ("error", "turn.failed") for e in events): + return {"status": "agent_error"} + finished = [e for e in events if e.get("type") == "turn.completed"] + if not finished: + raise ValueError("Codex did not emit turn.completed") + usage = finished[-1].get("usage") or {} + return {"status": "completed", "input_tokens": usage.get("input_tokens"), + "cached_input_tokens": usage.get("cached_input_tokens"), "output_tokens": usage.get("output_tokens")} + + +def verify(task, workspace, image, trial, env, timeout, oracle=False): + logs = trial / "verifier" + logs.mkdir() + candidate = trial / "verifier-workspace" + shutil.copytree(workspace, candidate) + name = "grudges-" + uuid.uuid4().hex + cmd = ["docker", "run", "--rm", "--name", name, "--network", "none", "--cap-drop", "ALL", + "--security-opt", "no-new-privileges", "--pids-limit", "128", "--memory", "1g", "--cpus", "1", + "--user", f"{os.getuid()}:{os.getgid()}", + "--mount", f"type=bind,src={candidate},dst=/app", + "--mount", f"type=bind,src={task / 'tests'},dst=/tests,readonly", + "--mount", f"type=bind,src={logs},dst=/logs/verifier", "--workdir", "/app", + "--entrypoint", "/bin/bash", image, "/tests/test.sh"] + if oracle: + cmd[-1:] = ["-c", "bash /solution/solve.sh && bash /tests/test.sh"] + cmd[2:2] = ["--mount", f"type=bind,src={task / 'solution'},dst=/solution,readonly"] + try: + code = execute(cmd, trial, env, trial / "verifier.log", timeout) + reward_path = logs / "reward.txt" + (logs / "exit_code").write_text(str(code)) + if not reward_path.is_file(): + return "grader_error" + reward = float(reward_path.read_text().strip()) + if not math.isfinite(reward) or reward not in (0, 1): + return "grader_error" + return "solved" if reward == 1 else "failed" + except (ValueError, OSError, subprocess.TimeoutExpired): + return "grader_error" + finally: + cleanup(["docker", "rm", "-f", name], env) + + +def cleanup(cmd, env): + try: + subprocess.run(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=30) + except (OSError, subprocess.TimeoutExpired): + pass # Preserve the run receipt even when Docker is unavailable during cleanup. + + +def controls(task, image, trial, env, timeout): + """Check the exact mount/UID/verifier path before consuming subscription usage.""" + for label, expected in (("baseline", "failed"), ("oracle", "solved")): + folder = trial / label + folder.mkdir() + actual = verify(task, task / "environment/repo", image, folder, env, timeout, oracle=label == "oracle") + if actual != expected: + raise ValueError(f"{label} control returned {actual}, expected {expected}; inspect {folder}") + + +def replay(task_dirs, runner, models, attempts=3, effort=None, rules=None, trust_local=False): + if not trust_local: + raise ValueError("local runners execute on your host; review the tasks and pass --trust-local") + if not sys.platform.startswith("linux"): + raise ValueError("local runners currently support Linux hosts only") + if type(attempts) is not int or attempts < 1: + raise ValueError("attempts must be positive") + if runner not in RUNNERS: + raise ValueError("unknown local runner") + models = list(models) or [None] + if len(set(models)) != len(models) or any(m is not None and (not isinstance(m, str) or not m.strip() or m != m.strip()) for m in models): + raise ValueError("models must be nonempty and unique") + tasks = run.expand(task_dirs) + for task in tasks: + validate_task(task) + env = environment() + version = authenticate(runner, env) + docker_env = docker_environment() + subprocess.run(["docker", "info"], env=docker_env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True, timeout=30) + directory = Path(tempfile.mkdtemp(prefix=time.strftime("%Y%m%d-%H%M%S-local-"), dir=store.subdir("runs"))) + frozen = run.make_dataset(tasks, directory / "tasks") + setups = [("as captured", frozen)] + agent = "codex" if runner == "local-codex" else "claude-code" + if rules is not None: + ruled = directory / "tasks-with-rules" + for task in frozen.iterdir(): + run.with_rules(task, rules, agent, ruled / task.name) + setups.append(("with rules", ruled)) + records = [] + for task_name in sorted(t.name for t in frozen.iterdir()): + for model in models: + for attempt in range(attempts): + for setup, dataset in setups: + task = dataset / task_name + records.append(dict(task=run._task_name(task), trial=f"trial-{len(records)+1:04d}", + agent=runner, agent_version=version, model=model or "CLI default", setup=setup if rules is not None else None, + status="missing", seconds=None, input_tokens=None, output_tokens=None, cost_usd=None, billing="subscription", + task_dir=str(task), attempt=attempt+1)) + result = dict(run_id=directory.name, runner=runner, agent=runner, models=models, attempts=attempts, + rules=rules is not None, source=run.source_of(tasks), titles=run.titles_for(tasks), + tasks=[str(t) for t in tasks], records=records, + config=dict(runner=runner, agent_version=version, models=models, attempts=attempts, effort=effort, + tool_profile="workspace shell" if runner == "local-codex" else "file editing only", + concurrency=1, billing="subscription", isolation="trusted host workspace; offline Docker verifier", + rules_sha256=hashlib.sha256(rules.encode()).hexdigest() if rules is not None else None)) + store.write_json(directory / "config.json", result["config"]) + started = time.monotonic() + def persist(): + result.update(summary=run.summarize(records), complete=all(r["status"] in run.SCORED for r in records), + wall_seconds=round(time.monotonic()-started, 1)) + store.write_json(directory / "results.json", result) + persist() + images = {} + try: + for record in records: + task = Path(record["task_dir"]) + cfg = validate_task(task) + trial = directory / record["trial"] + trial.mkdir(mode=0o700) + tick = time.monotonic() + phase = "setup" + try: + if task not in images: + image = "grudges-local:" + uuid.uuid4().hex + images[task] = image + code = execute(["docker", "build", "-t", image, str(task / "environment")], trial, docker_env, + trial / "build.log", 600) + if code: + raise RuntimeError("Docker image build failed; see build.log") + controls(task, image, trial, docker_env, float(cfg.get("verifier", {}).get("timeout_sec", 120))) + record["setup_seconds"] = round(time.monotonic()-tick, 1) + tick = time.monotonic() + with tempfile.TemporaryDirectory(prefix="grudges-work-") as tmp: + workspace = Path(tmp) / "repo" + shutil.copytree(task / "environment/repo", workspace) + if execute(["git", "-c", "core.hooksPath=/dev/null", "init", "--template="], workspace, env, trial / "git.log", 30): + raise RuntimeError("git init failed; see git.log") + prompt = prompt_for(task, workspace, runner) + phase = "agent" + code = execute(command(runner, record["model"] if record["model"] != "CLI default" else None, effort), + workspace, env, trial / "agent.json", float(cfg.get("agent", {}).get("timeout_sec", 1200)), prompt) + if code: + record.update(status="agent_error", exception=f"CLI exit {code}") + else: + record.update(agent_result(runner, trial / "agent.json")) + if record["status"] == "completed": + if any(p.is_symlink() for p in workspace.rglob("*")): + raise ValueError("agent produced symlinks; refusing candidate export") + shutil.copytree(workspace, trial / "candidate", ignore=shutil.ignore_patterns(".git")) + phase = "verifier" + record["status"] = verify(task, trial / "candidate", images[task], trial, docker_env, + float(cfg.get("verifier", {}).get("timeout_sec", 120))) + except subprocess.TimeoutExpired: + record.update(status="infra_error", exception=f"{phase} timeout; completion unverified") + except (ValueError, OSError, RuntimeError) as exc: + record.update(status="infra_error", exception=str(exc)) + finally: + record["seconds"] = round(time.monotonic()-tick, 1) + persist() + print(f"{record['trial']}: {record['task']} · {record['status']}", flush=True) + if record["status"] not in run.SCORED: + break # Do not burn allowance after an authentication/quota/runner failure. + except KeyboardInterrupt: + result["interrupted"] = True + for record in records: + if record["status"] == "completed": + record.update(status="infra_error", exception="verification interrupted") + persist() + finally: + for image in images.values(): + cleanup(["docker", "image", "rm", "-f", image], docker_env) + persist() + return directory, result diff --git a/src/grudges/run.py b/src/grudges/run.py index 40600c0..7ef361b 100644 --- a/src/grudges/run.py +++ b/src/grudges/run.py @@ -121,6 +121,7 @@ def summarize(records): "median_seconds": _median([r["seconds"] for r in scored if r["seconds"] is not None]), "output_tokens": _sum_known(rs, "output_tokens"), "cost_usd": _sum_known(rs, "cost_usd"), + "billing": "subscription" if all(r.get("billing") == "subscription" for r in rs) else None, "cost_complete": all(r["cost_usd"] is not None for r in rs), } return out diff --git a/tests/test_cli.py b/tests/test_cli.py index 4082233..71ef10d 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -38,3 +38,11 @@ def test_error_results_return_nonzero_but_a_failed_task_is_valid(self): patch.object(cli.card, "write", return_value=Path("/tmp/card.html")), \ contextlib.redirect_stdout(io.StringIO()): self.assertEqual(cli.main(["run", "--pack", "starter", "-a", "nop"]), code) + + def test_interrupted_or_incomplete_local_run_returns_nonzero(self): + for fields in ({'interrupted': True}, {'complete': False}): + result = {'records': [{'status': 'missing'}], 'summary': {}, **fields} + with patch.object(cli.local, 'replay', return_value=(Path('/tmp/receipt'), result)), \ + patch.object(cli.card, 'write', return_value=Path('/tmp/card.html')), \ + contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(cli.main(['run','--pack','starter','--runner','local-claude','--trust-local']), 2) diff --git a/tests/test_local.py b/tests/test_local.py new file mode 100644 index 0000000..02c1e87 --- /dev/null +++ b/tests/test_local.py @@ -0,0 +1,207 @@ +import json +import os +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from grudges import local, run, card + + +class LocalTest(unittest.TestCase): + def task(self, root): + task = Path(root) / 'task' + (task / 'environment/repo').mkdir(parents=True) + (task / 'tests').mkdir() + (task / 'solution').mkdir() + (task / 'solution/solve.sh').write_text('true') + (task / 'task.toml').write_text('[task]\nname="test/task"\n') + (task / 'environment/Dockerfile').write_text('FROM python:3.13-slim\nWORKDIR /app\nCOPY repo/ /app/\n') + (task / 'environment/repo/app.py').write_text('broken = True\n') + (task / 'instruction.md').write_text('Fix app.py') + (task / 'tests/test.sh').write_text('exit 0') + return task + + def test_environment_preserves_login_location_not_api_credentials(self): + env = {'HOME':'/user','CODEX_HOME':'/config/codex','CLAUDE_CONFIG_DIR':'/config/claude', + 'OPENAI_API_KEY':'placeholder','ANTHROPIC_API_KEY':'placeholder', + 'CODEX_API_KEY':'placeholder','CLAUDE_CODE_OAUTH_TOKEN':'placeholder', + 'ANTHROPIC_BASE_URL':'https://invalid','CLAUDE_CODE_USE_BEDROCK':'1'} + with patch.dict(os.environ, env, clear=True): + self.assertEqual(local.environment(), {k:env[k] for k in ('HOME','CODEX_HOME','CLAUDE_CONFIG_DIR')}) + + def test_commands_keep_auth_native_and_restrict_permissions(self): + c = local.command('local-codex','model','high') + self.assertIn('forced_login_method="chatgpt"',c) + self.assertIn('workspace-write',c) + self.assertNotIn('--dangerously-bypass-approvals-and-sandbox',c) + c = local.command('local-claude','opus') + self.assertIn('--restricted',c) + self.assertNotIn('--bare',c) + self.assertNotIn('--dangerously-skip-permissions',c) + self.assertIn('Read,Edit,Write,Glob,Grep',c) + self.assertIn('claudeai',c[c.index('--settings')+1]) + + def test_requires_explicit_trust_before_any_process(self): + with patch.object(local.subprocess,'run') as proc: + with self.assertRaisesRegex(ValueError,'trust-local'): + local.replay([], 'local-codex', []) + proc.assert_not_called() + + def test_auth_rejects_api_login(self): + with patch.object(local.shutil,'which',return_value='/bin/agent'), patch.object(local.subprocess,'run') as proc: + proc.return_value = subprocess.CompletedProcess([],0,'', 'Logged in using an API key') + with self.assertRaisesRegex(ValueError,'subscription'): + local.authenticate('local-codex',{}) + proc.return_value = subprocess.CompletedProcess([],0,json.dumps({'loggedIn':True,'authMethod':'api_key','apiProvider':'firstParty'}),'') + with self.assertRaisesRegex(ValueError,'subscription'): + local.authenticate('local-claude',{}) + + def test_unsupported_layout_and_symlink_rejected(self): + with tempfile.TemporaryDirectory() as tmp: + t=self.task(tmp) + local.validate_task(t) + (t/'task.toml').write_text('[environment]\ndocker_image="prebuilt"') + with self.assertRaises(ValueError):local.validate_task(t) + (t/'task.toml').write_text('') + (t/'environment/repo/link').symlink_to('/tmp') + with self.assertRaises(ValueError):local.validate_task(t) + + def test_structured_output_requires_success_not_just_exit_zero(self): + with tempfile.TemporaryDirectory() as tmp: + p=Path(tmp)/'agent.json' + p.write_text('{"type":"turn.completed","usage":{"output_tokens":23}}\n') + self.assertEqual(local.agent_result('local-codex',p)['output_tokens'],23) + p.write_text('{"type":"turn.failed"}\n') + self.assertEqual(local.agent_result('local-codex',p)['status'],'agent_error') + p.write_text('{"type":"thread.started"}\n') + with self.assertRaises(ValueError):local.agent_result('local-codex',p) + p.write_text('{"type":"system","subtype":"init","apiKeySource":"none"}\n'+json.dumps({'type':'result','subtype':'success','is_error':False,'permission_denials':[{}]})) + self.assertEqual(local.agent_result('local-claude',p)['permission_denials'],1) + p.write_text('{"type":"system","subtype":"init","apiKeySource":"none"}\n'+json.dumps({'type':'result','subtype':'success','is_error':False,'modelUsage':{'claude-opus-test':{}}})) + self.assertEqual(local.agent_result('local-claude',p)['resolved_model'],'claude-opus-test') + + def test_claude_runtime_requires_subscription_receipt(self): + with tempfile.TemporaryDirectory() as tmp: + output = Path(tmp) / 'agent.json' + for source in ('user', 'env', None): + output.write_text(json.dumps({'type':'system','subtype':'init','apiKeySource':source}) + '\n' + + json.dumps({'type':'result','subtype':'success'})) + with self.assertRaisesRegex(ValueError, 'subscription'): + local.agent_result('local-claude', output) + + def test_controls_require_failure_then_success(self): + with tempfile.TemporaryDirectory() as tmp: + task = self.task(tmp) + for i, outcomes in enumerate((['failed', 'solved'], ['solved'], ['failed', 'grader_error'])): + trial = Path(tmp) / str(i) + trial.mkdir() + with patch.object(local, 'verify', side_effect=outcomes) as verify: + if i == 0: + local.controls(task, 'image', trial, {}, 30) + self.assertEqual([c.kwargs['oracle'] for c in verify.call_args_list], [False, True]) + else: + with self.assertRaisesRegex(ValueError, 'control returned'): + local.controls(task, 'image', trial, {}, 30) + + def test_claude_root_rules_are_explicit_even_with_settings_disabled(self): + with tempfile.TemporaryDirectory() as tmp: + task = self.task(tmp) + workspace = task / 'environment/repo' + baseline = local.prompt_for(task, workspace, 'local-claude') + self.assertNotIn('RULES_CANARY', baseline) + (workspace / 'CLAUDE.md').write_text('RULES_CANARY: create the marker') + prompt = local.prompt_for(task, workspace, 'local-claude') + self.assertIn('Project instructions (root CLAUDE.md):\nRULES_CANARY', prompt) + self.assertIn('Fix app.py', prompt) + self.assertIn('file read/edit tools only', prompt) + + def exercise(self, failure=None, runner="local-codex"): + tmp=tempfile.TemporaryDirectory();self.addCleanup(tmp.cleanup) + task=self.task(tmp.name) + phases=[] + def execute(cmd,cwd,env,log,timeout,prompt=None): + phases.append(cmd) + if cmd[0]=='docker': + if failure=='setup-timeout':raise subprocess.TimeoutExpired(cmd,timeout) + return 0 + if cmd[0]=='git':return 0 + self.assertFalse((cwd/'tests').exists()) + self.assertFalse((cwd/'solution').exists()) + if failure=='agent-timeout':raise subprocess.TimeoutExpired(cmd,timeout) + if failure == 'symlink': + (cwd/'escape').symlink_to('/tmp') + if runner == 'local-claude': + self.assertIn('file read/edit tools only', prompt) + log.write_text('{"type":"system","subtype":"init","apiKeySource":"none"}\n' + '{"type":"result","subtype":"success"}\n') + else: + log.write_text('{"type":"turn.completed"}\n') + return 1 if failure=='agent-error' else 0 + with patch.dict(os.environ,{'GRUDGES_HOME':str(Path(tmp.name)/'data')}), \ + patch.object(local,'authenticate',return_value='test-version'), \ + patch.object(local.subprocess,'run',return_value=subprocess.CompletedProcess([],0)), \ + patch.object(local,'execute',side_effect=execute), \ + patch.object(local,'controls',side_effect=ValueError('invalid controls') if failure == 'controls' else None), \ + patch.object(local,'verify',return_value='solved',side_effect=KeyboardInterrupt if failure == 'interrupt-verifier' else None): + directory,result=local.replay([task],runner,['test-model'],attempts=2,rules='A rule',trust_local=True) + self.assertEqual(json.loads((directory/'results.json').read_text()),result) + self.assertEqual((task/'environment/repo/app.py').read_text(),'broken = True\n') + self.assertFalse((task/'environment/repo/AGENTS.md').exists()) + return result + + def test_claude_replay_completes_both_setups(self): + self.assertTrue(self.exercise(runner='local-claude')['complete']) + + def test_invalid_controls_stop_before_agent_usage(self): + result = self.exercise('controls') + self.assertEqual([r['status'] for r in result['records']], ['infra_error','missing','missing','missing']) + self.assertTrue(all(r['output_tokens'] is None for r in result['records'])) + + def test_agent_symlink_is_not_exported(self): + result = self.exercise('symlink') + self.assertEqual(result['records'][0]['status'], 'infra_error') + self.assertIn('symlinks', result['records'][0]['exception']) + + def test_verifier_interrupt_preserves_error_receipt(self): + result = self.exercise('interrupt-verifier') + self.assertTrue(result['interrupted']) + self.assertEqual(result['records'][0]['status'], 'infra_error') + self.assertFalse(result['complete']) + + def test_both_setups_complete_and_subscription_reported(self): + r=self.exercise() + self.assertEqual(len(r['records']),4) + self.assertTrue(r['complete']) + self.assertEqual([x['setup'] for x in r['records']], ['as captured','with rules']*2) + self.assertTrue(all(s['billing']=='subscription' for s in r['summary'].values())) + self.assertIn('subscription usage',card.render(r)) + self.assertNotIn('$0.00',card.render(r)) + + def test_cli_errors_stop_queue_and_preserve_missing_attempts(self): + r=self.exercise('agent-error') + self.assertEqual([x['status'] for x in r['records']],['agent_error','missing','missing','missing']) + self.assertEqual(sum(s['fixed'] for s in r['summary'].values()),0) + + def test_setup_timeout_is_not_a_model_failure(self): + self.assertEqual(self.exercise('setup-timeout')['records'][0]['status'],'infra_error') + + def test_agent_timeout_stops_without_scoring_unknown_failure(self): + r=self.exercise('agent-timeout') + self.assertEqual([x['status'] for x in r['records']],['infra_error','missing','missing','missing']) + + def test_verifier_has_no_network_and_readonly_inputs(self): + with tempfile.TemporaryDirectory() as tmp: + t=self.task(tmp);trial=Path(tmp)/'trial';trial.mkdir() + def execute(cmd,cwd,env,log,timeout): + self.assertEqual(cmd[cmd.index('--network')+1], 'none') + self.assertEqual(cmd[cmd.index('--user')+1], f'{os.getuid()}:{os.getgid()}') + mounts=[cmd[i+1] for i,v in enumerate(cmd) if v=='--mount'] + self.assertTrue(all('readonly' in m for m in mounts if 'dst=/tests' in m)) + (trial/'verifier/reward.txt').write_text('1') + return 1 + with patch.object(local,'execute',side_effect=execute),patch.object(local.subprocess,'run'): + self.assertEqual(local.verify(t,t/'environment/repo','image',trial,{},30),'solved') + +if __name__=='__main__':unittest.main()