Conversation
- generate-report.py: heatmap, leaderboard, task difficulty, radar, artifact breakdown - generate-cost-quality-chart.py: cost vs quality scatter plot with Pareto frontier - Generated reports/ with all PNGs and CSVs from current judge data
Generic name — not tied to any specific judge LLM.
Clearer naming — eval.json is the industry standard for benchmark evaluation scores.
…n and metrics.json
- Skill detects non-interactive mode when all params are provided inline (repo, problem, model, tag, answers) — skips all confirmations - run-swe-benchmark.sh: automates all 5 tasks for a given model using claude -p with bypassPermissions, pre-populated answers from the prompts doc
- run-swe-claude-p.sh: runs /swe non-interactively via claude -p with pre-populated answers, works with Opus on Bedrock or any model via LiteLLM - run-swe-headless.py: direct API approach with context inline, no tool calls
Re-ran via non-interactive mode (28 turns, .34, 41K output tokens). All 4 artifacts regenerated with codebase exploration.
Refactor hello_inference.py to follow CLAUDE.md standards: - Make main() a thin orchestrator that delegates to private helpers - Add error handling with actionable messages for request failures - Configure logging and hoist defaults to module-level constants - Support CLI flags that override environment variables - Add Google-style docstrings and a --help epilog with examples - Use plain ASCII in output text Add mypy to the dev dependency group for type checking.
Refactor the Python benchmark scripts to follow CLAUDE.md standards: - Make main() a thin orchestrator that delegates to private helpers - Add modern PEP 604/585 type annotations throughout - Configure logging and route diagnostics through it, keeping print() only for genuine program output (dry-run prompt, metrics JSON, tables) - Catch specific request exceptions with actionable error messages - Pass encoding to all file I/O and add Google-style docstrings - Split oversized functions and remove unused imports Replace non-ASCII characters in shell script chrome with plain ASCII, leaving model-facing content strings intact.
Introduce the continuous DuckDB metrics collector and its control script, the concurrent benchmark client, and build_dashboard.py, which renders the collected metrics into a self-contained HTML dashboard. Total tokens are reported in millions (Mtok) with a prompt/generation breakdown. Document the workflow in the vLLM README and add unit tests for all three clients.
Delete self-hosted/ollama and relocate its SSH tunnel helper into the vLLM path (retargeted to vLLM's port 8000). Update the top-level and Bedrock READMEs, AGENTS.md, THIRD_PARTY, the vLLM setup skill, and the HumanEval runner comment so the self-hosted path refers only to vLLM.
Introduce the first piece of the non-interactive SWE benchmark: a versioned dataset format plus a typed loader that every downstream consumer (run harness, reviewer, report generators) can rely on for an enforced shape. - benchmarks/dataset/mcp-gateway-registry.yaml: metadata header plus five design tasks drawn from real upstream issues in mcp-gateway-registry. Each task carries a repo URL, pinned ref, complexity, tags, a problem statement and/or issue URL, pre-supplied clarifying answers, and optional reviewer-facing ground truth. The header documents the full schema. - benchmarks/scripts/dataset_loader.py: Pydantic models (Dataset, Task, GroundTruth) with schema-version gating, complexity enum checks, unique-id and problem-source validation, and default-ref resolution. Includes a CLI that validates and summarizes a dataset file. - benchmarks/tests/test_dataset_loader.py: unittest coverage for the loader happy path and error cases. - benchmarks/pyproject.toml + uv.lock: own venv for the benchmark harness so it can depend on pydantic, pyyaml, requests, matplotlib, and numpy without pulling in the model runtimes.
Introduce a runner config layer and rewrite the headless harness so it drives claude -p through the /swe skill for each dataset task, sourcing all run-time parameters from a YAML config with CLI overrides. - runner_config.py: validated Pydantic RunnerConfig (endpoint, model, claude flags); rejects bypassPermissions and non-http endpoints. - run-swe-headless.py: clone at pinned ref, build the /swe prompt, run claude -p, and record token/latency/turn metrics per task. Always pass --settings so routing overrides a global ~/.claude/settings.json (otherwise a global Bedrock setting silently wins and rejects the local model id with a 400). Synthesize inline settings with an apiKeyHelper from endpoint/api_key when no settings_file is set, and capture the error message and api_error_status on failed runs. - runner.example.yaml: documented config template. runner.yaml is gitignored so local endpoint/model choices stay uncommitted. - hello-world.yaml: trivial single-task sanity dataset. - SKILL.md: allow cloning the target repo into /tmp when the repo path is missing in non-interactive mode. - README: dataset, loader, runner config, setup, and routing guidance. - Tests for RunnerConfig and the harness helpers. - gitignore benchmarks/swe-benchmark-data run outputs.
Add a metrics file section showing a real example, explaining that it enables comparing the same task across models by cost, speed, and turn count, and noting a forthcoming eval skill will add artifact scoring to the same file.
Support smoke-testing a large dataset without editing it: --count N keeps the first N tasks in dataset order (after any --tasks filter), with 0 meaning all. Add tests and document the flag in the README.
Add a --concurrency flag (default 1, serial) to run-swe-headless.py so tasks can run N-at-a-time via a ThreadPoolExecutor, with each task cloning to its own temp dir and running claude -p as an independent subprocess. When concurrency exceeds 1 the vllm_prometheus block is flagged as a server-wide aggregate, since window-delta metrics no longer isolate a single run; Claude API usage fields remain per-run correct. Sample gauges (KV-cache usage, running/waiting requests) every 1s during each run via a background poller and surface peak/mean under gauges_sampled. Drop kv_cache_utilization_perc from metrics_that_matter, since a serial single-tenant benchmark barely varies it; the sampled peak/mean carries that signal instead. Emit a conspicuous per-task banner with task id and N-of-M progress, and add the concurrency field to runner_config with validation and README documentation.
Add drag-to-zoom on the time axis of every line chart: dragging selects a time window that narrows the x-domain and rescales the y-axis to the visible points, so latency spikes no longer flatten the baseline detail underneath. Double-click or a Reset zoom button restores the full range; each chart owns its zoom state, so small multiples zoom independently. Data marks are clipped to the plot rect. Split the Token throughput panel into per-series small multiples, since prompt and generation rates differ by ~100x and a shared axis hides the smaller series. Each panel keeps its identity color via a colorOffset option rather than using a dual y-axis. Stamp the served model slug into the output filename so dashboards for different models do not overwrite each other.
Regenerated task outputs (github-issue, lld, review, testing) for five mcp-gateway-registry tasks from a fresh qwen3.6-35b benchmark run.
Introduce a repo-aware LLM-as-judge that scores generated SWE artifacts against the actual source of truth: - judge_common.py: shared judge core (strict Pydantic schema, prompt rendering, result validation, atomic JSON writes) used by both backends. - codex_judge.py: agentic judge driven by `codex exec`. Reads repo/ref from the artifact folder's metrics.json (failing loudly when absent), clones that repo at that ref into a content-addressed, reusable checkout under /tmp, and grounds the review in the real code. Captures codex token usage and wall-clock latency from the --json event stream. Defaults to model openai.gpt-5.6-sol at reasoning effort high. - llm_as_judge.py: direct Bedrock Mantle judge, refactored onto the shared core. - Tests for both judges and the shared core. - README: document the rubric and the eval path through the codex judge. - .gitignore: ignore .hftoken.
Add a routing provider switch to the headless SWE harness so models can be driven natively on Amazon Bedrock (CLAUDE_CODE_USE_BEDROCK=1 with an AWS region and ambient credentials) alongside the existing base-URL endpoint path. The vLLM Prometheus block is omitted for Bedrock runs since Bedrock exposes no /metrics endpoint; per-run API metrics are still recorded. Runs now pass --add-dir for the clone path so the agent can inspect files outside the sandbox cwd, and the stream trace prints a truncated preview of tool_result content. Update the /swe skill to fan out independent work across concurrent subagents: the codebase analysis (Step 5) and the five expert-review personas (Step 7) now run in parallel to cut wall-clock time, while the artifact chain stays sequential. Refresh the remove-faiss Opus 4.8 artifacts.
Point the judge at a top-level directory with --recursive and it walks that directory for every subdirectory containing a metrics.json, treats each as an artifact folder, and scores it in turn. A folder that fails (missing repo/ref, a codex or clone failure, invalid scores) is logged and skipped so one bad folder never aborts the batch; --no-overwrite resumes a run by skipping folders that already have an eval.json. The single-folder path is unchanged. Document the batch mode in the README and cover discovery, skip-on-failure, and resume in the tests.
Reorganize benchmarks/README.md into a conceptual overview (the three model-hosting paths, a block and sequence diagram, and the dataset model) and move the operational mechanics into benchmarks/docs/: a shared harness reference plus one setup guide per path (Anthropic on Bedrock, open-weight on Bedrock via a LiteLLM proxy, and self-hosted vLLM on EC2). Add the LiteLLM mantle proxy so open-weight Bedrock models can be driven through Claude Code: - config/litellm-mantle.yaml maps mantle model ids to the OpenAI-compatible bedrock-mantle Chat Completions endpoint, which returns structured tool calls (the Converse path leaks native tool-call tokens as plain text). - scripts/bedrock-mantle-proxy.sh mints a Bedrock bearer token and runs the proxy on 127.0.0.1:4000 with start/stop/status/refresh controls. - Widen the runner allowlist so non-Claude models' shell idioms (cd <dir> && git ..., ls/cat/find) are permitted equally for every model. Fix the artifact-folder mismatch for provider=bedrock: claude --model needs the full inference-profile id (us.anthropic.claude-opus-4-8), but the /swe skill normalizes that to a slug (claude-opus-4-8) for its output folder, so the harness was counting artifacts in a folder the skill never wrote to and reporting a false INCOMPLETE. RunnerConfig now derives the same slug via model_to_slug() and uses it for both the prompt's model: field and the artifact directory, keeping the full id for claude --model and the metrics record. Add a --verbose flag that prints assistant text and tool results in full (and tool-call inputs) in the --stream trace instead of truncating them.
Regenerated github-issue.md, lld.md, review.md, and testing.md for the migrate-ecs-env-vars-to-secrets-manager, remove-efs-from-terraform-aws-ecs, replace-keycloak-db-password-with-rds-iam, and ssrf-hardening-outbound-url-validation tasks.
The bedrock/ folder's only benchmarking value -- the LiteLLM proxy and the
mantle model list -- was already folded into benchmarks/ (scripts/bedrock-mantle-proxy.sh
and config/litellm-mantle.yaml, verified to cover the same 38 models). What
remained there (the HumanEval benchmark, its published results, and the
interactive-alias docs) is no longer needed, so remove the folder entirely.
Rewrite the top-level README around what the repo actually is: a benchmark and
harness for measuring how well different models perform real-world coding tasks
with Claude Code, across the three model-hosting paths (Anthropic on Bedrock,
open-weight on Bedrock via the LiteLLM proxy, and self-hosted vLLM on EC2). It
carries the block and sequence diagrams, the three-paths table, the dataset
model, and the worked-example results, and links into benchmarks/docs/ for the
operational mechanics. Slim benchmarks/README.md to a short landing page that
defers to the root README and the per-path guides.
Drop the HumanEval evaluation section and JUDGE_RESULTS.md; the per-cell scores
still live in each {task}/{model}/eval.json. Update AGENTS.md, the bug-report
issue template, and .gitignore (removing the HumanEval-results exceptions and
the stale bedrock/ comment) to match.
Introduce a security-check skill that runs the Cipher security-engineer persona over the pending diff, reviewing changes against a catalog of real-world security anti-patterns and fixing any problems found. The persona and pattern catalog are adapted from the agentic-community/ mcp-gateway-registry project. Wire the skill into CLAUDE.md as a mandatory gate to run before any commit or PR and around every new enhancement. Remove the now-unused implement and summarize skills and update the README skills tree.
Strip the source project's concrete file paths and PR numbers from security-patterns.md so the generic patterns (#1-#11) read as reusable rules, with a clear provenance note and illustrative-not-prescriptive framing. Add a Repository-Specific Patterns section (R1-R6) grounded in this harness's actual surface: subprocess/agent execution safety, running coding agents over untrusted cloned repos, provider-token and secret handling across the three hosting paths, loopback-by-default binds, supply-chain trust in setup scripts, and secret/PII hygiene in committed artifacts and logs. Extend the review checklist to match. Retarget the Cipher persona (security-engineer.md) from the original OAuth/FastAPI scope to this benchmarking-and-self-hosting harness: new 'what this repository is' framing, evaluation areas, checklist, and review questions centered on R1-R6.
The existing lint.yml ran `bandit -r . -f txt`, which scanned test files (and, locally, .venv) and failed on expected patterns: asserts (B101) and /tmp paths (B108) in tests, and "password"-substring matches (B105) on chart-label strings. Scope the CI scan to source directories (benchmarks/scripts, self-hosted/vllm/clients, self-hosted/vllm/scripts) so test-only asserts and temp paths are not flagged, and add inline `# nosec` justifications to the four genuine source findings (two B105 chart labels, two B101 invariant asserts). Bandit now exits clean.
Add CI merge checks (test, uv-lock, pre-commit, weekly dependency update)
Group benchmark artifacts by model first, then repo, then task -- so each
model's full set of results lives together while the same {repo}/{task} across
sibling model folders still supports side-by-side comparison. Previously the
layout was {repo}/{task}/{model}.
Code:
- run-swe-headless.py: _artifact_dir builds <model-slug>/<repo>/<task>.
- preflight_check.py: _target_dirs mirrors the new order.
- judge_common.py: the folder-name fallback for task/model identifiers now
reads <model>/<repo>/<task> (leaf = task, grandparent = model); metrics.json
values still take precedence.
- run-e2e-benchmark.sh: artifact-path hints and the judge target use the
<model-slug>/<repo> subtree.
- The /swe skill (SKILL.md) writes to the new path and its examples/tree match.
Tests updated for the new order (test_run_swe_headless, test_preflight_check,
test_codex_judge). Docs updated: top-level README, harness-reference,
end-to-end run-book, and the swe-benchmark-data README (layout tree, clone
guidance, artifact paths).
Reorganize the committed Hello-World example into the new layout via git mv
(claude-opus-4-8, claude-sonnet-5, moonshotai.kimi-k2.5).
Reorder artifact layout to {model}/{repo}/{task}
For the vllm path the skill now brings the backing service up itself instead of
just checking it:
- Gate on a HuggingFace token before starting vLLM (checks HF_TOKEN or a
.hf_token file the serve script reads); stop and ask the user to provide one
if missing, noting that a bare .hftoken file is not picked up.
- If vLLM is already serving a different model, stop it first, then start the
requested model using the serve parameters from its
self-hosted/vllm/models/{model}.md guide at the largest context window the
guide endorses, logging to self-hosted/vllm/logs/.
- Start the DuckDB metrics collector before the run.
At the end of the run, stop the collector and archive its snapshot by renaming
vllm-metrics.duckdb to vllm-metrics_{model}_{scope}_{timestamp}.duckdb, leaving
a fresh database for the next run.
Also refresh the intro, workflow list, and notes to match, and fix the
artifact-path references to the {model}/{repo}/{task} layout.
Add the /benchmark skill as the recommended entry point for running a benchmark: - Top-level README "Get started" now leads with the /benchmark one-liner (with the run-e2e-benchmark.sh script as the headless alternative), and the skills tree lists /benchmark. - benchmarks/README "One-command end-to-end run" leads with the skill (noting it manages the vLLM server and DuckDB collector for the vllm path) and keeps the script as the headless option.
Long headless /swe tasks against a custom endpoint (vLLM or the LiteLLM proxy) grew their conversation until the server rejected the request with "maximum context length is N tokens", which the client then retried forever. Claude Code cannot detect the context window of a custom model served over a custom base URL, so its auto-compaction never fired. Tell it the true window via CLAUDE_CODE_AUTO_COMPACT_WINDOW so compaction runs before the request overflows: - runner_config: new context_window (default 0 = unset) and auto_compact_fraction (default 0.9) fields, plus an auto_compact_window property computing floor(window * fraction). - run-swe-headless: set CLAUDE_CODE_AUTO_COMPACT_WINDOW in both the process env and the --settings env block (the latter takes precedence); add a --context-window CLI override. - run-e2e-benchmark.sh: on the vllm path, read the live server's max_model_len from /v1/models and pass it as --context-window, so a run is always calibrated to the window the server actually booted with. Also make --stream --verbose the default for the harness invocation. - Docs: document the mechanism in the harness reference (new "Context window and auto-compaction" section), the config templates, and the qwen3.6-35b guide's benchmarking note. - Tests: cover the window computation, precedence, validation bounds, and the env/settings wiring (+11 tests). Leaving context_window at 0 keeps Bedrock and known Claude models on Claude Code's built-in window detection.
The harness cloned each task's repo into tempfile.mkdtemp(prefix="swe-"), producing a random parent like /tmp/swe-0xddi_2w/. Agents could not reliably reproduce that random suffix in tool calls -- they kept mistranscribing it (swe-0-ddi_2w, swe-0/ddi_2w, ...) and burned turns and context recovering, polluting the benchmark. Clone into <clone_dir>/swe-<task-id>/<repo-name> instead: a stable, transcribable name. Task ids are unique within a dataset, so serial and concurrent runs do not collide. - run-swe-headless: add _safe_task_slug (allowlist [A-Za-z0-9._-], strip leading dots/dashes) so the id is filesystem-safe and cannot traverse out of clone_dir; clear any leftover dir from a killed run before cloning. The existing finally block still removes the clone after each task. Update the --dry-run placeholder path to match. - run-e2e-benchmark.sh: register an EXIT trap that removes exactly this dataset's swe-<task-id> dirs as a backstop (never a broad swe-* glob, which would hit swe-judge-repos). - .gitignore: ignore swe-*/ clone dirs as a backstop for a config that points clone_dir inside the repo. - Tests: cover _safe_task_slug incl. path-traversal neutralization.
The _cleanup_clones EXIT trap's last statement was a `[[ -d "$dir" ]] && rm -rf` test. The harness already removes each clone in its own finally block, so by the time the trap runs the directories are gone, the -d test is false, and the && short-circuits to a non-zero status. As an EXIT trap, that status became the script's exit code -- so a fully successful run (all tasks OK, judge complete, success banner printed) still exited 1. Return 0 explicitly; the trap is a best-effort backstop and must not override the run's real outcome.
Add Step 5c to the benchmark skill: after scoring, gather each task's
metrics.json (turns, tokens, latency) and eval.json (task_score) and write
a RUN-SUMMARY.md into the run's {model-slug}/{scope}/ folder -- a results
table plus notes (did compaction fire, any failures and why). Applies to
all three paths. The report step (now 5d) points the user at it first, and
the skill is instructed not to present a partial run as a clean sweep.
The clone dirs were named swe-<task-id>, and the backstop gitignore rule swe-*/ therefore also matched the swe-benchmark-data output dir -- benign (that dir is already ignored) but imprecise and confusing. Rename the clone parent to swe-clone-<task-id> and tighten the ignore to swe-clone-*/, which cannot collide with swe-benchmark-data. Updated the clone path, the --dry-run placeholder, and the orchestrator's cleanup trap to match.
Calibrate auto-compaction, deterministic task clones, and auto run-summaries
Agentic coding tasks on real repos need 100K-250K input tokens per request, so a context window below 200K is unusable: the /swe prompt alone (~12K) plus the output reserve overflows a small window on turn 1, before auto-compaction can help (the first message already does not fit). Confirmed empirically with qwen3-coder-next on a g6e.12xlarge (4xL40S), which is VRAM-bound to a 16384 window: task 1 failed 0/4 with "maximum context length is 16384 tokens ... prompt contains at least 12289 input tokens". - benchmark skill: new Step 2d gate reads the served max_model_len after boot and STOPS the run if it is under 200000, telling the user to use a model that fits or a larger-VRAM node. Renumbered the collector step to 2e; updated the workflow overview. - Harness + orchestrator: add --max-output-tokens to lower the per-response output cap on the CLI (threaded through the same override path as --max-turns), so a small-window smoke test does not require editing runner.yaml. Documented in the orchestrator --help. - qwen3-coder-next guide: document that a >=200K window (agentic coding) needs a g6e.48xlarge (8xL40S) or larger; a g6e.12xlarge only fits ~16K and is not benchmarkable. Point to qwen3.6-35b / qwen3-coder-30b for the 4xL40S node. - Test: cover the --max-output-tokens override.
Gate the benchmark on a >=200K context window
The orchestrator only checked codex at judge time, so a missing codex failed the run after the entire (long) harness phase had already completed. Check both claude and codex up front in pre-flight and fail fast: claude is always required; codex is required unless --skip-judge (otherwise a warning). Also state loudly, in both the script and the benchmark skill, that both CLIs are expected to be wired to Amazon Bedrock on this machine: the codex judge always calls Bedrock for its scoring model, and on --provider bedrock claude does too. The script does not configure them -- if either is pointed elsewhere or unauthenticated, the run or scoring fails.
Preflight both claude and codex CLIs; state Bedrock-wiring expectation
When a candidate run failed to produce a required artifact (e.g. the model misread a design task and never wrote github-issue.md), the judge raised JudgeError and dropped the folder from the results -- so a genuine model failure silently vanished from the scores instead of counting against the model. Treat missing/empty required artifacts as what they are: a model failure, scored 0. Add shared helpers in judge_common.py (missing_artifacts, zero_score_result, identify_folder) and short-circuit both judge backends (codex_judge and llm_as_judge) before the clone/model call: write a schema-valid eval.json with all criteria 0, task_score 0.0, and a "MODEL FAILURE: ... did not produce the required artifact(s): ..." verdict, plus a scored_zero_missing_artifacts marker in the judge block. Mirrored into metrics.json like any other score, and still gated by --no-overwrite. Add a test asserting a folder missing one artifact scores 0 without cloning or calling the model.
Judge: score missing-artifact folders 0 instead of erroring out
Replace the prior 5x6 results matrix (whose per-model numbers are not reproducible from artifacts in this repo) with only the runs we have actually executed end to end: three open-weight Qwen models self-hosted via vLLM on a single g6e.12xlarge (4x L40S), scored by the codex judge (gpt-5.6-sol, high effort). - Results table now 5 tasks x 3 models with the verified eval.json scores: Qwen3.6-35B (mean 56.25) and Qwen3-Coder-30B (mean 34.15). - Qwen3-Coder-Next marked not viable on this node (16K window vs the >=200K these tasks need; needs a larger-VRAM node), with a footnote. - ssrf for Qwen3-Coder-30B shown as 0.0, a genuine model failure (never produced the design artifacts on either attempt), with a footnote. - Everything else (Path 1 Anthropic-on-Bedrock, Path 2 open-weight via the LiteLLM proxy) marked "coming soon" -- we publish only what we measured. - Note prominently that all three ran self-hosted on vLLM on g6e.12xlarge. The architecture diagram and path-comparison table (which models each path CAN host) are unchanged; only the results/leaderboard/analysis are.
Add the Kimi-K2.7-Code run to the self-hosted results. It is a 1.06T-param MoE that does not fit the g6e.12xlarge the Qwen models used, so it ran on 8x H200 (p5en.48xlarge); the table now carries a hardware column/row so the differing nodes are explicit rather than implied. - Results matrix gains a Kimi-K2.7-Code column: remove-faiss 75.25, remove-efs 71.25, ssrf 72.75, migrate-secrets 75.5, keycloak 0.0. - keycloak is a genuine model failure (hit the 60-turn cap with 2 of 4 artifacts); scored 0 by the judge's missing-artifact path, footnoted alongside Qwen3-Coder-30B's ssrf 0. - Kimi leads on tasks completed (73.69 over 4) and is a hair ahead of Qwen3.6-35B on the strict 5-task mean (58.95 vs 56.25); commentary and leaderboard updated, with hardware shown per model.
Serving open-weight models with vLLM on a p5en.48xlarge (8x H200, NVSwitch) requires several fixes the 4x L40S reference node never hits, because 8-GPU tensor parallelism over NVSwitch activates CUDA-JIT code paths (DeepGemm, FlashInfer allreduce, FP8 blockscale GEMM) at server startup. Add self-hosted/vllm/skills' companion note p5en-h200-cuda-fixes.md capturing: - NVIDIA driver alignment (595 kernel vs 610 userspace) + reboot - putting the venv/weights on the large NVMe (root disk is tiny) - Fix 1: ninja on PATH for DeepGemm/FlashInfer JIT - Fix 2: unversioned libcudart.so/libcuda.so for FlashInfer mnnvl allreduce - Fix 3: libnvrtc.so plus a populated CUDA_HOME/lib64 for GLM-5.2's FP8 kernel - a symptom to fix table and a per-model fix matrix Reference it from the vllm-setup SKILL.md (intro and serve step) and correct the GLM-5.2 model guide, whose CUDA_HOME/libcudart tuning notes assumed a /usr/local/cuda layout that does not exist on this DLAMI (nvcc is at /opt/pytorch/cuda) and omitted the libnvrtc requirement.
Document p5en.48xlarge (8x H200) vLLM CUDA fixes
README: publish self-hosted results (Kimi-K2.7-Code + 3 Qwen models)
Kimi-K2.7-Code completed 4 of 5 tasks at a 131,072-token (128K) window, which contradicts the skill's hard 200K stop. Change the gate from a hard stop below 200K to: >=200K proceed, somewhat-below (e.g. 128K) warn and ask the user to confirm, tiny (~16K) still not benchmarkable. Update the README hardware note (Kimi ran 128K on 8x H200) and the qwen3-coder-next guide to match. 200K stays the recommended floor, now evidence-based.
New scripts/plot_cost_quality.py renders a cost-vs-quality scatter with a Pareto frontier from the swe-benchmark-data artifacts: one point per model, mean estimated cost per task (x) vs mean task score (y), non-dominated models connected by a highlighted frontier. Light and dark themes use the dataviz palette (validated: dot<->accent CVD dE ~29, contrast >=3:1); models are identified by direct labels, not hue. Missing-artifact tasks count as 0 (matching the leaderboard), and a model with no scored tasks (e.g. qwen3-coder-next, not viable on this node) is excluded with a logged warning rather than plotted at $0/0%. Also correct the qwen3-coder-30b column: its migrate-secrets eval.json had a corrupted byte prefix and was re-judged to 36.25 (was 43.5; the LLM judge is non-deterministic across re-runs). Updated the README to the current on-disk value -- mean(5) 32.7, mean(completed) 40.88. matplotlib is already a declared dependency; generated PNGs are gitignored run outputs (regenerate with `uv run scripts/plot_cost_quality.py`).
Commit the rendered chart to a tracked docs/images/ path and embed it in the README's Results section, just below the results matrix. Point the plot script's default output at docs/images/cost-quality.png so re-running it refreshes the committed image the README shows (swe-benchmark-data is gitignored, so the prior default landed only in an untracked file). Also commit the dark-theme variant.
A task that scored 0 is a genuine model failure (missing artifacts) -- an unresolved anomaly, not a quality measurement -- so it should not drag the average. Exclude 0-score tasks from both the score and cost means in the chart and the README leaderboard, keep the per-task 0.0 cells visible, and note which task was excluded (pending investigation) on both. - plot_cost_quality.py: means are over non-zero-score tasks; excluded task names are tracked per model, the point label gets a `*`, and a footnote under the chart names them. - README: the results "Mean" row and the leaderboard now use the failed-task-excluded mean (Kimi 73.69 over 4, Qwen3.6-35B 56.25 over 5, Qwen3-Coder-30B 40.88 over 4), with a note explaining the exclusion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Brief description of what this PR changes and why.
Fixes # (issue number, if applicable)
Type of Change
Checklist
bash -nsyntax checkpython3 -m py_compile