fix(benchmark): remove personal SWE-bench path defaults - #739
Draft
beruro wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
src-tauri/src/benchmark/mod.rshardcoded a developer's personal absolute path (/Users/laptop-h/Documents/GitHub/SWE-bench_Pro-os) as the production default for the SWE-bench Pro harness repository (evaluator script + per-task run scripts). On any other machine this default is guaranteed wrong, it leaks a personal directory layout into the codebase, and a missing harness only surfaced late as a file-not-found style failure instead of a clear "configure first" state.Solution
resolve_swe_bench_repo_path(explicit, env_value)inbenchmark/paths.rswith explicit precedence: explicit value (app setting / command argument, reserved — no caller passes one yet) →ORGII_SWE_BENCH_PRO_REPO_PATHenvironment variable → typedSweBenchRepoNotConfigurederror. Blank/whitespace values count as unset; resolved values are trimmed.ORGII_SWE_BENCH_PRO_REPO_PATHsemantics are preserved (the E2E suite fixture intests/e2e/wdio.conf.mjskeeps working unchanged).swe_bench_evaluator_script_path()/swe_bench_run_scripts_dir()now returnResult, so:benchmark_preflightreports the unconfigured state as failing checks (evaluator_script,run_scripts_dir) whose detail is an actionable message naming the environment variable — the frontend preflight panel shows "configure first" up front;benchmark_create_run_plan/benchmark_start_runinlocal_dockermode fail immediately at plan-build time with the same clear message instead of spawning an evaluator that dies on a nonexistent path.swe_bench_command_previewnow receives the already-resolved paths instead of re-resolving them internally.benchmark/paths.rscover the precedence contract: explicit wins over env, env fallback, blank-counts-as-unset, trimming, typed not-configured error, and that the error message names the env var.Potential risks
ORGII_SWE_BENCH_PRO_REPO_PATH. This is intended behavior, not a regression.explicitparameter ofresolve_swe_bench_repo_path.local_dockermode, run-plan creation now errors out when unconfigured instead of returning a plan with a failing preflight. Frontend surfaces the returned error string; no DTO shape changed.Verification
rg -n "/Users/laptop-h|/Users/junyu" src-tauri/src/benchmark/→ empty (exit 1, no matches).cargo fmt -p org2 --check→ the only diffs reported are pre-existing unformatted files elsewhere in the crate (untouched by this PR); the four changed benchmark files are clean.cargo check -p org2 --lib→ NOT RUN to completion: full cold compile of the app crate repeatedly exceeded the available time budget on a machine already running several parallel builds. Relying on CI for compile + test gating — hence Draft.cargo test -p org2 --lib(newbenchmark::paths::tests) → NOT RUN for the same reason; the tests are pure-function assertions with no I/O.