Skip to content

fix(benchmark): remove personal SWE-bench path defaults - #739

Draft
beruro wants to merge 1 commit into
developfrom
junyu/fix-benchmark-personal-paths
Draft

fix(benchmark): remove personal SWE-bench path defaults#739
beruro wants to merge 1 commit into
developfrom
junyu/fix-benchmark-personal-paths

Conversation

@beruro

@beruro beruro commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

src-tauri/src/benchmark/mod.rs hardcoded a developer's personal absolute path (/Users/laptop-h/Documents/GitHub/SWE-bench_Pro-os) as the production default for the SWE-bench Pro harness repository (evaluator script + per-task run scripts). On any other machine this default is guaranteed wrong, it leaks a personal directory layout into the codebase, and a missing harness only surfaced late as a file-not-found style failure instead of a clear "configure first" state.

Solution

  • Deleted the personal-path default constant entirely; there is no filesystem default anymore.
  • Extracted resolution into a pure function resolve_swe_bench_repo_path(explicit, env_value) in benchmark/paths.rs with explicit precedence: explicit value (app setting / command argument, reserved — no caller passes one yet) → ORGII_SWE_BENCH_PRO_REPO_PATH environment variable → typed SweBenchRepoNotConfigured error. Blank/whitespace values count as unset; resolved values are trimmed.
  • The existing ORGII_SWE_BENCH_PRO_REPO_PATH semantics are preserved (the E2E suite fixture in tests/e2e/wdio.conf.mjs keeps working unchanged).
  • swe_bench_evaluator_script_path() / swe_bench_run_scripts_dir() now return Result, so:
    • benchmark_preflight reports the unconfigured state as failing checks (evaluator_script, run_scripts_dir) whose detail is an actionable message naming the environment variable — the frontend preflight panel shows "configure first" up front;
    • benchmark_create_run_plan / benchmark_start_run in local_docker mode fail immediately at plan-build time with the same clear message instead of spawning an evaluator that dies on a nonexistent path.
  • swe_bench_command_preview now receives the already-resolved paths instead of re-resolving them internally.
  • Unit tests in benchmark/paths.rs cover the precedence contract: explicit wins over env, env fallback, blank-counts-as-unset, trimming, typed not-configured error, and that the error message names the env var.

Potential risks

  • Machines that relied on the old personal-path default silently "working" (only the original developer's machine) must now set ORGII_SWE_BENCH_PRO_REPO_PATH. This is intended behavior, not a regression.
  • There is no in-app settings UI for the harness path yet; the env var is the only configuration source. Follow-up: add an app-settings field and pass it through the reserved explicit parameter of resolve_swe_bench_repo_path.
  • Behavior change: in local_docker mode, run-plan creation now errors out when unconfigured instead of returning a plan with a failing preflight. Frontend surfaces the returned error string; no DTO shape changed.
  • Local compile verification did not complete on this machine (see Verification); CI must gate this PR.

Verification

  • rg -n "/Users/laptop-h|/Users/junyu" src-tauri/src/benchmark/ → empty (exit 1, no matches).
  • cargo fmt -p org2 --check → the only diffs reported are pre-existing unformatted files elsewhere in the crate (untouched by this PR); the four changed benchmark files are clean.
  • cargo check -p org2 --libNOT RUN to completion: full cold compile of the app crate repeatedly exceeded the available time budget on a machine already running several parallel builds. Relying on CI for compile + test gating — hence Draft.
  • cargo test -p org2 --lib (new benchmark::paths::tests) → NOT RUN for the same reason; the tests are pure-function assertions with no I/O.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant