bench: 3-config SWE-bench harness (baseline/lsp/code_graph_mcp) - #693

Open

DvirDukhan wants to merge 55 commits into

dvirdukhan/bench-combined

Commits on Mar 14, 2026

Merge pull request #618 from FalkorDB/staging
gkorland
authored
4e3c11c

Commits on May 26, 2026

Add benchmark workstream scaffold + CONTEXT.md
Copilot
committed
9f00b8b
Consolidate grill decisions (Q1-Q11)
Copilot
committed
1e0048f
Upgrade graphrag-sdk 0.8 -> 1.1.1

DvirDukhan
and
Copilot
committed
84c56ac
Add benchmark metrics + report aggregation modules

DvirDukhan
and
Copilot
committed
aed0c56
Add code-graph HTTP adapter for the benchmark agent

DvirDukhan
and
Copilot
committed
c52f6e6
Add indexing-cache registry for benchmark runner

DvirDukhan
and
Copilot
committed
20ba1ed
Add LSP adapter (multilspy + jedi) with end-to-end shim tests

DvirDukhan
and
Copilot
committed
453dcfa
Add mini-swe-agent benchmark runner with dry-run mode

DvirDukhan
and
Copilot
committed
c6ef736
Add --real-run mode with synthetic smoke task and outcome verification

DvirDukhan
and
Copilot
committed
57a406d
Add SWE-bench Verified dataset loader and --swe-bench runner mode

DvirDukhan
and
Copilot
committed
020cf64
Add report CLI and SWE-bench Docker verify adapter

DvirDukhan
and
Copilot
committed
7088e23

Commits on May 27, 2026

Load .env in mini_runner; document Anthropic / Azure providers

DvirDukhan
and
Copilot
committed
dcf4ac0
bench: wire cg/lsp shims, pre-index code-graph track, sharpen preambles

DvirDukhan
and
Copilot
committed
a6b1b48
bench: add --limit flag for quick single-instance runs

DvirDukhan
and
Copilot
committed
03c7a73
bench: force tool usage in per-config instance template

DvirDukhan
and
Copilot
committed
13ac345
bench: fix find_symbol exact-match against nested properties.name

DvirDukhan
and
Copilot
committed
b0c9ce9
feat(mcp): impact_analysis tool — variable-depth Cypher (T6 #654)

DvirDukhan
and
Copilot
committed
c23a206
feat(mcp): GraphRAG ask tool — init module + prompt seam + tool (T9/T10/T11)

DvirDukhan
and
Copilot
committed
a3b3206
feat(mcp): auto-init — ensure FalkorDB + opt-in auto-index (T12 #660)

DvirDukhan
and
Copilot
committed
5e376e6
MCP-T13 + T14: cgraph init-agent + Docker mode switch

DvirDukhan
and
Copilot
committed
2a71e7f
MCP smoke harness + template fixes from end-to-end run

DvirDukhan
and
Copilot
committed
a18854b
bench: add MCP-transport sibling of the code_graph track

DvirDukhan
and
Copilot
committed
60e2bd1
bench: wire code_graph_mcp into mini_runner dispatch

DvirDukhan
and
Copilot
committed
f17d437
bench: fail loudly on indexing errors + bump analyze_folder timeout

DvirDukhan
and
Copilot
committed
b14432b
fix(analyzer): resolve LSP CALLS edges on repos without a venv

dvirdukhan
and
Copilot
committed
532d849
bench: add resume support + ignore sympy rubi rules

dvirdukhan
and
Copilot
committed
476bc73

Commits on May 28, 2026

fix(analyzer): defensive skip when second_pass references untracked file

DvirDukhan
and
Copilot
committed
d23ef79
feat(analyzers): tree-sitter Python symbol resolver (T18 #689)

DvirDukhan
and
Copilot
committed
3e8935f
perf(analyzers): memoise compiled tree-sitter queries

DvirDukhan
and
Copilot
committed
612b04f
bench: add start-api.sh helper enabling tree-sitter fast resolver

DvirDukhan
and
Copilot
committed
ec7fac6
Merge dvirdukhan/mcp-smoke-combined into bench-combined for calibration
DvirDukhan
committed
f9e8156
Merge bench harness with T18 + query-cache stack
DvirDukhan
committed
b264700
fix(mcp): lazy-import KnowledgeGraph so server starts on graphrag 1.x
DvirDukhan
committed
5e6c63c
fix(bench): silence cgraph-mcp stderr so DEBUG logs don't bloat agent context

DvirDukhan
and
Copilot
committed
38d2411
fix(bench): bump default cgraph-mcp timeout 60s → 300s for large repos

DvirDukhan
and
Copilot
committed
bbb5d95
feat(bench): tool-availability precheck + per-trajectory tool-usage rate

DvirDukhan
and
Copilot
committed
aa850d6
fix(bench): defensive stdin redirect + anti-fallback preamble rules

DvirDukhan
and
Copilot
committed
4a6956e
fix(bench): grade via official swebench Docker harness; deprecate broken pytest verifier

DvirDukhan
and
Copilot
committed
4daad7e
chore(bench): gitignore swebench harness output (regrade reports + logs)

DvirDukhan
and
Copilot
committed
bfdf60d
bench: add fallback_rate metric (passive grep/find tracking)

DvirDukhan
and
Copilot
committed
7ab59f4
bench: add median wall-clock column to report

DvirDukhan
and
Copilot
committed
e5f5631
bench: capture one-time indexing wall-clock per task

DvirDukhan
and
Copilot
committed
8dd3055
bench: robust indexing precheck (GRAPH.LIST + bounded timeout)

DvirDukhan
and
Copilot
committed
e82c05c
fix(bench): add /api/_health probe + harness sanity check for tree-sitter resolver

DvirDukhan
and
Copilot
committed
6508e3e
fix(bench): MCP adapter defaults CODE_GRAPH_PY_RESOLVER=tree_sitter + raises timeout

DvirDukhan
and
Copilot
committed
4c46736
perf(bench): compact cg/cg-mcp output + trim system preambles

DvirDukhan
and
Copilot
committed
dc8534e
fix(bench): restore submission sentinel in cg/cg-mcp preambles

DvirDukhan
and
Copilot
committed
805d0ad

Commits on May 29, 2026

fix(bench): SIGKILL whole pgid on timeout to stop orphan leak

DvirDukhan
and
Copilot
committed
4758ea1
fix(bench/mcp): collect ALL TextContent chunks, prefer structuredContent

DvirDukhan
and
Copilot
committed
3125946
perf(bench/mcp): cap impact_analysis at --limit 50, strip worktree path prefix

DvirDukhan
and
Copilot
committed
403f958

Commits on Jun 8, 2026

Merge origin/staging into bench-combined; resolve conflicts

DvirDukhan
and
Copilot
committed
5db867a

Commits on Jun 9, 2026