[v0.5.0] benchmark — public token/correctness/latency harness vs docs MCPs (human-led)

[ayhammouda]

Context

• Roadmap: STRATEGIC-ROADMAP-2026-05-29.md §4 (v0.5.0 — "Public benchmark harness"), §1.1 (success signal: "Token / correctness benchmark cited as canonical for Python docs MCPs"), §9.1 (classified human-led: methodology + corpus selection), and Amendment 2026-06-01 decision 5.17 (evidence ladder — no public comparative claim until this harness has data).
• Token methodology must match the internal token study (Study A): Claude tokenizer, measured after client-side rewrap (decision 5.8; ADR-006; future docs/architecture/TOKEN-STUDY.md).
• Market reference: competitive-brief.docx (private).

Goal

Ship a reproducible public benchmark comparing python-docs-mcp-server against all eligible docs MCPs and a no-MCP baseline on a 50-question Python eval, reporting correctness, tokens, and latency, with mandatory methodology disclosure.

Scope

Competitor matrix (finalize at execution; eligibility = exposes Python stdlib docs retrieval):

• Context7
• GitMCP
• DeepWiki
• Ref.tools
• no-MCP baseline (model answering from parametric knowledge alone)

Eval design:

• 50 questions across: exact symbols, concepts, cross-version behavior, PEP-adjacent.
• Lead with compare_versions — cross-version stdlib diffs competitors structurally cannot answer cleanly. This is the differentiator the launch post leads on.

Metrics:

• Correctness — rubric-scored against official docs.
• Tokens — Claude tokenizer, measured after client-side rewrap (consistent with Study A / decision 5.8).
• Latency — wall-clock per query.

Reproducibility & honesty:

• Runnable from a clean clone; competitor versions pinned.
• Methodology disclosure mandatory.
• No comparative/benchmark claim enters README / PyPI / launch copy until this harness produces data (decision 5.17). No "we benchmarked it, trust me."

Out of scope

• Any public comparative claim before data exists.
• Benchmarking non-Python-docs MCPs.
• Shipping in v0.3.x or v0.4.0 — this is a v0.5.0 artifact.

Execution note (human-led)

Per roadmap §9.1 this is not an agent-ready issue: methodology and corpus/question selection are a maintainer (Vision) judgment call. An autonomous agent may later scaffold harness plumbing (runners, token counting, result tables) under a separate, tightly-scoped issue once the methodology is fixed — but do not apply the agent-ready label to this issue.

---

Milestone: v0.5.0 (target ~12 weeks after v0.3.0, per roadmap §4).

[ayhammouda]

Follow-up benchmark execution plan created after the methodology PR (#70):

• [v0.5.0] benchmark corpus — define schema and 50-question eval pack #71 — corpus schema + 50-question eval pack (human-led; not agent-ready yet)
• [v0.5.0] benchmark runner — add reproducible CLI and artifact layout #72 — reproducible benchmark runner CLI + artifact layout (agent-ready)
• [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #73 — OpenAI/Google model matrix + mocked provider adapter contracts (agent-ready for plumbing only; live runs stay Vision-controlled)
• [v0.5.0] benchmark reporting — generate raw report and README-safe summary #74 — report generator + README-safe summary gate (agent-ready)

This keeps #63 as the umbrella issue. No README/PyPI/launch comparative claim should ship until these produce reproducible raw results.

[ayhammouda]

Credibility review follow-up integrated into the child issues:

• [v0.5.0] benchmark runner — add reproducible CLI and artifact layout #72 now requires query-level tool failures, timeouts, and MCP protocol crashes to be recorded as errors and scored 0.0; the denominator stays fixed at 50 for every eligible tool/model cell.
• [v0.5.0] benchmark adapters — define OpenAI/Google model matrix #73 now requires the full tool x model matrix, forbids averaging tool scores across model families, and labels methodology token counts as Claude Tokens (Normalized Payload) so they are not confused with provider billing tokens.
• [v0.5.0] benchmark reporting — generate raw report and README-safe summary #74 now requires README-safe summaries to use strict tool + model pairings and include an error/timeout rate column.

Publication principle: never average away the context. Robustness is part of the benchmark, not a footnote.