Skip to content

docs: document DeepSWE harness versions and release timing - #1877

Merged
ZeroPoint95 merged 3 commits into
devfrom
zeropoint95/deepswe-benchmark-versions
Oct 19, 2026
Merged

ZeroPoint95 merged 3 commits into
devfrom
zeropoint95/deepswe-benchmark-versions

Conversation

@ZeroPoint95

@ZeroPoint95 ZeroPoint95 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

The DeepSWE comparison published scores without identifying the harness releases, making it unclear whether its OpenCode result measured 1.x or 2.x.

  • Document the nine competing harness versions with recorded source revisions, public release dates and links; explicitly identify OpenCode 1.18.30 and explain the patch-release timing for Claude Code and DeepSeek Harness.
  • Identify senior-dev through its public integration commit dd0fcc654 (#1488) and first stable codeaf release v0.5.0. Explain that these establish public lineage for the agent measured before integration.
  • Add a five-column versions.csv beside arms.csv: harness, version, source revision, release publication timestamp and source URL. The senior-dev row uses the public integration/release references; the other rows identify the tested versions.
  • Link DeepSWE from the benchmark index, correct the campaign dates to September 10–12 UTC, and add the required change entry. Published scores are unchanged.

Fixes #1841.

How it was checked

  • Matched the nine competing version identifiers to 113 unique saved task records per arm; checked release-timing claims against the records and upstream release metadata.
  • Confirmed that integration commit dd0fcc654 is contained in v0.5.0 and absent from the preceding stable v0.4.1; checked the v0.5.0 release timestamp and merge reference against GitHub.
  • Checked the five-column CSV against all ten results rows and the documentation; validated links and scanned the changes for private references and local paths.
  • make pr-ready and git diff --check pass; no Go or module files changed.

@ZeroPoint95
ZeroPoint95 marked this pull request as ready for review October 19, 2026 18:21
@ZeroPoint95
ZeroPoint95 merged commit 26662f1 into dev Oct 19, 2026
6 checks passed
@ZeroPoint95
ZeroPoint95 deleted the zeropoint95/deepswe-benchmark-versions branch October 19, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document versions used for the DeepSWE comparision

1 participant