Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
69 lines (51 loc) · 5.17 KB

CONTRIBUTING.md

File metadata and controls

69 lines (51 loc) · 5.17 KB
Raw

Contributing

OpenChainBench is community-run. Anyone can submit a benchmark, fix a number, propose a methodology change. The web tutorial at /contribute walks through the steps; this file is the long-form reference. For a concrete end-to-end worked example, see docs/walkthrough.md.

What lives where

benchmarks/        Self-contained YAML spec per benchmark
harnesses/         Open-source code for each benchmark runner (you host yours)
infrastructure/    Shared services. currently just one Prometheus
src/               The Next.js site that renders specs
scripts/           Validate + dry-run tooling
docs/              Methodology, ADRs, style guide
.github/           CI, issue templates, PR template

OpenChainBench is a federation. Each harness is hosted by whoever wrote it. the project shares only one Prometheus that scrapes every harness's public /metrics URL. You never share API keys with the maintainers.

Where to start

Goal Right channel
Float a rough idea Discussions → Ideas
Ask a methodology / harness question Discussions → Q&A
Propose a benchmark formally Issue → 📊 Propose a benchmark
Report a number that looks wrong Issue → 🐞 Data quality
Submit a provider correction Issue → ✏️ Provider correction
See what's planned Roadmap project board

Submitting a benchmark

  1. Open an issue with the 📊 Propose a benchmark template. Sketch the metric, providers, methodology, where you'll host the harness. get feedback before you write code. The issue lands in the Requested column of the roadmap.
  2. Write the spec. Drop a YAML at benchmarks/<slug>.yml. The format is described in benchmarks/README.md and validated by src/lib/spec-schema.ts.
  3. Build the harness at harnesses/<slug>/. A harness is a data producer only. it exposes /metrics over HTTPS with the metric names and labels your spec references. See harnesses/README.md for the contract and the existing Go harnesses as reference implementations.
  4. Deploy the harness on whatever infra fits. Railway, Fly, Cloud Run, a VPS. Expose /metrics over HTTPS at a stable public URL. You own the runtime, the secrets and the budget. The maintainers never see your API keys.
  5. Append a scrape job to infrastructure/prometheus/prometheus.yml pointing at your public URL so the shared Prometheus picks up your harness. Format documented in infrastructure/README.md.
  6. Open a PR referencing the issue (Closes #N). CI runs pnpm validate (schema lint), pnpm typecheck, pnpm lint, and pnpm build. Once green and reviewed, a maintainer merges, redeploys the central Prometheus to apply the new scrape job, and rebuilds the materialization worker (its image bakes the spec list at build time). The site serves the new bench on the next worker sweep + ISR cycle (≤ 2 min).

Local development

pnpm install
pnpm validate            # schema-lint every spec in benchmarks/
pnpm spec:dry-run <slug> # query Prometheus and print numbers without rendering
pnpm dev                 # http://localhost:3000
pnpm build               # production build

Editorial conventions

  • No pre-determined winners. Specs do not mark a "winner". The leader on every page is computed at render time from the lowest p50.
  • Keep titles factual. Bridge. End-to-End Latency not The Fastest Bridge of 2026.
  • Tail before mean. Headlines use p50 and p99. The arithmetic mean is reported in the table but never used as a takeaway.
  • State the timeout. Failures are excluded from latency aggregates and counted toward success rate. Both numbers are reported.
  • Write methodology before findings. A spec without methodology is rejected.

Corrections

If you can't reproduce a number, file a provider correction (you measured a different value for your service) or a data quality issue (the site is showing something obviously wrong or stale). Material errors are corrected in place with a dated note on the report.

Code of conduct & security

  • Code of conduct. short, applies to all project spaces (issues, PRs, discussions).
  • Security policy. private vulnerability reporting via GitHub advisories.

License

Code: MIT. Reports & figures: CC-BY-4.0.