Skip to content

docs: add JSON output format specification - #2303

Open
ReturnKartikey wants to merge 1 commit into
google:mainfrom
ReturnKartikey:doc-json-format
Open

ReturnKartikey wants to merge 1 commit into
google:mainfrom
ReturnKartikey:doc-json-format

Conversation

@ReturnKartikey

Copy link
Copy Markdown

This PR documents the JSON output format produced by --benchmark_format=json, as requested in #720.

The existing user_guide.md has an abbreviated example but lacked a complete field-by-field reference. This adds docs/json_format.md detailing:

  • All context fields (host, CPU, caches, load averages, schema version)
  • All benchmarks fields (iterations, time metrics, aggregates, Big-O, counters, labels)
  • Special floating-point handling (NaN / Infinity)
  • List mode (--benchmark_list_tests) format
  • A complete output example

Also adds a cross-reference link in docs/user_guide.md and updates docs/index.md.

Closes #720

Add docs/json_format.md with a field-by-field reference for the JSON
output produced by --benchmark_format=json. The existing user_guide.md
has a brief overview and one abbreviated example, but doesn't document
every field -- this makes it hard for tool authors and CI pipeline
maintainers to consume the output without reading json_reporter.cc
directly.

The new doc covers:
- Context object fields (CPU, caches, load_avg, custom context, etc.)
- Benchmark result fields (timing, iterations, threads, counters)
- Aggregate results (mean, median, stddev, cv)
- Big-O complexity and RMS fields
- Error/skip, memory metrics, labels
- Special float handling (NaN/Infinity)
- List mode (--benchmark_list_tests) output
- A full realistic example with iteration + aggregate results

Also adds a cross-reference from user_guide.md and an entry in
docs/index.md.

Closes google#720
@dmah42

dmah42 commented Sep 23, 2026

Copy link
Copy Markdown
Member

i think i've changed my mind about #720. keeping documentation in sync with the code is going to be a real pain. it's not that hard for folks to just read the code, or run the tests, to see the json output.

@LebedevRI

Copy link
Copy Markdown
Collaborator

i think i've changed my mind about #720. keeping documentation in sync with the code is going to be a real pain. it's not that hard for folks to just read the code, or run the tests, to see the json output.

Honestly, i was wrestling myself from posting the same yesterday...
I do think there needs to be a spec, but not a user-readable one.

@ReturnKartikey

Copy link
Copy Markdown
Author

Thanks @dmah42 and @LebedevRI for the feedback.... that makes total sense. Manual documentation markdown files inevitably suffer from drift as new metrics and fields are added to json_reporter.cc.

If you're open to it, I'd be happy to pivot this PR from a markdown document to a formal machine-readable JSON Schema (e.g. docs/benchmark_schema.json or tools/benchmark_schema.json) coupled with an automated test in CI:

  1. Automated validation in CI: A test (e.g. validating test JSON outputs or reporter test artifacts against the schema) would ensure the schema and json_reporter.cc never go out of sync. Any change to the output format without updating the schema would fail CI.
  2. Formalizing json_schema_version: Benchmark already emits "json_schema_version": 1 in context, so having an official schema formally specifies what version 1 covers.
  3. Tooling & consumers:** External tools, CI dashboards, and parsers can use the schema directly for automated validation and code generation.

All the field types and descriptions already compiled here can be embedded directly into the schema's description properties.

Would you be receptive to that direction? If so, I'll update the PR with the schema and test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document JSON format

3 participants