Evaluating tool-free reasoning over protein structures.
pdbthink is a reproducible, single-turn benchmark that measures how well language models parse and reason about protein structures when they are given nothing but the coordinates: no tools, no internet, no precomputed annotation. Each item shows a model one or two sanitised, randomly rotated structures in minimal PDB format and asks a question whose answer is recomputed from exactly those coordinates. Grading is deterministic — there is no LLM judge anywhere in the primary score.
The benchmark separates capabilities that are usually tangled together:
| level | what it isolates | families |
|---|---|---|
| parsing | reading a fixed-column coordinate file | P01–P03 |
| geometry | elementary 3D arithmetic | G01–G04 |
| local structure | turning geometry into structural-biology concepts | S01–S09 |
| interface & networks | reasoning across chains and contact graphs | I01, N01 |
| two-state | comparing two conformations | T01 |
| mechanism | connecting a local change to a functional consequence | 6 curated episodes |
pip install -e ".[dev,tokenizer]"structural-reasoning acquire --config configs/dataset_smoke.yamlstructural-reasoning build --config configs/dataset_smoke.yaml --output datasets/smokestructural-reasoning validate --dataset datasets/smoke --config configs/dataset_smoke.yamlEverything after acquire runs offline.
The build step is not optional: gold answers are deliberately not committed,
so that they cannot end up in a training corpus. build regenerates them
byte-identically and validate checks them against the committed hashes. See
docs/contamination.md.
The mock-gold provider answers every prompt with its own gold label, so a run
must score exactly 1.0. This is the no-model validation step of the
cost-controlled workflow and the fastest way to confirm an environment is sane.
structural-reasoning evaluate --dataset datasets/smoke --model-config configs/models/mock_gold.yaml --output runs/mockstructural-reasoning score --dataset datasets/smoke --responses runs/mock --output scores/mockServe a local model with any OpenAI-compatible server (vLLM, llama.cpp, …):
vllm serve Qwen/Qwen3-8B --port 8000structural-reasoning evaluate --dataset datasets/smoke --model-config configs/models/local_vllm.yaml --output runs/qwen3-8b --resumestructural-reasoning score --dataset datasets/smoke --responses runs/qwen3-8b --output scores/qwen3-8bstructural-reasoning report --scores scores/qwen3-8b --output reports/qwen3-8b--resume reuses every successful completion already recorded in the run. The
shared response cache also avoids repeat calls when prompts recur across runs. API
models use the same commands with
configs/models/anthropic_opus.yaml or configs/models/openai_gpt.yaml.
Hosted gateways use the same native runner. For example, the checked-in OpenRouter configuration can make a one-request, free-model transport check before a larger run. See docs/api-providers.md for key setup, model switching, what is sent to the provider and how truncation is recorded.
structural-reasoning review --dataset data/datasets/candidates_v1 --decisions data/review_decisions/v1.jsonlThe review interface opens at http://127.0.0.1:8787; to let colleagues in, see
docs/deployment.md. Each candidate shows an
interactive 3D view with the queried residues, gold answer and evidence
highlighted, the exact model-visible prompt, the continuous measurements and
ambiguity margins behind the label, and — curator-only — the source entry,
release date, method and file hashes. a accepts, r rejects (a reason is
required), j/k move through the list. Decisions append to a JSONL file that
the builder consumes:
structural-reasoning build --config configs/dataset_v1.yaml --output datasets/final --decisions data/review_decisions/v1.jsonl --accepted-only| command | what it does | network |
|---|---|---|
acquire |
download and cache PDB/AFDB sources, recording hashes and metadata | yes |
build |
generate semantic instances and rendered variants | no |
validate |
check schemas, provenance, budgets, gold consistency | no |
review |
launch the curator interface | no¹ |
evaluate |
run a model over the dataset, resumable | model calls |
score |
score stored responses | no |
report |
aggregate metrics, bootstrap CIs, controls | no |
¹ the 3D viewer library is fetched from a CDN the first time; everything else, including the prompts and decisions, is local.
pdbthink and structural-reasoning are the same program.
- Acquire the mmCIF from RCSB or AlphaFold DB; store it with its sha256, release date, method and resolution.
- Sanitise it: build the biological assembly when the task needs one, drop hydrogens, waters and crystallisation additives, resolve alternate locations per residue, anonymise ligand component codes, renumber serials, flatten occupancy and zero B-factors (which also removes AlphaFold pLDDT).
- Propose a question. Each generator searches for parameters that satisfy the Appendix A ambiguity margins and records, in machine-readable form, why every candidate was accepted or rejected.
- Render: apply a proper random rotation derived from the instance's seed, round to three decimals, crop if the family allows it and the budget requires it, and emit minimal PDB plus a matched normalized-coordinate table.
- Recompute the gold answer from the rendered coordinates, and confirm it is unchanged under a second rotation and in the paired representation.
- Review: a curator accepts, rejects or annotates. Only accepted instances enter the final dataset.
Steps 3–5 are why the guarantees are mechanical rather than aspirational: gold answers are computed from the same numbers the model reads, so a rotation variant, a cropped variant and a normalized-coordinate variant either agree or the build fails.
- docs/architecture.md — package layout and data flow
- docs/proteins.md — where the source structures come from and which families each can support
- docs/definitions.md — the operational definitions and where each one is enforced
- docs/evaluation.md — protocol, metrics and controls
- docs/api-providers.md — running models through OpenAI-compatible APIs
- docs/evalchemy.md — running the benchmark under Evalchemy
- docs/contamination.md — why the answers are not in this repository
- docs/deployment.md — letting colleagues reach the review interface
configs/ versioned definitions, dataset configs, model configs
data/manifests/ the reproducible list of source structures
data/datasets/ built datasets (gitignored except the frozen manifests)
src/pdbthink/ the package
tests/ unit, golden and end-to-end tests
pytest -qThe suite runs entirely offline against committed fixtures and covers geometry, preprocessing, prompt rendering, answer parsing, every scorer, rotation invariance, rebuild determinism and the full build → validate → evaluate → score → report path.
Release dates are recorded for every source structure as a contamination covariate. They are not a guarantee. The answers themselves are kept out of this repository — see docs/contamination.md and CANARY.md.
Apache 2.0. Structures are redistributed only as small test fixtures; everything else is downloaded from RCSB and the AlphaFold DB under their own terms.