feat(proof): pin real proof-eval digest and operator topic path - #222
Conversation
Ship a harvest-compatible proof-eval binary at /usr/bin/proof-eval with PROOF_METRICS=/PROOF_EVAL_OK markers, baked proxy Qwen/Qwen3.8-0.6B, and 12.5 Gbit/s / no-IB / no-NVLink / no-NCCL-fast-fabric enforcement. The scoring image (CUDA + torch) is what the control plane will pin; the contract-only digest is not a pin. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Stack on the 2000/8000 + WTA/discovery control plane. Pin the scoring image published by run 33892650063 (sha256:78b614a1…, baked proxy Qwen/Qwen3.8-0.6B). Empty digest still 503s until a real sha256 is present; this one was pulled and proved after push. Operator script covers holdout → baseline-on-image → xtask proof-topic → admin POST. Does not touch challenges.toml or the topic schema. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Greptile SummaryThis change adds proof evaluator images, scoring logic, baseline generation, and operator documentation. It must not merge yet: throughput claims are measured as holdout inference instead of the submitted workload, failed bandwidth shaping is recorded as successful, and baseline output cannot be consumed by the baseline gate. The prohibited-platform references must also be removed before merging. Confidence Score: 2/5Not safe to merge: the scoring path can accept unrelated workload measurements, represent an unenforced network constraint as enforced, and cannot seal the baseline produced by its documented command. The explicit repository naming requirement must also be met before merging. Three independently executed checks reproduced failures in the workload measurement, fabric enforcement, and baseline-loading paths. Files Needing Attention: eval/src/proof_eval/harness.py, eval/src/proof_eval/fabric.py, eval/src/proof_eval/cli.py, eval/Dockerfile, eval/Dockerfile.scoring, and eval/README.md
 T-Rex LogsWhat T-Rex did
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| tok = AutoTokenizer.from_pretrained(proxy, trust_remote_code=True) | ||
| model = AutoModelForCausalLM.from_pretrained( | ||
| artifact_dir or proxy, | ||
| torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32, | ||
| trust_remote_code=True, | ||
| ) | ||
| except Exception as exc: # noqa: BLE001 | ||
| raise ContractError(f"no model: {exc}") from exc | ||
| model.eval() | ||
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") | ||
| model.to(device) | ||
|
|
||
| split_nll: dict[str, list[float]] = {s: [] for s in SCORED_SPLITS} | ||
| texts = [] | ||
| for rec in request.holdout: | ||
| split = str(rec.get("split") or rec.get("task") or "web_ood") | ||
| if split not in split_nll: | ||
| split = "web_ood" | ||
| texts.append((split, _shard_text(rec))) | ||
|
|
||
| nlls: list[float] = [] | ||
| tokens = 0 | ||
| import time | ||
|
|
||
| t0 = time.perf_counter() | ||
| with torch.no_grad(): | ||
| for split, text in texts: | ||
| enc = tok(text, return_tensors="pt", truncation=True, max_length=1024) | ||
| enc = {k: v.to(device) for k, v in enc.items()} | ||
| out = model(**enc, labels=enc["input_ids"]) | ||
| nll = float(out.loss.detach().cpu()) | ||
| split_nll[split].append(nll) | ||
| nlls.append(nll) | ||
| tokens += int(enc["input_ids"].numel()) | ||
| wall = max(time.perf_counter() - t0, 1e-6) | ||
| mean = sum(nlls) / max(len(nlls), 1) | ||
| per_split = { | ||
| name: (sum(vals) / len(vals) if vals else mean) for name, vals in split_nll.items() | ||
| } | ||
| tps = tokens / wall if request.family == "throughput" else None | ||
| return { | ||
| "holdout_nll": mean, | ||
| "split_nll": per_split, | ||
| "public_nll": None, | ||
| "tokens_per_sec": tps, | ||
| "step_latency_ms": None, | ||
| "wall_s": int(wall) if request.family == "throughput" else None, |
There was a problem hiding this comment.
For throughput topics, the scorer only checks the submitted recipe text, then measures a next-token forward pass over the holdout. It reports flops_used as zero and derives wall time and throughput from that inference rather than executing the submitted optimizer or training recipe. An unexecuted workload can therefore pass reproduction and supply the measurements used for emission.
Artifacts
- An authored, self-contained runtime harness invokes the targeted scoring path with observable model-runtime doubles, ending with assertions that distinguish holdout inference from recipe execution.
- The evidence script ran successfully and captured the clean zero-FLOPs agent verdict, next-token forward event, positive tokens-per-second result, and zero integer wall time, proving the claim.
- The direct scoring-runtime import in the prepared eval virtualenv failed because torch is not installed, showing why an actual model-runtime invocation could not be run there.
- A command capture contains the evidence script's SHA-256 checksum and complete source, making the executed test source independently inspectable.
| subprocess.run( # noqa: S603 | ||
| [tc, "qdisc", "replace", "dev", dev, "root", "tbf", | ||
| "rate", f"{kbit}kbit", "burst", "256kb", "latency", "50ms"], | ||
| check=False, | ||
| capture_output=True, | ||
| ) | ||
| applied[f"tc:{dev}"] = f"{kbit}kbit" |
There was a problem hiding this comment.
When tc qdisc replace fails, this code ignores its nonzero result and still records the 12.5-Gbit/s cap as applied. Scoring can then continue with an uncapped network while downstream verification sees a successful cap record, allowing an invalid throughput result to be accepted.
Artifacts
- This source invokes the changed fabric enforcement path with a synthetic interface and mocks tc success or failure without changing host networking, directly exercising the result-handling branch.
- The baseline command ran the evidence script with tc return code 0 and recorded the expected 12.5-Gbit/s qdisc cap, establishing the normal comparison case.
- The command ran the same evidence script with tc return code 1 yet returned successfully and recorded the 12.5-Gbit/s qdisc cap, proving failures are ignored.
| if baseline: | ||
| artifact_dir = os.environ.get("PROOF_PROXY_MODEL_DIR") or artifact_dir | ||
| harness = measure(request, artifact_dir) | ||
| if "artifact_fingerprint" in harness: | ||
| harness = {k: v for k, v in harness.items() if k != "artifact_fingerprint"} | ||
| document = { | ||
| "schema_version": PROOF_METRICS_SCHEMA, | ||
| "submission_digest": request.submission_digest, | ||
| "artifact_digest": request.artifact_digest, | ||
| "topic_id": request.topic_id, | ||
| "eval_image_digest": request.eval_image_digest, | ||
| "holdout_commitment": request.holdout_commitment, | ||
| "agent": agent, | ||
| "harness": harness, | ||
| } |
There was a problem hiding this comment.
The baseline command writes a full document with measurements nested under harness, but the configured baseline loader deserializes the entire file as a flat measurement. The nested values are silently defaulted to empty or zero, so required split metrics and the metric commitment cannot verify. The generated baseline cannot be sealed, leaving scoring unavailable.
Artifacts
- A safe fixture invokes the CLI baseline writer and the Rust flat baseline parser, capturing the actual document shapes and parsed metric values; it demonstrates the structural incompatibility.
- The documented flat baseline control was parsed by the Rust loader with holdout NLL 2, five split scores, and throughput present, showing the expected operator-file shape.
- The actual CLI baseline writer emitted a nested harness document that the Rust flat loader parsed as zero NLL, zero split scores, and no throughput, confirming the operator verification failure.
- The proof-eval baseline commitment test passed, confirming that a complete flat metric vector is required and committed.
- All eight existing proof-eval Python contract tests passed after the narrow compatibility exercise, showing the fixture did not disturb tracked code.
| # | ||
| # docker build -f eval/Dockerfile --build-arg WITH_RUNTIME=0 -t proof-eval:contract . | ||
| # | ||
| # No secret, holdout item, teacher endpoint, or Modal reference is baked in. |
There was a problem hiding this comment.
This new comment explicitly names Modal; the same reference also appears in eval/Dockerfile.scoring and eval/README.md. This violates the repository directive prohibiting Modal tokens. The repository requirement must be satisfied before merging.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
2 participants
Summary
Rebased onto
mainafter #220 merged (7bb56c3b). #221 could not be reopened: its base branchcursor/proof-emission-2000-8000-payout-modeswas deleted with that merge. This is the same work, retargeted atmain. Does not touchchallenges.tomlor the topic schema.Pins a real scoring image digest from
publish-proof-eval-imagerun 33892650063 (packaging commit51f937c7). That digest was pulled after push; harvest-PATH + selftest + baked proxy + 12.5 Gbit/s fabric enforcement proved on those bytes. Contract-only image is not pinned.Empty digest remains fail-closed 503 in code (
can_rent/EvalImageUnpinned). This pin is a real sha256, not invented.Operator path:
deploy/scripts/proof-operator-path.sh(prints holdout → baseline-on-pinned-image →xtask proof-topic→ adminPOST /v1/admin/proof/topics).can_scorestill needs LIUM harvest wired + ≥1 open signed topic + verified holdout + sealed baseline on the host.Greptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.@greptileai reviewTest plan
cargo test -p proof-task -p proof-http -p proof-eval -p trustrootcargo fmt --all -- --checkcommitted_pin_is_proof_with_a_real_eval_digestempty_digest_and_unwired_harvest_are_503(fail-closed path still present)cargo test --workspacenot run (pin + eval packaging + operator script)Risk
Once merged, Proof submits stop 503-ing on empty digest. Remaining 503s: unwired harvest, no open topic, unsealed baseline. 8000 bps still burns until those land on the droplet. No ceremony/key rotation in this PR.
Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags.