diff --git a/AGENTS.md b/AGENTS.md index 135683405..0bc8a5122 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ Short contract for agents and operators. Prefer linking over restating runbooks. -**Product:** Cortex ([`CortexLM/cortex`](https://github.com/CortexLM/cortex)) — Bittensor subnet control plane. **Two live challenges:** `bounty` (2000 bps) and `proof` (8000 bps). Proof-weighted 20%/80% lock regardless of eval digest. Proof eval digest is pinned (`ghcr.io/cortexlm/proof-eval@sha256:78b614a1…`, RLM judge `Qwen/Qwen3.8-0.6B`); live submits still 503 until harvest is wired, a baseline is sealed, and ≥1 topic is open. Empty digest stays fail-closed (do not invent a sha256). Sum is 10000. `relearn`, `relearn-image`, `relearn-agent`, `relearn-mm`, `design`, and `prism` are **removed as products** — no trust-root row, no compose services, no emission, and no leaf may verify. Historical miner stubs stay under [`docs/external-miner/`](docs/external-miner/) so old links do not 404. Frozen specs (`docs/DESIGN_CHALLENGE.md`, `docs/PRISM.md`) remain for xtask gates. Leftover `prism-*` crates are the **Lium harvest stack** used by Proof, not a live Prism challenge. Proof scores operator-published research topics (dynamic `topic_id`, digest-pinned RLM judge, `wta` or `discovery` payout). Naming split (Cortex vs leftover `base` / `BASE_*`): [`docs/NAMING.md`](docs/NAMING.md). +**Product:** Cortex ([`CortexLM/cortex`](https://github.com/CortexLM/cortex)) — Bittensor subnet control plane. **Two live challenges:** `bounty` (2000 bps) and `proof` (8000 bps). Proof-weighted 20%/80% lock regardless of eval digest. Proof eval digest is pinned (`ghcr.io/cortexlm/proof-eval@sha256:78b614a1…`, RLM judge via digest-pinned `InferenceOffer`); live submits still 503 until harvest is wired, a baseline is sealed, and ≥1 topic is open. Empty digest stays fail-closed (do not invent a sha256). Sum is 10000. `relearn`, `relearn-image`, `relearn-agent`, `relearn-mm`, `design`, and `prism` are **removed as products** — no trust-root row, no compose services, no emission, and no leaf may verify. Historical miner stubs stay under [`docs/external-miner/`](docs/external-miner/) so old links do not 404. Frozen specs (`docs/DESIGN_CHALLENGE.md`, `docs/PRISM.md`) remain for xtask gates. Leftover `prism-*` crates are the **Lium harvest stack** used by Proof, not a live Prism challenge. Proof scores operator-published research topics (dynamic `topic_id`, digest-pinned RLM judge, `wta` or `discovery` payout). Naming split (Cortex vs leftover `base` / `BASE_*`): [`docs/NAMING.md`](docs/NAMING.md). PRs require a [Greptile](https://greptile.com) review (`.greptile/`). If the bot is silent, comment `@greptileai review`. @@ -69,7 +69,7 @@ When verifying a challenge (local-e2e, staging, or focused tests), **simulate a 2. Edge / failure probes: bad harness, sanitize reject, quota, wrong routes/auth. 3. **Bounty — pair + report:** `ctx bounty pair --hotkey --account-id --accept-terms`, then `POST /v1/pair` (terms + signature) and `POST /v1/reports`. Operator bearer `POST /v1/admin/adjudicate` (`valid` / `already_fixed_not_prod` / `invalid_malicious` / `duplicate`). Scoring **reads** CortexLM/backend public JSON (`BOUNTY_BACKEND_PUBLIC_URL`); do not serve `/v1/public/*` from this repo. 4. **Bounty — fail-closed scorer:** the CortexLM/backend public feed is the only scorer. With no readable `BOUNTY_BACKEND_PUBLIC_URL`, `POST /v1/reports` must answer **503** and the emitter must pay **nobody** — it still covers `E` with `NoScore(ChallengeInternal)`, because a paid challenge with no leaves 409s the seal for every challenge. `BOUNTY_FORCE_SIM` is retired — do not reintroduce an offline bounty scorer. See [`docs/BOUNTY.md`](docs/BOUNTY.md). -5. **Proof — submit:** `POST /v1/submissions` with a `topic_id`. Missing / unknown / not-open → **400** (no row). Architecture ≠ the baked RLM judge id → **400**. Empty `eval_image_digest`, zero open topics, or an unsealed baseline → **503**. Contamination / empty manifest persist **rejected** without rent. `GET /v1/proof/topics` must never leak holdout records. +5. **Proof — submit:** `POST /v1/submissions` with a `topic_id`. Missing/unknown/not-open → **400** (no row). Empty `eval_image_digest`, missing/closed/misconfigured RLM judge `InferenceOffer`, missing judge API key, spoofed topic origin, zero open topics, or an unsealed baseline → **503**. Miners submit claim + code + FLOPs + artifact; they do not bind the judge offer. Contamination / empty manifest persist **rejected** without rent. `GET /v1/proof/topics` must never leak holdout records. 6. Leaf emission → `POST /v1/weights/raw` → seal → `GET /v1/weights/latest` with **`sealed: true`** (burn fallback alone is not a real seal). **Never host Sim in staging/prod** for live scoring. `PROOF_FORCE_SIM=1` is CI/local opt-in only (`deploy/scripts/assert-compose-matrix.sh` fails if a droplet overlay sets one). Live Proof rent requires a digest pin in `config/proof-pin.toml` plus miner BYOK (`LIUM_API_KEY` / `X-Lium-Api-Key`). Never log or commit that key. Do not invent `eval_image_digest`. diff --git a/Cargo.lock b/Cargo.lock index 32d81f173..7bd9222e4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3499,9 +3499,20 @@ dependencies = [ "proof-task", "serde", "serde_json", + "tokio", "tracing", ] +[[package]] +name = "proof-holdout" +version = "0.1.0" +dependencies = [ + "hex", + "serde", + "sha2 0.10.9", + "thiserror 2.0.19", +] + [[package]] name = "proof-http" version = "0.1.0" @@ -3549,6 +3560,7 @@ version = "0.1.0" dependencies = [ "crypto", "hex", + "proof-holdout", "serde", "serde_json", "sha2 0.10.9", diff --git a/bins/ctx/src/main.rs b/bins/ctx/src/main.rs index a1935a8a9..595ee0026 100644 --- a/bins/ctx/src/main.rs +++ b/bins/ctx/src/main.rs @@ -172,9 +172,6 @@ struct ProofSubmitArgs { /// Optional locator for the artifact (git url, object URL). #[arg(long, value_name = "URL")] artifact_uri: Option, - /// Architecture / proxy id baked by the pin. - #[arg(long, value_name = "ID")] - architecture: String, /// What the recipe achieved (the RLM re-runs this claim). #[arg(long, value_name = "TEXT")] claim: String, @@ -231,7 +228,6 @@ async fn run_proof(client: &Client, cmd: ProofCmd, json: bool) -> Result<(), Str topic_id: args.topic_id, artifact_digest: args.artifact_digest, artifact_uri: args.artifact_uri, - architecture: args.architecture, claim: args.claim, declared_flops: args.declared_flops, manifest_file: args.manifest_file, diff --git a/bins/ctx/src/proof.rs b/bins/ctx/src/proof.rs index a8ea03058..84876084a 100644 --- a/bins/ctx/src/proof.rs +++ b/bins/ctx/src/proof.rs @@ -25,8 +25,6 @@ pub struct SubmitInput { pub artifact_digest: String, /// Optional locator for the artifact. pub artifact_uri: Option, - /// Architecture / proxy id baked by the pin. - pub architecture: String, /// Public claim the RLM re-runs (what you say the recipe achieved). pub claim: String, /// FLOPs you spent. Must be ≤ the topic budget. @@ -50,10 +48,6 @@ pub async fn submit(client: &Client, input: &SubmitInput, json_out: bool) -> Res if topic_id.is_empty() { return Err("topic-id is required (ctx proof topics lists currently open ids)".into()); } - let architecture = input.architecture.trim(); - if architecture.is_empty() { - return Err("architecture is required (must match the proxy the pin bakes)".into()); - } let claim = input.claim.trim(); if claim.is_empty() { return Err("claim is required (what the recipe achieved)".into()); @@ -63,7 +57,6 @@ pub async fn submit(client: &Client, input: &SubmitInput, json_out: bool) -> Res "miner_hotkey": hotkey, "topic_id": topic_id, "artifact_digest": digest, - "architecture": architecture, "claim": claim, "declared_flops": input.declared_flops, "manifest": manifest, @@ -257,7 +250,8 @@ fn explain_failure(status: u16, message: &str) -> String { 400 => format!("refused ({message}). Nothing was stored and nothing was rented."), 503 => format!( "HTTP 503: {message}\n The host cannot score right now (empty eval digest, \ - no open topics, or an unsealed baseline). Nothing was stored, nothing was rented." + missing/closed RLM judge backend, no open topics, or an unsealed baseline). \ + Nothing was stored, nothing was rented." ), other => format!("HTTP {other}: {message}"), } diff --git a/bins/proof-challenge/src/main.rs b/bins/proof-challenge/src/main.rs index 6e0d728de..9a5937052 100644 --- a/bins/proof-challenge/src/main.rs +++ b/bins/proof-challenge/src/main.rs @@ -20,7 +20,8 @@ use clap::Parser; use prism_lium::LiumClient; use proof_challenge::{ hash_admin_token, parse_holdout_file, proof_router, AppState, BaselineMeasurement, EvalBackend, - LiveScorer, MemoryStore, ProofPin, TopicDocument, CHALLENGE_ID, SCORING_VERSION, + InferenceOffer, LiveScorer, MemoryStore, ProofPin, TopicDocument, CHALLENGE_ID, + SCORING_VERSION, }; use proof_eval::supported_custom; use proof_harvest::{HarvestLimits, LiumProofHarvest}; @@ -64,6 +65,12 @@ struct Cli { /// Sealed baseline measurements (JSON map keyed by topic id). #[arg(long, env = "PROOF_BASELINE_FILE")] baseline_file: Option, + /// Live RLM judge `InferenceOffer` JSON. Operator state; never a git pin. Missing/closed → 503. + #[arg(long, env = "PROOF_INFERENCE_OFFER_FILE")] + inference_offer_file: Option, + /// Provider API key file. Never logged, never on `/v1/status`. + #[arg(long, env = "PROOF_INFERENCE_API_KEY_FILE")] + inference_api_key_file: Option, } fn main() -> ExitCode { @@ -110,13 +117,33 @@ fn run(cli: &Cli) -> Result<(), String> { Ok(n) => tracing::info!(topics = n, "sealed baselines recorded"), Err(e) => tracing::warn!("baselines unavailable ({e}); submissions will 503 until fixed"), } + let offer = match load_offer(&pin, cli.inference_offer_file.as_deref()) { + Ok(o) => { + tracing::info!(offer_id = %o.offer_id, status = ?o.status, "inference offer loaded"); + Some(o) + } + Err(e) => { + tracing::warn!("inference offer unavailable ({e}); submissions will 503 until fixed"); + None + } + }; + let judge_api_key = load_inference_api_key(cli.inference_api_key_file.as_deref()); + match (&offer, &judge_api_key) { + (Some(o), Some(_)) if o.is_open() => { + tracing::info!("inference api key file present (contents not logged)"); + } + (Some(o), None) if o.is_open() => tracing::warn!( + "open InferenceOffer needs PROOF_INFERENCE_API_KEY_FILE; live submits will 503" + ), + _ => {} + } let rt = tokio::runtime::Builder::new_multi_thread() .enable_all() .build() .map_err(|e| e.to_string())?; - let live_scorer = build_live_scorer(backend, cli.eval_timeout_secs); + let live_scorer = build_live_scorer(backend, cli.eval_timeout_secs, judge_api_key.clone()); match backend { EvalBackend::Lium if live_scorer.is_some() => { tracing::info!("live harvest wired: digest-pinned proof-eval image on Lium"); @@ -133,13 +160,19 @@ fn run(cli: &Cli) -> Result<(), String> { pin, backend, live_scorer, + offer, + judge_api_key, admin_hashes: Arc::new(load_admin_hashes(cli.admin_tokens_file.as_deref())), epoch: 0, }; rt.block_on(serve(cli.bind, state)) } -fn build_live_scorer(backend: EvalBackend, run_timeout_secs: u64) -> Option> { +fn build_live_scorer( + backend: EvalBackend, + run_timeout_secs: u64, + judge_api_key: Option, +) -> Option> { if backend != EvalBackend::Lium { return None; } @@ -173,11 +206,18 @@ fn build_live_scorer(backend: EvalBackend, run_timeout_secs: u64) -> Option) -> Option { + let p = path?; + std::fs::read_to_string(p) + .ok() + .map(|s| s.trim().to_owned()) + .filter(|s| !s.is_empty()) } fn load_ssh_public_key() -> Option { @@ -287,6 +327,14 @@ fn record_one_baseline( Ok(()) } +fn load_offer(pin: &ProofPin, path: Option<&Path>) -> Result { + let p = path.ok_or("PROOF_INFERENCE_OFFER_FILE not set")?; + let body = std::fs::read_to_string(p).map_err(|e| format!("read {}: {e}", p.display()))?; + let offer = InferenceOffer::from_json(&body).map_err(|e| e.to_string())?; + offer.validate(pin).map_err(|e| e.to_string())?; + Ok(offer) +} + fn load_admin_hashes(path: Option<&Path>) -> Vec { let Some(p) = path else { return Vec::new(); @@ -349,26 +397,50 @@ mod tests { std::env::set_var("LIUM_SSH_PUBLIC_KEY_FILE", &pubkey); assert!( - build_live_scorer(EvalBackend::Lium, 900).is_some(), + build_live_scorer(EvalBackend::Lium, 900, None).is_some(), "Lium boot must wire the digest-pinned harvest" ); assert!( - build_live_scorer(EvalBackend::Sim, 900).is_none(), + build_live_scorer(EvalBackend::Sim, 900, None).is_none(), "sim scores in-process; a Lium harvest there would spend money" ); std::env::remove_var("LIUM_API_KEY"); - assert!(build_live_scorer(EvalBackend::Lium, 900).is_none()); + assert!(build_live_scorer(EvalBackend::Lium, 900, None).is_none()); std::env::set_var("LIUM_API_KEY", " "); - assert!(build_live_scorer(EvalBackend::Lium, 900).is_none()); + assert!(build_live_scorer(EvalBackend::Lium, 900, None).is_none()); std::env::set_var("LIUM_API_KEY", "test-key-not-a-real-secret"); std::env::set_var("LIUM_SSH_PUBLIC_KEY_FILE", "/nonexistent/id.pub"); - assert!(build_live_scorer(EvalBackend::Lium, 900).is_none()); + assert!(build_live_scorer(EvalBackend::Lium, 900, None).is_none()); std::env::remove_var("LIUM_API_KEY"); std::env::remove_var("LIUM_SSH_PUBLIC_KEY_FILE"); } + #[test] + fn inference_api_key_file_is_read_not_existence_only() { + let dir = std::env::temp_dir().join(format!( + "proof-key-{}", + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("clock") + .as_nanos() + )); + std::fs::create_dir_all(&dir).expect("dir"); + let missing = dir.join("nope"); + assert!(load_inference_api_key(Some(&missing)).is_none()); + let empty = dir.join("empty"); + std::fs::write(&empty, " \n").expect("write"); + assert!(load_inference_api_key(Some(&empty)).is_none()); + let present = dir.join("key"); + std::fs::write(&present, " sk-live-not-a-real-secret \n").expect("write"); + assert_eq!( + load_inference_api_key(Some(&present)).as_deref(), + Some("sk-live-not-a-real-secret") + ); + let _ = std::fs::remove_dir_all(&dir); + } + fn stub_ssh_pubkey(tag: &str) -> PathBuf { let path = std::env::temp_dir().join(format!("proof-test-{tag}.pub")); std::fs::write(&path, "ssh-ed25519 AAAAtest proof-test\n").expect("write pubkey"); diff --git a/config/proof-pin.toml b/config/proof-pin.toml index 45d161900..4e241695a 100644 --- a/config/proof-pin.toml +++ b/config/proof-pin.toml @@ -1,16 +1,17 @@ # Cortex pin for the Proof challenge (`proof`). # Global floors only — there is no topic catalog here. Problems are # operator-published signed documents; git never carries holdout records, -# teacher hosts, or a live topic list. +# teacher hosts, live InferenceOffer origins, or a live topic list. # # Deploy = bump eval_image_digest after proof-eval CI is green. Empty digest # is the pre-launch state: live submits answer 503. Do not invent a sha256. -# proxy_model must be an id the image actually bakes (see proxy_models). +# `proxy_model` / `proxy_models` stay empty — no HF bake. The RLM judge is +# the live InferenceOffer, not a model id compiled into the image. # # Scoring image published by .github/workflows/publish-proof-eval-image.yml # run 33892650063 (commit 51f937c7818f0eb1e3ed1412972de98b6994952b). -# Pulled that digest after push; harvest-PATH + selftest + baked proxy -# Qwen/Qwen3.8-0.6B + 12.5 Gbit/s fabric enforcement proved on those bytes. +# Pulled that digest after push; harvest-PATH + selftest + 12.5 Gbit/s +# fabric enforcement proved on those bytes. # # Topic documents are signed by the `proof` row key in config/challenges.toml # (sr25519, domain `base-proof-topic-v1`). `topic_pubkey` must match that row. @@ -23,6 +24,17 @@ # epsilon_topic_max_regress_min = 0.05 # epsilon_throughput_rel_min = 0.05 # quality_floor_nll_max = 0.02 +# max_input_tokens_ceiling = 32768 +# max_output_tokens_ceiling = 8192 +# +# Inference: pin ceilings / modes / commitment bound the **judge** offer that +# digest-pinned proof-eval / harvest call. A topic's signed `inference{…}` may +# override / tighten those judge defaults. Optional topic field +# `require_judge_offer_commitment` is not a miner bind. Empty `model` or empty +# `base_url` is pre-launch fail-closed (like an empty digest), not a boot +# reject. The live offer (`PROOF_INFERENCE_OFFER_FILE`) is operator state +# (open/closed). Miners do not bind it. No baked Qwen; `proxy_model` / +# `proxy_models` stay empty. Missing/closed/judge down → 503. # # Holdout size / stratum size are the measurement harness (5 scored splits × 24), # not the problem list. @@ -30,9 +42,13 @@ challenge_id = "proof" scoring_version = 1 base_model_family = "Qwen/Qwen3.8" -# Baked into ghcr.io/cortexlm/proof-eval@sha256:78b614a1… (see baked_proxies.json). -proxy_model = "Qwen/Qwen3.8-0.6B" -proxy_models = ["Qwen/Qwen3.8-0.6B"] +proxy_model = "" +proxy_models = [] +inference_config_schema_version = 1 +allowed_modes = ["chat", "completions", "embeddings"] +max_input_tokens_ceiling = 32768 +max_output_tokens_ceiling = 8192 +inference_offer_commitment_alg = "sha256" eval_image = "ghcr.io/cortexlm/proof-eval" eval_image_digest = "sha256:78b614a1f51ce5dd80076c4e343a2b31b85d6c36025e02836cb83929867e7009" proof_git = "https://github.com/CortexLM/cortex" @@ -45,3 +61,11 @@ epsilon_throughput_rel_min = 0.05 quality_floor_nll_max = 0.02 holdout_size = 120 stratum_size = 24 + +[inference] +provider = "openai_compatible" +base_url = "" +model = "" +mode = "chat" +max_input_tokens = 32768 +max_output_tokens = 8192 diff --git a/crates/proof-challenge/src/lib.rs b/crates/proof-challenge/src/lib.rs index 55ddd1046..d411c48dc 100644 --- a/crates/proof-challenge/src/lib.rs +++ b/crates/proof-challenge/src/lib.rs @@ -22,8 +22,9 @@ pub use proof_eval::{ pub use proof_http::{hash_admin_token, proof_router, AppState}; pub use proof_store::{ArtifactManifest, MemoryStore}; pub use proof_task::{ - HoldoutRecord, ProofPin, TopicDocument, BASE_MODEL_FAMILY, CHALLENGE_ID, - CHALLENGE_ID_BYTES as PROOF_ID_BYTES, SCORE_MAX as PROOF_SCORE_MAX, SCORING_VERSION, + HoldoutRecord, InferenceOffer, OfferError, ProofPin, TopicDocument, BASE_MODEL_FAMILY, + CHALLENGE_ID, CHALLENGE_ID_BYTES as PROOF_ID_BYTES, SCORE_MAX as PROOF_SCORE_MAX, + SCORING_VERSION, }; /// Build a D24-complete score map: each expected hotkey is a **sum** of diff --git a/crates/proof-eval/src/lib.rs b/crates/proof-eval/src/lib.rs index 5f605941e..3c17aa0d3 100644 --- a/crates/proof-eval/src/lib.rs +++ b/crates/proof-eval/src/lib.rs @@ -17,7 +17,8 @@ clippy::doc_markdown, clippy::module_name_repetitions, clippy::cast_precision_loss, - clippy::must_use_candidate + clippy::must_use_candidate, + clippy::too_many_arguments )] use std::collections::{BTreeMap, BTreeSet}; @@ -27,8 +28,9 @@ use prism_lium_types::{EvalReceipt, NoScoreGate}; use proof_score::{AgentVerdict, HarnessMetrics, ProofCheatCode, ProofKind, SealedBaseline}; use proof_store::ArtifactManifest; use proof_task::{ - canonical_json, contamination, HoldoutRecord, HoldoutSplit, MetricFamily, ProofPin, - TopicDocument, BASELINE_DOMAIN, + canonical_json, contamination, require_open_offer, resolve_inference, HoldoutRecord, + HoldoutSplit, InferenceOffer, MetricFamily, OfferError, ProofPin, TopicDocument, + BASELINE_DOMAIN, }; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; @@ -93,6 +95,18 @@ pub enum EvalError { /// The operator-recorded baseline does not match the topic/pin. #[error("recorded baseline: {0}")] Baseline(String), + /// No live RLM judge InferenceOffer on this host. + #[error("inference offer missing; refuse scoring")] + InferenceOfferMissing, + /// Live InferenceOffer is closed. + #[error("inference offer is closed; refuse scoring")] + InferenceOfferClosed, + /// Live InferenceOffer failed pin validation. + #[error("inference offer: {0}")] + InferenceOffer(String), + /// Live open judge offer needs auth and the key file is missing/unreadable. + #[error("inference API key missing; refuse scoring")] + InferenceAuthMissing, } /// Schema version of the metrics+verdict document the eval image emits. @@ -200,10 +214,12 @@ impl ProofEvalDocument { #[async_trait] pub trait LiveScorer: Send + Sync { /// Score one artifact on one topic's verified holdout. + #[allow(clippy::too_many_arguments)] async fn score( &self, pin: &ProofPin, topic: &TopicDocument, + offer: &InferenceOffer, frozen_digest: &str, artifact_digest: &str, holdout: &[HoldoutRecord], @@ -336,16 +352,27 @@ pub fn metrics_commitment( hex::encode(h.finalize()) } +fn map_offer_err(e: OfferError) -> EvalError { + match e { + OfferError::Missing => EvalError::InferenceOfferMissing, + OfferError::Closed => EvalError::InferenceOfferClosed, + other => EvalError::InferenceOffer(other.to_string()), + } +} + /// Whether this host can produce a verdict at all. pub fn scoring_readiness( pin: &ProofPin, backend: EvalBackend, live: Option<&dyn LiveScorer>, has_open_sealed_topic: bool, + offer: Option<&InferenceOffer>, + judge_api_key: Option<&str>, ) -> Result<(), EvalError> { if !has_open_sealed_topic { return Err(EvalError::NoOpenTopic); } + require_open_offer(offer, pin).map_err(map_offer_err)?; match backend { EvalBackend::Sim => Ok(()), EvalBackend::Lium => { @@ -353,11 +380,40 @@ pub fn scoring_readiness( return Err(EvalError::EvalImageUnpinned); } let scorer = live.ok_or(EvalError::LiveHarvestUnavailable)?; - scorer.ready() + scorer.ready()?; + judge_api_key_ready(judge_api_key) } } } +/// Live Lium scoring needs a readable non-empty judge API key. +/// +/// # Errors +/// +/// [`EvalError::InferenceAuthMissing`] when the key is absent. +pub fn judge_api_key_ready(judge_api_key: Option<&str>) -> Result<(), EvalError> { + if judge_api_key.map(str::trim).is_some_and(|s| !s.is_empty()) { + Ok(()) + } else { + Err(EvalError::InferenceAuthMissing) + } +} + +/// `PROOF_INFERENCE_BASE_URL`, else first non-empty line of `PROOF_INFERENCE_BASE_URL_FILE`. +pub fn secret_backed_base_url() -> Option { + let env = std::env::var("PROOF_INFERENCE_BASE_URL") + .ok() + .map(|s| s.trim().to_owned()) + .filter(|s| !s.is_empty()); + if env.is_some() { + return env; + } + std::fs::read_to_string(std::env::var("PROOF_INFERENCE_BASE_URL_FILE").ok()?) + .ok() + .map(|s| s.trim().to_owned()) + .filter(|s| !s.is_empty()) +} + /// One finished eval. #[derive(Debug, Clone)] pub struct EvalOutcome { @@ -477,17 +533,38 @@ pub const BASELINE_SKILL: f64 = 0.40; pub async fn eval_after_freeze( pin: &ProofPin, topic: &TopicDocument, + offer: &InferenceOffer, frozen_digest: &str, artifact_digest: &str, holdout: &[HoldoutRecord], claim: &str, backend: EvalBackend, live: Option<&dyn LiveScorer>, + judge_api_key: Option<&str>, ) -> Result { if frozen_digest.trim().is_empty() || holdout.is_empty() { return Err(EvalError::HoldoutSealed); } - scoring_readiness(pin, backend, live, true)?; + scoring_readiness(pin, backend, live, true, Some(offer), judge_api_key)?; + offer + .serves_topic(pin, topic) + .map_err(|e| EvalError::InferenceOffer(e.to_string()))?; + let resolved = resolve_inference( + pin, + Some(&topic.inference), + secret_backed_base_url().as_deref(), + Some(offer), + ); + if !resolved.ready_to_score() { + return Err(EvalError::InferenceOffer( + OfferError::Incomplete.to_string(), + )); + } + if resolved.base_url.trim() != offer.provider.base_url.trim() { + return Err(EvalError::InferenceOffer( + OfferError::OriginMismatch.to_string(), + )); + } let doc = match backend { EvalBackend::Sim => { let skill = unit(&[artifact_digest, "skill"], 0); @@ -496,7 +573,15 @@ pub async fn eval_after_freeze( EvalBackend::Lium => { let scorer = live.ok_or(EvalError::LiveHarvestUnavailable)?; scorer - .score(pin, topic, frozen_digest, artifact_digest, holdout, claim) + .score( + pin, + topic, + offer, + frozen_digest, + artifact_digest, + holdout, + claim, + ) .await? } }; @@ -544,17 +629,42 @@ pub fn parse_agent_verdict(body: &str) -> Result { #[cfg(test)] mod tests { use proof_task::{ - default_adamw, holdout_commitment, synthetic_holdout, TopicDocument, TopicStatus, - FLOPS_BUDGET_MAX, STRATUM_SIZE, + default_adamw, holdout_commitment, inference_config_commitment, synthetic_holdout, + InferenceConfig, InferenceMode, InferenceOffer, InferenceProvider, InferenceProviderKind, + OfferStatus, TopicDocument, TopicStatus, FLOPS_BUDGET_MAX, STRATUM_SIZE, }; use super::*; fn pin(digest: &str) -> ProofPin { - ProofPin { + let mut p = ProofPin { eval_image_digest: digest.to_owned(), topic_pubkey: "ab".repeat(32), ..ProofPin::default() + }; + p.inference.model = "master-proxy-v0".into(); + p + } + + fn offer() -> InferenceOffer { + let config = InferenceConfig { + mode: InferenceMode::Chat, + model_ref: "master-proxy-v0".into(), + max_input_tokens: 32_768, + max_output_tokens: 8_192, + temperature: Some(0.0), + top_p: None, + timeout_ms: None, + }; + InferenceOffer { + offer_id: "master-v0".into(), + provider: InferenceProvider { + kind: InferenceProviderKind::OpenaiCompatible, + base_url: "http://127.0.0.1:8000/v1".into(), + }, + config_commitment: inference_config_commitment(&config, "http://127.0.0.1:8000/v1"), + config, + status: OfferStatus::Open, } } @@ -583,6 +693,7 @@ mod tests { &self, pin: &ProofPin, topic: &TopicDocument, + _offer: &InferenceOffer, frozen: &str, artifact: &str, _holdout: &[HoldoutRecord], @@ -612,12 +723,14 @@ mod tests { let unpinned = eval_after_freeze( &pin(""), &t, + &offer(), "d", "art", &recs, "claim", EvalBackend::Lium, None, + None, ) .await .expect_err("no digest"); @@ -629,12 +742,14 @@ mod tests { let unwired = eval_after_freeze( &pin(&format!("sha256:{}", "ab".repeat(32))), &t, + &offer(), "d", "art", &recs, "claim", EvalBackend::Lium, None, + None, ) .await .expect_err("no harvest"); @@ -652,12 +767,14 @@ mod tests { let out = eval_after_freeze( &p, &t, + &offer(), "digest-a", "art", &recs, "claim", EvalBackend::Lium, Some(&Harvest { reproduced: true }), + Some("test-judge-key"), ) .await .expect("live"); @@ -691,17 +808,44 @@ mod tests { #[test] fn readiness_names_the_root_cause() { let live = pin(&format!("sha256:{}", "ab".repeat(32))); + let o = offer(); assert!(matches!( - scoring_readiness(&live, EvalBackend::Sim, None, false), + scoring_readiness(&live, EvalBackend::Sim, None, false, Some(&o), None), Err(EvalError::NoOpenTopic) )); - scoring_readiness(&ProofPin::default(), EvalBackend::Sim, None, true).expect("sim"); + scoring_readiness( + &ProofPin::default(), + EvalBackend::Sim, + None, + true, + Some(&o), + None, + ) + .expect("sim"); assert!(matches!( - scoring_readiness(&ProofPin::default(), EvalBackend::Lium, None, true), + scoring_readiness( + &ProofPin::default(), + EvalBackend::Sim, + None, + true, + None, + None + ), + Err(EvalError::InferenceOfferMissing) + )); + assert!(matches!( + scoring_readiness( + &ProofPin::default(), + EvalBackend::Lium, + None, + true, + Some(&o), + None, + ), Err(EvalError::EvalImageUnpinned) )); assert!(matches!( - scoring_readiness(&live, EvalBackend::Lium, None, true), + scoring_readiness(&live, EvalBackend::Lium, None, true, Some(&o), None), Err(EvalError::LiveHarvestUnavailable) )); scoring_readiness( @@ -709,8 +853,21 @@ mod tests { EvalBackend::Lium, Some(&Harvest { reproduced: true }), true, + Some(&o), + Some("test-judge-key"), ) .expect("ready"); + assert!(matches!( + scoring_readiness( + &live, + EvalBackend::Lium, + Some(&Harvest { reproduced: true }), + true, + Some(&o), + None, + ), + Err(EvalError::InferenceAuthMissing) + )); } #[test] diff --git a/crates/proof-harvest/Cargo.toml b/crates/proof-harvest/Cargo.toml index 5a14f26bd..85e113a56 100644 --- a/crates/proof-harvest/Cargo.toml +++ b/crates/proof-harvest/Cargo.toml @@ -18,5 +18,8 @@ serde = { version = "1", features = ["derive"] } serde_json = "1" tracing = "0.1" +[dev-dependencies] +tokio = { version = "1", features = ["macros", "rt", "rt-multi-thread"] } + [lints] workspace = true diff --git a/crates/proof-harvest/src/lib.rs b/crates/proof-harvest/src/lib.rs index 002e07bd7..cf7d7f99b 100644 --- a/crates/proof-harvest/src/lib.rs +++ b/crates/proof-harvest/src/lib.rs @@ -2,20 +2,29 @@ //! //! The RLM agent and the metric harness live inside the image. This crate //! boots that image on a Lium pod, hands it the run request (including the -//! topic constraints the image must enforce — e.g. a 12.5 Gbit/s cap), reads -//! back the metrics document, and tears the pod down. Nothing here computes -//! a score. There is no sim fallback. +//! live `InferenceOffer` judge backend and the topic constraints the image +//! must enforce — e.g. a 12.5 Gbit/s cap), reads back the metrics document, +//! and tears the pod down. Nothing here computes a score. Miners do not bind +//! or train against the offer. There is no sim fallback. #![forbid(unsafe_code)] -#![allow(clippy::missing_errors_doc, clippy::doc_markdown)] +#![allow( + clippy::missing_errors_doc, + clippy::doc_markdown, + clippy::too_many_arguments +)] use std::sync::Arc; use async_trait::async_trait; use harvest_pod::{harvest_template_name, EvalPod, PodProgram}; use prism_lium_types::InstanceSpec; -use proof_eval::{EvalError, LiveScorer, ProofEvalDocument, PROOF_METRICS_SCHEMA}; -use proof_task::{HoldoutRecord, ProofPin, TopicDocument, CHALLENGE_ID}; +use proof_eval::{ + secret_backed_base_url, EvalError, LiveScorer, ProofEvalDocument, PROOF_METRICS_SCHEMA, +}; +use proof_task::{ + resolve_inference, HoldoutRecord, InferenceOffer, ProofPin, TopicDocument, CHALLENGE_ID, +}; use serde::{Deserialize, Serialize}; /// Prefix the eval image prints before its metrics document. @@ -57,8 +66,22 @@ pub struct HarvestRequest { pub topic_id: String, /// Metric family wire name. pub family: String, - /// RLM judge id the image must use (eval-image InferenceOffer, not a miner training proxy). - pub proxy_model: String, + /// Live judge offer id the eval image must call. + pub inference_offer_id: String, + /// Provider kind wire name. + pub provider_kind: String, + /// Judge origin (operator state; not a public status field). + pub base_url: String, + /// Serving mode. + pub mode: String, + /// Provider model id (not an HF bake). + pub model_ref: String, + /// Input token cap for this run (min of offer and topic). + pub max_input_tokens: u32, + /// Output token cap for this run. + pub max_output_tokens: u32, + /// Judge config commitment. + pub config_commitment: String, /// Eval image digest, so the image can stamp its own provenance. pub eval_image_digest: String, /// Commitment the records below must hash to. @@ -96,11 +119,25 @@ impl Default for HarvestLimits { } } +/// Env file staged as `teacher.env` and sourced by the eval image. +/// +/// The key never enters [`HarvestRequest`] or `/v1/status`. Values are +/// single-quoted so a special character cannot break `set -a` sourcing. +pub fn judge_teacher_env(api_key: &str) -> Result, EvalError> { + let key = api_key.trim(); + if key.is_empty() || key.contains('\n') || key.contains('\r') || key.contains('\0') { + return Err(EvalError::InferenceAuthMissing); + } + let escaped = key.replace('\'', "'\\''"); + Ok(format!("OPENAI_API_KEY='{escaped}'\nPROOF_INFERENCE_API_KEY='{escaped}'\n").into_bytes()) +} + /// [`LiveScorer`] over a digest-pinned eval image on a Lium pod. pub struct LiumProofHarvest { pod: Arc, limits: HarvestLimits, ssh_public_keys: Vec, + judge_api_key: Option, } impl LiumProofHarvest { @@ -111,9 +148,22 @@ impl LiumProofHarvest { pod, limits, ssh_public_keys, + judge_api_key: None, } } + /// Inject the judge API key staged into `teacher.env` on the pod. + /// + /// Empty / whitespace is treated as missing (fail-closed on a live run). + #[must_use] + pub fn with_judge_api_key(mut self, key: Option) -> Self { + self.judge_api_key = key.and_then(|s| { + let t = s.trim().to_owned(); + (!t.is_empty()).then_some(t) + }); + self + } + fn spec(&self, pin: &ProofPin, frozen_digest: &str) -> InstanceSpec { InstanceSpec { name: format!("proof-{}", &frozen_digest[..12.min(frozen_digest.len())]), @@ -141,6 +191,7 @@ impl LiveScorer for LiumProofHarvest { &self, pin: &ProofPin, topic: &TopicDocument, + offer: &InferenceOffer, frozen_digest: &str, artifact_digest: &str, holdout: &[HoldoutRecord], @@ -153,6 +204,30 @@ impl LiveScorer for LiumProofHarvest { return Err(EvalError::HoldoutSealed); } self.ready()?; + offer + .serves_topic(pin, topic) + .map_err(|e| EvalError::InferenceOffer(e.to_string()))?; + let resolved = resolve_inference( + pin, + Some(&topic.inference), + secret_backed_base_url().as_deref(), + Some(offer), + ); + if !resolved.ready_to_score() { + return Err(EvalError::InferenceOffer( + proof_task::OfferError::Incomplete.to_string(), + )); + } + if resolved.base_url.trim() != offer.provider.base_url.trim() { + return Err(EvalError::InferenceOffer( + proof_task::OfferError::OriginMismatch.to_string(), + )); + } + let env = judge_teacher_env(self.judge_api_key.as_deref().unwrap_or(""))?; + let max_in = resolved.max_input_tokens.min(offer.config.max_input_tokens); + let max_out = resolved + .max_output_tokens + .min(offer.config.max_output_tokens); let request = HarvestRequest { schema_version: PROOF_METRICS_SCHEMA, challenge_id: CHALLENGE_ID.to_owned(), @@ -160,7 +235,14 @@ impl LiveScorer for LiumProofHarvest { artifact_digest: artifact_digest.to_owned(), topic_id: topic.id.clone(), family: topic.metric.family.as_str().to_owned(), - proxy_model: pin.proxy_for(topic.proxy_model.as_deref()), + inference_offer_id: offer.offer_id.clone(), + provider_kind: resolved.provider.as_str().to_owned(), + base_url: resolved.base_url.clone(), + mode: resolved.mode.as_str().to_owned(), + model_ref: resolved.model.clone(), + max_input_tokens: max_in, + max_output_tokens: max_out, + config_commitment: offer.config_commitment.clone(), eval_image_digest: pin.eval_image_digest.clone(), holdout_commitment: topic.holdout_commitment.clone(), constraints: topic.constraints, @@ -177,7 +259,7 @@ impl LiveScorer for LiumProofHarvest { .boot(&self.spec(pin, frozen_digest)) .await .map_err(EvalError::Backend)?; - let run = self.pod.run(&instance, &body, &[]).await; + let run = self.pod.run(&instance, &body, &env).await; let shutdown = self.pod.shutdown(&instance).await; match shutdown { Ok(true) => {} @@ -253,7 +335,14 @@ mod tests { artifact_digest: "a".into(), topic_id: topic.id.clone(), family: topic.metric.family.as_str().into(), - proxy_model: "Qwen/Qwen3.8-0.6B".into(), + inference_offer_id: "master-v0".into(), + provider_kind: "openai_compatible".into(), + base_url: "http://127.0.0.1:8000/v1".into(), + mode: "chat".into(), + model_ref: "master-proxy-v0".into(), + max_input_tokens: 4_096, + max_output_tokens: 256, + config_commitment: "ab".repeat(32), eval_image_digest: String::new(), holdout_commitment: topic.holdout_commitment.clone(), constraints: topic.constraints, @@ -266,5 +355,208 @@ mod tests { assert_eq!(v["constraints"]["max_inter_node_gbps"], 12.5); assert_eq!(v["constraints"]["no_infiniband"], true); assert_eq!(v["challenge_id"], "proof"); + assert_eq!(v["provider_kind"], "openai_compatible"); + assert_eq!(v["mode"], "chat"); + assert!(v.get("proxy_model").is_none()); + assert!(v.get("api_key").is_none()); + } + + #[test] + fn teacher_env_carries_the_judge_key_and_never_the_request() { + let env = judge_teacher_env("sk-live-not-a-real-secret").expect("env"); + let text = String::from_utf8(env).expect("utf8"); + assert!(text.contains("OPENAI_API_KEY='sk-live-not-a-real-secret'")); + assert!(text.contains("PROOF_INFERENCE_API_KEY='sk-live-not-a-real-secret'")); + assert!(judge_teacher_env("").is_err()); + assert!(judge_teacher_env("has\nnewline").is_err()); + let quoted = judge_teacher_env("o'reilly").expect("quote"); + assert_eq!( + String::from_utf8(quoted).expect("utf8"), + "OPENAI_API_KEY='o'\\''reilly'\nPROOF_INFERENCE_API_KEY='o'\\''reilly'\n" + ); + } + + struct CapturePod { + env: std::sync::Mutex>, + request: std::sync::Mutex>, + booted: std::sync::Mutex, + } + + impl CapturePod { + fn new() -> Arc { + Arc::new(Self { + env: std::sync::Mutex::new(Vec::new()), + request: std::sync::Mutex::new(Vec::new()), + booted: std::sync::Mutex::new(false), + }) + } + } + + #[async_trait] + impl EvalPod for CapturePod { + async fn boot(&self, _spec: &InstanceSpec) -> Result { + *self.booted.lock().expect("boot") = true; + Ok("pod-1".into()) + } + + async fn run( + &self, + _instance_id: &str, + request: &[u8], + env_file: &[u8], + ) -> Result { + *self.request.lock().expect("req") = request.to_vec(); + *self.env.lock().expect("env") = env_file.to_vec(); + Err("captured".into()) + } + + async fn shutdown(&self, _instance_id: &str) -> Result { + Ok(true) + } + } + + fn harvest_pin() -> ProofPin { + let mut p = ProofPin { + eval_image_digest: format!("sha256:{}", "ab".repeat(32)), + topic_pubkey: "ab".repeat(32), + ..ProofPin::default() + }; + p.inference.model = "master-proxy-v0".into(); + p + } + + fn harvest_offer() -> InferenceOffer { + let config = proof_task::InferenceConfig { + mode: proof_task::InferenceMode::Chat, + model_ref: "master-proxy-v0".into(), + max_input_tokens: 32_768, + max_output_tokens: 8_192, + temperature: Some(0.0), + top_p: None, + timeout_ms: None, + }; + InferenceOffer { + offer_id: "master-v0".into(), + provider: proof_task::InferenceProvider { + kind: proof_task::InferenceProviderKind::OpenaiCompatible, + base_url: "http://127.0.0.1:8000/v1".into(), + }, + config_commitment: proof_task::inference_config_commitment( + &config, + "http://127.0.0.1:8000/v1", + ), + config, + status: proof_task::OfferStatus::Open, + } + } + + fn harvest_topic(recs: &[proof_task::HoldoutRecord]) -> TopicDocument { + let mut b = default_adamw(proof_task::FLOPS_BUDGET_MAX); + b.script_sha256 = "11".repeat(32); + b.metrics_commitment = "22".repeat(32); + TopicDocument { + id: "dt-no-ib-v0".into(), + holdout_commitment: holdout_commitment(recs), + baseline: b, + status: proof_task::TopicStatus::Open, + ..TopicDocument::default() + } + } + + #[tokio::test] + async fn harvest_stages_teacher_env_and_never_puts_the_key_on_the_request() { + let recs = synthetic_holdout(STRATUM_SIZE, 1); + let topic = harvest_topic(&recs); + let pod = CapturePod::new(); + let harvest = LiumProofHarvest::new( + pod.clone(), + HarvestLimits::default(), + vec!["ssh-ed25519 AAAAtest proof".into()], + ) + .with_judge_api_key(Some("sk-live-not-a-real-secret".into())); + let err = harvest + .score( + &harvest_pin(), + &topic, + &harvest_offer(), + "digest-abcdef", + "artifact", + &recs, + "claim", + ) + .await + .expect_err("capture"); + assert!(matches!(err, EvalError::Backend(_)), "{err}"); + assert!(*pod.booted.lock().expect("booted")); + let env = String::from_utf8(pod.env.lock().expect("env").clone()).expect("utf8"); + assert!( + env.contains("OPENAI_API_KEY='sk-live-not-a-real-secret'"), + "{env}" + ); + assert!( + env.contains("PROOF_INFERENCE_API_KEY='sk-live-not-a-real-secret'"), + "{env}" + ); + let req: serde_json::Value = + serde_json::from_slice(&pod.request.lock().expect("req")).expect("json"); + assert!(req.get("api_key").is_none(), "{req}"); + let dump = req.to_string(); + assert!(!dump.contains("sk-live"), "{dump}"); + assert_eq!(req["base_url"], "http://127.0.0.1:8000/v1"); + } + + #[tokio::test] + async fn harvest_without_judge_key_does_not_boot() { + let recs = synthetic_holdout(STRATUM_SIZE, 1); + let topic = harvest_topic(&recs); + let pod = CapturePod::new(); + let harvest = LiumProofHarvest::new( + pod.clone(), + HarvestLimits::default(), + vec!["ssh-ed25519 AAAAtest proof".into()], + ); + let err = harvest + .score( + &harvest_pin(), + &topic, + &harvest_offer(), + "digest-abcdef", + "artifact", + &recs, + "claim", + ) + .await + .expect_err("no key"); + assert!(matches!(err, EvalError::InferenceAuthMissing), "{err}"); + assert!(!*pod.booted.lock().expect("booted")); + } + + #[tokio::test] + async fn harvest_refuses_a_spoofed_topic_origin_before_boot() { + let recs = synthetic_holdout(STRATUM_SIZE, 1); + let mut topic = harvest_topic(&recs); + topic.inference.max_input_tokens = Some(4_096); + topic.inference.base_url = Some("http://evil.example/v1".into()); + let pod = CapturePod::new(); + let harvest = LiumProofHarvest::new( + pod.clone(), + HarvestLimits::default(), + vec!["ssh-ed25519 AAAAtest proof".into()], + ) + .with_judge_api_key(Some("sk-live-not-a-real-secret".into())); + let err = harvest + .score( + &harvest_pin(), + &topic, + &harvest_offer(), + "digest-abcdef", + "artifact", + &recs, + "claim", + ) + .await + .expect_err("spoof"); + assert!(err.to_string().contains("committed judge origin"), "{err}"); + assert!(!*pod.booted.lock().expect("booted")); } } diff --git a/crates/proof-holdout/Cargo.toml b/crates/proof-holdout/Cargo.toml new file mode 100644 index 000000000..15e138a7d --- /dev/null +++ b/crates/proof-holdout/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "proof-holdout" +description = "Proof holdout records, commitments, and stratification" +version.workspace = true +edition.workspace = true +license.workspace = true +repository.workspace = true +rust-version.workspace = true +publish = false + +[dependencies] +hex = "0.4" +serde = { version = "1", features = ["derive"] } +sha2 = "0.10" +thiserror = "2" + +[lints] +workspace = true diff --git a/crates/proof-task/src/holdout.rs b/crates/proof-holdout/src/lib.rs similarity index 95% rename from crates/proof-task/src/holdout.rs rename to crates/proof-holdout/src/lib.rs index 9d7bfb082..5a5a630ef 100644 --- a/crates/proof-task/src/holdout.rs +++ b/crates/proof-holdout/src/lib.rs @@ -14,13 +14,31 @@ //! that visible. [`HoldoutSplit::CanaryOffpath`] never enters the 120 and //! never enters the score. +#![allow( + clippy::doc_markdown, + clippy::module_name_repetitions, + clippy::must_use_candidate +)] + use std::collections::{BTreeMap, BTreeSet}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use thiserror::Error; -use crate::HOLDOUT_DOMAIN; +/// Domain tag for per-topic holdout commitments. +pub const HOLDOUT_DOMAIN: &[u8] = b"base-proof-holdout-v1"; + +/// Holdout records per topic. +pub const HOLDOUT_SIZE: usize = 120; + +/// Records per scored split (`HOLDOUT_SIZE / scored splits`). +pub const STRATUM_SIZE: usize = 24; + +fn is_hex64(s: &str) -> bool { + let t = s.trim(); + t.len() == 64 && t.chars().all(|c| c.is_ascii_hexdigit()) +} /// Smallest packed sequence length a `longctx` record may carry. pub const LONGCTX_MIN_TOKENS: u32 = 8_192; @@ -48,7 +66,6 @@ pub enum HoldoutSplit { } impl HoldoutSplit { - /// Splits that enter the paid metric, in stable order. pub const SCORED: [Self; 5] = [ Self::WebOod, Self::CodeOod, @@ -57,8 +74,6 @@ impl HoldoutSplit { Self::MultilingualOod, ]; - /// Wire name. - #[must_use] pub const fn as_str(self) -> &'static str { match self { Self::WebOod => "web_ood", @@ -70,8 +85,6 @@ impl HoldoutSplit { } } - /// Whether this stratum may contribute to the paid metric. - #[must_use] pub const fn is_scored(self) -> bool { !matches!(self, Self::CanaryOffpath) } @@ -93,17 +106,10 @@ pub struct HoldoutRecord { } impl HoldoutRecord { - /// Canonical fingerprint used by the contamination gate. - #[must_use] pub fn fingerprint(&self) -> String { format!("shard:{}", self.content_sha256.to_ascii_lowercase()) } - /// A minimal valid record, for CI and local catalogs. - /// - /// Production must use a private catalog: a documented generator over a - /// public corpus would let a miner rebuild the scored shards. - #[must_use] pub fn synthetic(id: u32, split: HoldoutSplit) -> Self { let mut h = Sha256::new(); h.update(b"proof-synthetic-shard-v1"); @@ -194,7 +200,6 @@ fn field(h: &mut Sha256, value: &str) { /// Domain-separated, id-sorted, length-prefixed, and covering every field the /// harness reads — stratum and token count included, so a "verified" holdout /// cannot be re-labelled to move records between strata after the fact. -#[must_use] pub fn holdout_commitment(records: &[HoldoutRecord]) -> String { let mut sorted: Vec<&HoldoutRecord> = records.iter().collect(); sorted.sort_by_key(|r| r.id); @@ -216,11 +221,6 @@ pub fn holdout_commitment(records: &[HoldoutRecord]) -> String { hex::encode(h.finalize()) } -fn hex64(s: &str) -> bool { - let t = s.trim(); - t.len() == 64 && t.chars().all(|c| c.is_ascii_hexdigit()) -} - fn validate(records: &[HoldoutRecord]) -> Result<(), HoldoutError> { if records.is_empty() { return Err(HoldoutError::Empty); @@ -234,7 +234,7 @@ fn validate(records: &[HoldoutRecord]) -> Result<(), HoldoutError> { if r.dataset_id.trim().is_empty() { return Err(HoldoutError::EmptyDataset(r.id)); } - if !hex64(&r.content_sha256) { + if !is_hex64(&r.content_sha256) { return Err(HoldoutError::MalformedHash(r.id)); } if r.token_count == 0 { @@ -315,7 +315,6 @@ pub fn verify_holdout( /// Matching is on shard content hashes and corpus ids, because those are the /// two things a miner can honestly declare and the two things that make a /// holdout NLL meaningless if they overlap. -#[must_use] pub fn contamination( declared_content_hashes: &BTreeSet, declared_dataset_ids: &BTreeSet, @@ -343,7 +342,6 @@ pub fn contamination( } /// A stratified synthetic holdout of `per_split * 5` records (CI / local only). -#[must_use] pub fn synthetic_holdout(per_split: usize, first_id: u32) -> Vec { let mut out = Vec::new(); let mut id = first_id; @@ -359,7 +357,6 @@ pub fn synthetic_holdout(per_split: usize, first_id: u32) -> Vec #[cfg(test)] mod tests { use super::*; - use crate::{HOLDOUT_SIZE, STRATUM_SIZE}; fn holdout() -> Vec { synthetic_holdout(STRATUM_SIZE, 1_000) diff --git a/crates/proof-http/src/lib.rs b/crates/proof-http/src/lib.rs index a7f73831e..6fb6e153d 100644 --- a/crates/proof-http/src/lib.rs +++ b/crates/proof-http/src/lib.rs @@ -16,7 +16,8 @@ clippy::missing_errors_doc, clippy::doc_markdown, clippy::must_use_candidate, - clippy::too_many_lines + clippy::too_many_lines, + clippy::too_many_arguments )] use std::sync::Arc; @@ -28,8 +29,8 @@ use axum::routing::{get, post}; use axum::{Json, Router}; use proof_eval::{ - contamination_evidence, eval_after_freeze, force_sim, scoring_readiness, supported_custom, - EvalBackend, EvalError, LiveScorer, + contamination_evidence, eval_after_freeze, force_sim, scoring_readiness, + secret_backed_base_url, supported_custom, EvalBackend, EvalError, LiveScorer, }; use proof_score::{ judge_topic, primary_from_harness, AgentVerdict, GateFail, HarnessMetrics, MinerTopicRun, @@ -39,7 +40,8 @@ use proof_store::{ freeze_submission_digest, ArtifactManifest, MemoryStore, Submission, SubmissionState, }; use proof_task::{ - ProofPin, TopicDocument, TopicError, TopicStatus, CHALLENGE_ID, SCORE_MAX, SCORING_VERSION, + resolve_inference, InferenceOffer, OfferError, ProofPin, TopicDocument, TopicError, + TopicStatus, CHALLENGE_ID, SCORE_MAX, SCORING_VERSION, }; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; @@ -56,6 +58,10 @@ pub struct AppState { /// Harvest handle for the digest-pinned eval image. `None` on a live host /// means nothing can score, so submissions refuse. pub live_scorer: Option>, + /// Live RLM judge backend (operator state). Missing/closed → can_score false. + pub offer: Option, + /// Judge API key from `PROOF_INFERENCE_API_KEY_FILE`. Never on `/v1/status`. + pub judge_api_key: Option, /// Operator bearer hashes (sha256 hex). Empty → admin 503. pub admin_hashes: Arc>, /// Chain epoch used for topic windows. v0 hosts pass 0. @@ -69,7 +75,29 @@ impl AppState { fn can_score(&self) -> bool { let open = self.store.any_open_scorable(self.epoch).unwrap_or(false); - scoring_readiness(&self.pin, self.backend, self.live(), open).is_ok() + if scoring_readiness( + &self.pin, + self.backend, + self.live(), + open, + self.offer.as_ref(), + self.judge_api_key.as_deref(), + ) + .is_err() + { + return false; + } + let secret = secret_backed_base_url(); + self.store.topics().unwrap_or_default().iter().any(|t| { + t.is_open_at(self.epoch) + && resolve_inference( + &self.pin, + Some(&t.inference), + secret.as_deref(), + self.offer.as_ref(), + ) + .ready_to_score() + }) } } @@ -103,7 +131,14 @@ async fn status(State(st): State) -> impl IntoResponse { "score_max": SCORE_MAX, "eval_image": st.pin.eval_image, "eval_image_digest": st.pin.eval_image_digest, - "proxy_model": st.pin.proxy_model, + "inference_offer": st.offer.as_ref().map(InferenceOffer::public_view), + "inference": { + "provider": st.pin.inference.provider.as_str(), + "model": st.pin.inference.model, + "mode": st.pin.inference.mode.as_str(), + "max_input_tokens": st.pin.inference.max_input_tokens, + "max_output_tokens": st.pin.inference.max_output_tokens, + }, "eval_backend": st.backend, "force_sim": force_sim(), "can_score": st.can_score(), @@ -141,6 +176,7 @@ struct SubmitBody { declared_flops: u64, #[serde(default)] topic_id: String, + /// Optional miner label. Not compared to an HF id (that check is retired). #[serde(default)] architecture: String, #[serde(default)] @@ -196,17 +232,6 @@ async fn submit( if !topic.is_open_at(st.epoch) { return Err(err(StatusCode::BAD_REQUEST, "topic is not open")); } - - let want_proxy = st.pin.proxy_for(topic.proxy_model.as_deref()); - if want_proxy.is_empty() { - return Err(err(StatusCode::BAD_REQUEST, "proxy not baked")); - } - if body.architecture.trim() != want_proxy { - return Err(err( - StatusCode::BAD_REQUEST, - "architecture is not the topic/pin proxy", - )); - } if body.declared_flops > topic.flops_budget { return Err(err( StatusCode::BAD_REQUEST, @@ -222,8 +247,25 @@ async fn submit( st.backend, st.live(), st.store.any_open_scorable(st.epoch).unwrap_or(false), + st.offer.as_ref(), + st.judge_api_key.as_deref(), ) .map_err(|e| eval_err(&e))?; + let Some(offer) = st.offer.as_ref() else { + return Err(eval_err(&EvalError::InferenceOfferMissing)); + }; + offer + .serves_topic(&st.pin, &topic) + .map_err(|e| offer_err(&e))?; + let resolved = resolve_inference( + &st.pin, + Some(&topic.inference), + secret_backed_base_url().as_deref(), + Some(offer), + ); + if !resolved.ready_to_score() { + return Err(offer_err(&OfferError::Incomplete)); + } let sealed = st .store @@ -265,12 +307,14 @@ async fn submit( let eval = eval_after_freeze( &st.pin, &topic, + offer, &submission_digest, &artifact, &holdout, &body.claim, st.backend, st.live(), + st.judge_api_key.as_deref(), ) .await .map_err(|e| eval_err(&e))?; @@ -338,6 +382,16 @@ fn persist_pre_eval_reject( claim: body.claim, declared_flops: body.declared_flops, architecture: body.architecture, + inference_offer_id: st + .offer + .as_ref() + .map(|o| o.offer_id.clone()) + .unwrap_or_default(), + config_commitment: st + .offer + .as_ref() + .map(|o| o.config_commitment.clone()) + .unwrap_or_default(), manifest: body.manifest, nonce, submission_digest, @@ -406,6 +460,16 @@ fn persist_scored( claim: body.claim, declared_flops: body.declared_flops, architecture: body.architecture, + inference_offer_id: st + .offer + .as_ref() + .map(|o| o.offer_id.clone()) + .unwrap_or_default(), + config_commitment: st + .offer + .as_ref() + .map(|o| o.config_commitment.clone()) + .unwrap_or_default(), manifest: body.manifest, nonce, submission_digest, @@ -516,6 +580,10 @@ fn topic_err(e: &TopicError) -> (StatusCode, Json) { err(code, &e.to_string()) } +fn offer_err(e: &OfferError) -> (StatusCode, Json) { + err(StatusCode::SERVICE_UNAVAILABLE, &e.to_string()) +} + fn eval_err(e: &EvalError) -> (StatusCode, Json) { let code = match e { EvalError::Integrity(_) => StatusCode::INTERNAL_SERVER_ERROR, @@ -542,9 +610,11 @@ mod tests { use http_body_util::BodyExt; use proof_eval::{sim_document, BaselineMeasurement, BASELINE_SKILL}; use proof_task::{ - default_adamw, holdout_commitment, synthetic_holdout, Constraints, MetricDirection, - MetricFamily, MetricSpec, TopicDocument, TopicStatus, FLOPS_BUDGET_MAX, HOLDOUT_SIZE, - METRIC_TOKENS_PER_SEC, STRATUM_SIZE, + default_adamw, holdout_commitment, inference_config_commitment, synthetic_holdout, + Constraints, InferenceConfig, InferenceMode, InferenceOffer, InferenceProvider, + InferenceProviderKind, MetricDirection, MetricFamily, MetricSpec, OfferStatus, + TopicDocument, TopicStatus, FLOPS_BUDGET_MAX, HOLDOUT_SIZE, METRIC_TOKENS_PER_SEC, + STRATUM_SIZE, }; use tower::ServiceExt; @@ -567,12 +637,34 @@ mod tests { } fn pin(digest: &str) -> ProofPin { - ProofPin { + let mut p = ProofPin { eval_image_digest: digest.to_owned(), topic_pubkey: pk_hex(), - proxy_model: "Qwen/Qwen3.8-0.6B".into(), - proxy_models: vec!["Qwen/Qwen3.8-0.6B".into()], ..ProofPin::default() + }; + p.inference.model = "master-proxy-v0".into(); + p + } + + fn offer() -> InferenceOffer { + let config = InferenceConfig { + mode: InferenceMode::Chat, + model_ref: "master-proxy-v0".into(), + max_input_tokens: 32_768, + max_output_tokens: 8_192, + temperature: Some(0.0), + top_p: None, + timeout_ms: None, + }; + InferenceOffer { + offer_id: "master-v0".into(), + provider: InferenceProvider { + kind: InferenceProviderKind::OpenaiCompatible, + base_url: "http://127.0.0.1:8000/v1".into(), + }, + config_commitment: inference_config_commitment(&config, "http://127.0.0.1:8000/v1"), + config, + status: OfferStatus::Open, } } @@ -662,6 +754,7 @@ mod tests { &self, pin: &ProofPin, topic: &TopicDocument, + _offer: &InferenceOffer, frozen: &str, artifact: &str, _holdout: &[proof_task::HoldoutRecord], @@ -693,6 +786,7 @@ mod tests { live: Option>, load: bool, baseline: bool, + with_offer: bool, ) -> Router { let p = pin(eval_digest); let store = MemoryStore::new(); @@ -707,18 +801,24 @@ mod tests { .expect("baseline"); } } + let judge_api_key = + (backend == EvalBackend::Lium && live.is_some()).then(|| "test-judge-key".to_owned()); proof_router(AppState { store, pin: p, backend, live_scorer: live, + offer: with_offer.then(offer), + // Lium + a wired harvest is the live path: a missing key is the + // Testeur blocker. Sim does not call the judge, so it stays None. + judge_api_key, admin_hashes: Arc::new(vec![hash_admin_token(token)]), epoch: 0, }) } fn app(token: &str) -> Router { - app_full(token, EvalBackend::Sim, "", None, true, true) + app_full(token, EvalBackend::Sim, "", None, true, true, true) } async fn json_req( @@ -750,7 +850,6 @@ mod tests { "claim": "beats the sealed reference under the cap", "declared_flops": FLOPS_BUDGET_MAX / 2, "topic_id": "dt-no-ib-v0", - "architecture": "Qwen/Qwen3.8-0.6B", "manifest": { "train_dataset_ids": ["public-pretrain-v0"] }, @@ -781,6 +880,15 @@ mod tests { let dump = body.to_string(); assert!(!dump.contains("synthetic-dev"), "{dump}"); assert!(!dump.contains("holdout_nll")); + assert_eq!(body["inference_offer"]["offer_id"], "master-v0"); + assert_eq!(body["inference_offer"]["status"], "open"); + assert_eq!(body["inference"]["provider"], "openai_compatible"); + assert_eq!(body["inference"]["mode"], "chat"); + assert_eq!(body["inference"]["model"], "master-proxy-v0"); + assert!(!dump.contains("8000"), "{dump}"); + assert!(!dump.contains("base_url"), "{dump}"); + assert!(!dump.contains("api_key"), "{dump}"); + assert!(!dump.contains("evil.example"), "{dump}"); } #[tokio::test] @@ -836,26 +944,6 @@ mod tests { assert_eq!(st, StatusCode::BAD_REQUEST, "{body}"); } - #[tokio::test] - async fn architecture_must_match_the_proxy() { - let (st, body) = json_req( - app("op"), - "POST", - "/v1/submissions", - submit_body( - "x", - &serde_json::json!({ "architecture": "meta-llama/Llama-3-8B" }), - ), - None, - ) - .await; - assert_eq!(st, StatusCode::BAD_REQUEST, "{body}"); - assert!(body["error"] - .as_str() - .unwrap_or_default() - .contains("architecture")); - } - #[tokio::test] async fn reproduced_true_scores_and_false_zeros() { let win = Arc::new(StubScorer::win()); @@ -863,7 +951,15 @@ mod tests { let digest = format!("sha256:{}", "ab".repeat(32)); let (st, created) = json_req( - app_full("op", EvalBackend::Lium, &digest, Some(win), true, true), + app_full( + "op", + EvalBackend::Lium, + &digest, + Some(win), + true, + true, + true, + ), "POST", "/v1/submissions", submit_body("miner-strong-proof", &serde_json::json!({})), @@ -876,7 +972,15 @@ mod tests { assert_eq!(created["state"], "awaiting_admin"); let (st, created) = json_req( - app_full("op", EvalBackend::Lium, &digest, Some(lose), true, true), + app_full( + "op", + EvalBackend::Lium, + &digest, + Some(lose), + true, + true, + true, + ), "POST", "/v1/submissions", submit_body("miner-unreproduced", &serde_json::json!({})), @@ -900,6 +1004,7 @@ mod tests { Some(scorer.clone()), true, true, + true, ); for manifest in [ serde_json::json!({}), @@ -929,7 +1034,7 @@ mod tests { #[tokio::test] async fn empty_digest_and_unwired_harvest_are_503() { let (st, body) = json_req( - app_full("op", EvalBackend::Lium, "", None, true, true), + app_full("op", EvalBackend::Lium, "", None, true, true, true), "POST", "/v1/submissions", submit_body("x", &serde_json::json!({})), @@ -950,6 +1055,7 @@ mod tests { None, true, true, + true, ), "POST", "/v1/submissions", @@ -966,7 +1072,7 @@ mod tests { #[tokio::test] async fn refused_submissions_leave_no_rows() { - let unpinned = app_full("op", EvalBackend::Lium, "", None, true, true); + let unpinned = app_full("op", EvalBackend::Lium, "", None, true, true, true); let no_baseline = app_full( "op", EvalBackend::Lium, @@ -974,8 +1080,9 @@ mod tests { Some(Arc::new(StubScorer::win())), true, false, + true, ); - let sealed = app_full("op", EvalBackend::Sim, "", None, false, false); + let sealed = app_full("op", EvalBackend::Sim, "", None, false, false, true); for (label, app) in [ ("unpinned digest", unpinned), @@ -1009,7 +1116,7 @@ mod tests { #[tokio::test] async fn can_score_is_false_until_open_holdout_and_baseline() { let (st, body) = json_req( - app_full("op", EvalBackend::Sim, "", None, false, false), + app_full("op", EvalBackend::Sim, "", None, false, false, true), "GET", "/v1/status", serde_json::json!({}), @@ -1028,6 +1135,7 @@ mod tests { Some(Arc::new(StubScorer::win())), true, false, + true, ), "GET", "/v1/status", @@ -1040,6 +1148,35 @@ mod tests { assert_eq!(body["can_score"], false, "{body}"); } + #[tokio::test] + async fn missing_or_closed_offer_is_503() { + let (st, body) = json_req( + app_full("op", EvalBackend::Sim, "", None, true, true, false), + "GET", + "/v1/status", + serde_json::json!({}), + None, + ) + .await; + assert_eq!(st, StatusCode::OK); + assert_eq!(body["can_score"], false, "{body}"); + assert!(body["inference_offer"].is_null(), "{body}"); + + let (st, body) = json_req( + app_full("op", EvalBackend::Sim, "", None, true, true, false), + "POST", + "/v1/submissions", + submit_body("x", &serde_json::json!({})), + None, + ) + .await; + assert_eq!(st, StatusCode::SERVICE_UNAVAILABLE, "{body}"); + assert!(body["error"] + .as_str() + .unwrap_or_default() + .contains("inference offer")); + } + #[tokio::test] async fn admin_publish_requires_bearer_and_a_valid_signature() { let token = "op-test-token"; @@ -1136,4 +1273,130 @@ mod tests { assert_eq!(body["id"], "agent-harness-improve-v0"); assert_eq!(body["payout_mode"], "discovery"); } + + fn app_lium_missing_judge_key() -> Router { + let p = pin(&format!("sha256:{}", "ab".repeat(32))); + let store = MemoryStore::new(); + let recs = synthetic_holdout(STRATUM_SIZE, 1); + let (topic, meas) = seal_topic(&p, unsigned_topic(&recs)); + store.put_topic(topic.clone()).expect("topic"); + store.load_holdout(&topic.id, recs).expect("holdout"); + store + .set_baseline(&topic.id, meas.into_sealed()) + .expect("baseline"); + proof_router(AppState { + store, + pin: p, + backend: EvalBackend::Lium, + live_scorer: Some(Arc::new(StubScorer::win())), + offer: Some(offer()), + judge_api_key: None, + admin_hashes: Arc::new(vec![hash_admin_token("op")]), + epoch: 0, + }) + } + + #[tokio::test] + async fn lium_without_judge_api_key_cannot_score() { + let (st, body) = json_req( + app_lium_missing_judge_key(), + "GET", + "/v1/status", + serde_json::json!({}), + None, + ) + .await; + assert_eq!(st, StatusCode::OK); + assert_eq!(body["live_harvest_wired"], true); + assert_eq!(body["can_score"], false, "{body}"); + let dump = body.to_string(); + assert!(!dump.contains("api_key"), "{dump}"); + + let (st, body) = json_req( + app_lium_missing_judge_key(), + "POST", + "/v1/submissions", + submit_body("no-key", &serde_json::json!({})), + None, + ) + .await; + assert_eq!(st, StatusCode::SERVICE_UNAVAILABLE, "{body}"); + assert!( + body["error"] + .as_str() + .unwrap_or_default() + .contains("API key"), + "{body}" + ); + } + + #[tokio::test] + async fn lium_with_judge_api_key_can_score_and_status_omits_the_secret() { + let digest = format!("sha256:{}", "ab".repeat(32)); + let (st, body) = json_req( + app_full( + "op", + EvalBackend::Lium, + &digest, + Some(Arc::new(StubScorer::win())), + true, + true, + true, + ), + "GET", + "/v1/status", + serde_json::json!({}), + None, + ) + .await; + assert_eq!(st, StatusCode::OK); + assert_eq!(body["can_score"], true, "{body}"); + let dump = body.to_string(); + assert!(!dump.contains("api_key"), "{dump}"); + assert!(!dump.contains("test-judge-key"), "{dump}"); + assert!(!dump.contains("base_url"), "{dump}"); + assert!(!dump.contains("8000"), "{dump}"); + } + + #[tokio::test] + async fn spoofed_topic_origin_is_503() { + let p = pin(""); + let store = MemoryStore::new(); + let recs = synthetic_holdout(STRATUM_SIZE, 1); + let (mut topic, meas) = seal_topic(&p, unsigned_topic(&recs)); + topic.inference.max_input_tokens = Some(4_096); + topic.inference.base_url = Some("http://evil.example/v1".into()); + topic.signature = topic.sign_with(&sk()).expect("sign"); + store.put_topic(topic.clone()).expect("topic"); + store.load_holdout(&topic.id, recs).expect("holdout"); + store + .set_baseline(&topic.id, meas.into_sealed()) + .expect("baseline"); + let app = proof_router(AppState { + store, + pin: p, + backend: EvalBackend::Sim, + live_scorer: None, + offer: Some(offer()), + judge_api_key: None, + admin_hashes: Arc::new(vec![hash_admin_token("op")]), + epoch: 0, + }); + let (st, body) = json_req( + app, + "POST", + "/v1/submissions", + submit_body("spoof", &serde_json::json!({})), + None, + ) + .await; + assert_eq!(st, StatusCode::SERVICE_UNAVAILABLE, "{body}"); + assert!( + body["error"] + .as_str() + .unwrap_or_default() + .contains("committed judge origin"), + "{body}" + ); + } } diff --git a/crates/proof-store/src/lib.rs b/crates/proof-store/src/lib.rs index c78aa8716..1be8cb37f 100644 --- a/crates/proof-store/src/lib.rs +++ b/crates/proof-store/src/lib.rs @@ -73,8 +73,15 @@ pub struct Submission { pub claim: String, /// Declared FLOP budget (must be ≤ topic). pub declared_flops: u64, - /// Architecture id; must equal the topic/pin proxy. + /// Deprecated HF architecture id. Ignored; not an architecture lock. + #[serde(default)] pub architecture: String, + /// RLM judge offer id that scored this run (host stamp, not a miner bind). + #[serde(default)] + pub inference_offer_id: String, + /// Judge `config_commitment` stamped from the host offer. + #[serde(default)] + pub config_commitment: String, /// Declared training fingerprints. #[serde(default)] pub manifest: ArtifactManifest, diff --git a/crates/proof-task/Cargo.toml b/crates/proof-task/Cargo.toml index b83d03851..95b780c59 100644 --- a/crates/proof-task/Cargo.toml +++ b/crates/proof-task/Cargo.toml @@ -11,6 +11,7 @@ publish = false [dependencies] crypto = { path = "../crypto" } hex = "0.4" +proof-holdout = { path = "../proof-holdout" } serde = { version = "1", features = ["derive"] } serde_json = "1" sha2 = "0.10" diff --git a/crates/proof-task/src/canonical.rs b/crates/proof-task/src/canonical.rs index 4655a02b9..7839b6eae 100644 --- a/crates/proof-task/src/canonical.rs +++ b/crates/proof-task/src/canonical.rs @@ -19,7 +19,6 @@ use serde_json::Value; /// /// Sorted keys, no whitespace. `Value`'s map is already ordered, but the sort /// is explicit so the rule does not depend on a serde_json feature flag. -#[must_use] pub fn canonical_json(value: &Value) -> String { match value { Value::Null => "null".into(), diff --git a/crates/proof-task/src/inference.rs b/crates/proof-task/src/inference.rs new file mode 100644 index 000000000..0a7b4d7e1 --- /dev/null +++ b/crates/proof-task/src/inference.rs @@ -0,0 +1,723 @@ +//! Master-owned RLM **judge** backend: pin defaults in git, topic tighten, +//! live InferenceOffer off git. +//! +//! The digest-pinned eval image (and harvest, which boots it) call this +//! provider as the RLM **judge** backend. Miners submit claim + code + FLOPs + +//! artifact against a topic; they do **not** bind this offer. No baked Qwen. + +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use crate::{canonical_json, is_hex64, is_http_origin, ProofPin, TopicDocument, TopicError}; + +/// Pin `inference_config_schema_version`. +pub const INFERENCE_CONFIG_SCHEMA_VERSION: u32 = 1; + +/// Pin `inference_offer_commitment_alg`. +pub const INFERENCE_OFFER_COMMITMENT_ALG: &str = "sha256"; + +/// Pin / crate ceiling on prompt tokens (covers longctx 32k). +pub const MAX_INPUT_TOKENS_CEILING: u32 = 32_768; + +/// Pin / crate ceiling on completion tokens. +pub const MAX_OUTPUT_TOKENS_CEILING: u32 = 8_192; + +/// Modes the pin may allow. A pin may subset; it cannot add unknown names. +pub const ALLOWED_MODES: [InferenceMode; 3] = [ + InferenceMode::Chat, + InferenceMode::Completions, + InferenceMode::Embeddings, +]; + +/// Wire name of a scoring / serving mode. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum InferenceMode { + /// Chat completions. + Chat, + /// Text completions. + Completions, + /// Embeddings. + Embeddings, +} + +impl InferenceMode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Chat => "chat", + Self::Completions => "completions", + Self::Embeddings => "embeddings", + } + } +} + +/// How the master exposes the provider. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum InferenceProviderKind { + /// OpenAI-compatible HTTP (`/v1/chat/completions`, …). + OpenaiCompatible, + /// vLLM OpenAI-compatible server. + Vllm, + /// Operator-defined HTTP shape the eval image already knows. + Custom, +} + +impl InferenceProviderKind { + pub const fn as_str(self) -> &'static str { + match self { + Self::OpenaiCompatible => "openai_compatible", + Self::Vllm => "vllm", + Self::Custom => "custom", + } + } +} + +/// Live offer lifecycle. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum OfferStatus { + /// Judge backend is live; required for `can_score`. + Open, + /// Host cannot score. + Closed, +} + +/// Provider identity. `base_url` is operator state — never on public status. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InferenceProvider { + /// `openai_compatible` | `vllm` | `custom`. + pub kind: InferenceProviderKind, + /// Provider origin the eval image calls. Not a git pin. + pub base_url: String, +} + +/// Public, secret-free knobs hashed into `config_commitment`. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InferenceConfig { + /// Serving mode. + pub mode: InferenceMode, + /// Provider model id (not an HF bake into `proof-eval`). + pub model_ref: String, + /// Prompt token cap for this offer. + pub max_input_tokens: u32, + /// Completion token cap for this offer. + pub max_output_tokens: u32, + /// Sampling temperature. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub temperature: Option, + /// Nucleus sampling. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub top_p: Option, + /// Provider timeout. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub timeout_ms: Option, +} + +/// Operator live offer (`PROOF_INFERENCE_OFFER_FILE`). Never committed. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InferenceOffer { + /// Immutable slug identifying this judge backend. + pub offer_id: String, + /// Provider kind + origin. + pub provider: InferenceProvider, + /// Secret-free config. + pub config: InferenceConfig, + /// `sha256` hex of canonical JSON of config knobs **plus** `provider.base_url`. + pub config_commitment: String, + /// `open` | `closed`. + pub status: OfferStatus, +} + +/// Pin `[inference]` defaults. Empty `model` / `base_url` is pre-launch +/// fail-closed (like an empty digest), not a boot reject. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(default)] +pub struct PinInference { + pub provider: InferenceProviderKind, + pub base_url: String, + pub model: String, + pub mode: InferenceMode, + pub max_input_tokens: u32, + pub max_output_tokens: u32, +} + +impl Default for PinInference { + fn default() -> Self { + Self { + provider: InferenceProviderKind::OpenaiCompatible, + base_url: String::new(), + model: String::new(), + mode: InferenceMode::Chat, + max_input_tokens: MAX_INPUT_TOKENS_CEILING, + max_output_tokens: MAX_OUTPUT_TOKENS_CEILING, + } + } +} + +impl PinInference { + pub fn ready_to_score(&self) -> bool { + !self.model.trim().is_empty() + && self.max_input_tokens > 0 + && self.max_output_tokens > 0 + && is_http_origin(&self.base_url) + } +} + +/// Topic override. Omitted fields inherit the pin; tokens may only tighten. +#[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields, default)] +pub struct TopicInference { + /// Optional 64-hex pin of the live judge offer `config_commitment`. + /// Not a miner-facing bind. + pub require_judge_offer_commitment: Option, + pub provider: Option, + pub base_url: Option, + pub model: Option, + pub mode: Option, + pub max_input_tokens: Option, + pub max_output_tokens: Option, +} + +/// Topic may tighten tokens / name model+mode. **Origin is the committed +/// offer URL** — pin, secret, and topic cannot redirect the RLM after +/// `serves_topic`. +pub fn resolve_inference( + pin: &ProofPin, + topic: Option<&TopicInference>, + secret_url: Option<&str>, + offer: Option<&InferenceOffer>, +) -> PinInference { + let d = &pin.inference; + let offer_url = offer.map_or("", |o| o.provider.base_url.as_str()); + let fallback = [d.base_url.as_str(), secret_url.unwrap_or("")] + .into_iter() + .find(|u| is_http_origin(u)) + .unwrap_or(""); + let base_url = if is_http_origin(offer_url) { + offer_url + } else { + fallback + } + .to_owned(); + PinInference { + provider: topic.and_then(|t| t.provider).unwrap_or(d.provider), + base_url, + model: topic + .and_then(|t| t.model.as_deref()) + .map(str::trim) + .filter(|s| !s.is_empty()) + .unwrap_or(d.model.trim()) + .to_owned(), + mode: topic.and_then(|t| t.mode).unwrap_or(d.mode), + max_input_tokens: topic + .and_then(|t| t.max_input_tokens) + .unwrap_or(d.max_input_tokens), + max_output_tokens: topic + .and_then(|t| t.max_output_tokens) + .unwrap_or(d.max_output_tokens), + } +} + +impl TopicInference { + /// Pin allowlist, token tighten-only, and usable override shape. + /// + /// # Errors + /// + /// [`TopicError::InferenceModeNotAllowed`], [`TopicError::InferenceCeiling`], + /// [`TopicError::IncompleteInference`], or [`TopicError::BadOfferCommitment`]. + pub fn validate(&self, pin: &ProofPin) -> Result<(), TopicError> { + if let Some(mode) = self.mode { + if !pin.allows_mode(mode) { + return Err(TopicError::InferenceModeNotAllowed(mode)); + } + } + let pin_in = pin + .inference + .max_input_tokens + .min(pin.max_input_tokens_ceiling); + let pin_out = pin + .inference + .max_output_tokens + .min(pin.max_output_tokens_ceiling); + for (field, got, ceiling) in [ + ("max_input_tokens", self.max_input_tokens, pin_in), + ("max_output_tokens", self.max_output_tokens, pin_out), + ] { + if let Some(got) = got { + if got == 0 || got > ceiling { + return Err(TopicError::InferenceCeiling(field, got, ceiling)); + } + } + } + if self.base_url.as_deref().is_some_and(|u| !is_http_origin(u)) + || self + .model + .as_deref() + .is_some_and(|m| m.trim().is_empty() || m.len() > 256) + { + return Err(TopicError::IncompleteInference); + } + if let Some(need) = self.require_judge_offer_commitment.as_deref() { + if !is_hex64(need) { + return Err(TopicError::BadOfferCommitment); + } + } + Ok(()) + } +} + +/// Why an offer was refused or cannot score. +#[derive(Debug, Clone, PartialEq, thiserror::Error)] +pub enum OfferError { + /// JSON did not parse, or carried an unknown key. + #[error("parse inference offer: {0}")] + Parse(String), + /// `offer_id` is not a slug. + #[error("offer_id {0:?} must match [a-z0-9][a-z0-9-]{{1,62}}")] + BadId(String), + /// Origin missing or not http(s). + #[error("provider.base_url must be an http(s) origin")] + BadBaseUrl, + /// `model_ref` empty. + #[error("config.model_ref is required")] + BadModelRef, + /// Token cap is zero or above the pin ceiling. + #[error("config.{0} = {1} must be 1..={2}")] + BadTokenCap(&'static str, u32, u32), + /// Sampling knob is non-finite or out of range. + #[error("config.{0} is not a usable sampling value")] + BadSampling(&'static str), + /// Mode not in the pin allowlist. + #[error("config.mode {0:?} is not in the pin allowed_modes")] + ModeNotAllowed(InferenceMode), + /// Declared commitment is not 64 hex or does not match config+origin. + #[error("config_commitment does not match sha256(canonical config+origin)")] + CommitmentMismatch, + /// Topic `inference.base_url` is not the committed judge origin. + #[error("topic inference.base_url does not match the committed judge origin")] + OriginMismatch, + /// No offer loaded on this host. + #[error("inference offer missing; refuse scoring")] + Missing, + /// Offer is present but closed. + #[error("inference offer is closed; refuse scoring")] + Closed, + /// Open offer cannot serve this topic. + #[error("open inference offer cannot serve topic inference constraints")] + CannotServeTopic, + /// Resolved pin+topic config is missing model or http(s) origin. + #[error("resolved inference config is incomplete; refuse scoring")] + Incomplete, +} + +#[derive(Serialize)] +struct CommitmentMaterial<'a> { + base_url: &'a str, + max_input_tokens: u32, + max_output_tokens: u32, + mode: InferenceMode, + model_ref: &'a str, + #[serde(skip_serializing_if = "Option::is_none")] + temperature: Option, + #[serde(skip_serializing_if = "Option::is_none")] + timeout_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + top_p: Option, +} + +/// `sha256` hex of canonical JSON of config knobs **and** the judge origin. +/// +/// Secrets never enter this digest. Public `/v1/status` still omits `base_url`. +pub fn inference_config_commitment(config: &InferenceConfig, origin: &str) -> String { + let material = CommitmentMaterial { + base_url: origin.trim(), + max_input_tokens: config.max_input_tokens, + max_output_tokens: config.max_output_tokens, + mode: config.mode, + model_ref: config.model_ref.trim(), + temperature: config.temperature, + timeout_ms: config.timeout_ms, + top_p: config.top_p, + }; + let value = serde_json::to_value(&material).unwrap_or(serde_json::Value::Null); + let body = canonical_json(&value); + let mut h = Sha256::new(); + h.update(body.as_bytes()); + hex::encode(h.finalize()) +} + +impl InferenceOffer { + /// Parse one operator offer document. + /// + /// # Errors + /// + /// [`OfferError::Parse`] on malformed JSON or an unknown key. + pub fn from_json(body: &str) -> Result { + serde_json::from_str(body).map_err(|e| OfferError::Parse(e.to_string())) + } + + /// Structural check against the pin. Does not require `open`. + /// + /// # Errors + /// + /// See [`OfferError`]. A closed-but-valid offer is legal to load. + pub fn validate(&self, pin: &ProofPin) -> Result<(), OfferError> { + if !crate::is_slug(&self.offer_id) { + return Err(OfferError::BadId(self.offer_id.clone())); + } + if !is_http_origin(self.provider.base_url.trim()) { + return Err(OfferError::BadBaseUrl); + } + if self.config.model_ref.trim().is_empty() || self.config.model_ref.len() > 256 { + return Err(OfferError::BadModelRef); + } + if !pin.allows_mode(self.config.mode) { + return Err(OfferError::ModeNotAllowed(self.config.mode)); + } + let cfg = &self.config; + for (field, got, ceiling) in [ + ( + "max_input_tokens", + cfg.max_input_tokens, + pin.max_input_tokens_ceiling, + ), + ( + "max_output_tokens", + cfg.max_output_tokens, + pin.max_output_tokens_ceiling, + ), + ] { + if got == 0 || got > ceiling { + return Err(OfferError::BadTokenCap(field, got, ceiling)); + } + } + for (name, v, hi) in [ + ("temperature", cfg.temperature, 2.0), + ("top_p", cfg.top_p, 1.0), + ] { + if v.is_some_and(|x| !x.is_finite() || !(0.0..=hi).contains(&x)) { + return Err(OfferError::BadSampling(name)); + } + } + let want = inference_config_commitment(&self.config, &self.provider.base_url); + if !is_hex64(&self.config_commitment) || !self.config_commitment.eq_ignore_ascii_case(&want) + { + return Err(OfferError::CommitmentMismatch); + } + Ok(()) + } + + /// Whether this judge backend is open for scoring. + pub fn is_open(&self) -> bool { + self.status == OfferStatus::Open + } + + /// Public status payload (no origin, no secrets). + pub fn public_view(&self) -> serde_json::Value { + serde_json::json!({ + "offer_id": self.offer_id, + "provider_kind": self.provider.kind, + "mode": self.config.mode, + "model_ref": self.config.model_ref, + "max_input_tokens": self.config.max_input_tokens, + "max_output_tokens": self.config.max_output_tokens, + "config_commitment": self.config_commitment, + "status": self.status, + }) + } + + /// Whether this open judge offer can score `topic` (resolved pin+topic vs offer). + /// + /// Origin is bound: a topic that spoofs `inference.base_url` fails closed + /// before lattice ([`OfferError::OriginMismatch`]). + /// + /// # Errors + /// + /// [`OfferError::Closed`], [`OfferError::OriginMismatch`], or + /// [`OfferError::CannotServeTopic`]. + pub fn serves_topic(&self, pin: &ProofPin, topic: &TopicDocument) -> Result<(), OfferError> { + if !self.is_open() { + return Err(OfferError::Closed); + } + if let Some(url) = topic + .inference + .base_url + .as_deref() + .map(str::trim) + .filter(|s| !s.is_empty()) + { + if url != self.provider.base_url.trim() { + return Err(OfferError::OriginMismatch); + } + } + let r = resolve_inference(pin, Some(&topic.inference), None, Some(self)); + if r.base_url.trim() != self.provider.base_url.trim() { + return Err(OfferError::OriginMismatch); + } + if self.provider.kind != r.provider + || self.config.mode != r.mode + || self.config.max_input_tokens < r.max_input_tokens + || self.config.max_output_tokens < r.max_output_tokens + || (!r.model.is_empty() && r.model != self.config.model_ref) + { + return Err(OfferError::CannotServeTopic); + } + if let Some(need) = topic.inference.require_judge_offer_commitment.as_deref() { + if !need.trim().eq_ignore_ascii_case(&self.config_commitment) { + return Err(OfferError::CannotServeTopic); + } + } + Ok(()) + } +} + +/// Fail-closed readiness: missing / closed / invalid offer cannot score. +/// +/// # Errors +/// +/// [`OfferError::Missing`], [`OfferError::Closed`], or a validate error. +pub fn require_open_offer<'a>( + offer: Option<&'a InferenceOffer>, + pin: &ProofPin, +) -> Result<&'a InferenceOffer, OfferError> { + let offer = offer.ok_or(OfferError::Missing)?; + offer.validate(pin)?; + if !offer.is_open() { + return Err(OfferError::Closed); + } + Ok(offer) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn pin() -> ProofPin { + ProofPin { + topic_pubkey: "ab".repeat(32), + ..ProofPin::default() + } + } + + fn config() -> InferenceConfig { + InferenceConfig { + mode: InferenceMode::Chat, + model_ref: "master-proxy-v0".into(), + max_input_tokens: 32_768, + max_output_tokens: 8_192, + temperature: Some(0.0), + top_p: None, + timeout_ms: None, + } + } + + fn offer() -> InferenceOffer { + let config = config(); + InferenceOffer { + offer_id: "master-v0".into(), + provider: InferenceProvider { + kind: InferenceProviderKind::OpenaiCompatible, + base_url: "http://127.0.0.1:8000/v1".into(), + }, + config_commitment: inference_config_commitment(&config, "http://127.0.0.1:8000/v1"), + config, + status: OfferStatus::Open, + } + } + + #[test] + fn a_well_formed_open_offer_validates() { + let o = offer(); + o.validate(&pin()).expect("valid"); + assert!(o.is_open()); + require_open_offer(Some(&o), &pin()).expect("ready"); + } + + #[test] + fn commitment_is_stable_and_binds_origin() { + let a = inference_config_commitment(&config(), "http://127.0.0.1:8000/v1"); + let b = inference_config_commitment(&config(), "http://127.0.0.1:8000/v1"); + assert_eq!(a, b); + assert_eq!(a.len(), 64); + assert_ne!( + a, + inference_config_commitment(&config(), "http://evil.example/v1") + ); + } + + #[test] + fn missing_or_closed_offer_cannot_score() { + assert!(matches!( + require_open_offer(None, &pin()), + Err(OfferError::Missing) + )); + let mut closed = offer(); + closed.status = OfferStatus::Closed; + closed.validate(&pin()).expect("closed may load"); + assert!(matches!( + require_open_offer(Some(&closed), &pin()), + Err(OfferError::Closed) + )); + } + + #[test] + fn empty_origin_or_model_is_refused() { + let mut o = offer(); + o.provider.base_url = String::new(); + assert!(matches!(o.validate(&pin()), Err(OfferError::BadBaseUrl))); + o = offer(); + o.provider.base_url = "file:///weights".into(); + assert!(matches!(o.validate(&pin()), Err(OfferError::BadBaseUrl))); + o = offer(); + o.config.model_ref = String::new(); + o.config_commitment = inference_config_commitment(&o.config, &o.provider.base_url); + assert!(matches!(o.validate(&pin()), Err(OfferError::BadModelRef))); + } + + #[test] + fn token_caps_cannot_loosen_the_pin() { + let mut o = offer(); + o.config.max_input_tokens = MAX_INPUT_TOKENS_CEILING + 1; + o.config_commitment = inference_config_commitment(&o.config, &o.provider.base_url); + assert!(matches!( + o.validate(&pin()), + Err(OfferError::BadTokenCap(..)) + )); + o = offer(); + o.config.max_output_tokens = 0; + o.config_commitment = inference_config_commitment(&o.config, &o.provider.base_url); + assert!(matches!( + o.validate(&pin()), + Err(OfferError::BadTokenCap(..)) + )); + } + + #[test] + fn a_wrong_commitment_is_refused() { + let mut o = offer(); + o.config_commitment = "cd".repeat(32); + assert!(matches!( + o.validate(&pin()), + Err(OfferError::CommitmentMismatch) + )); + } + + #[test] + fn public_view_never_leaks_origin() { + let v = serde_json::to_value(offer().public_view()).expect("json"); + let dump = v.to_string(); + assert!(!dump.contains("127.0.0.1"), "{dump}"); + assert!(!dump.contains("base_url"), "{dump}"); + assert!(!dump.contains("api_key"), "{dump}"); + assert_eq!(v["offer_id"], "master-v0"); + assert_eq!(v["provider_kind"], "openai_compatible"); + assert_eq!(v["status"], "open"); + } + + #[test] + fn open_offer_must_cover_the_topic() { + let mut topic = TopicDocument::default(); + topic.inference.mode = Some(InferenceMode::Chat); + topic.inference.max_input_tokens = Some(4_096); + topic.inference.max_output_tokens = Some(256); + offer().serves_topic(&pin(), &topic).expect("covers"); + topic.inference.mode = Some(InferenceMode::Embeddings); + assert!(matches!( + offer().serves_topic(&pin(), &topic), + Err(OfferError::CannotServeTopic) + )); + topic.inference.mode = Some(InferenceMode::Chat); + topic.inference.require_judge_offer_commitment = Some("ab".repeat(32)); + assert!(matches!( + offer().serves_topic(&pin(), &topic), + Err(OfferError::CannotServeTopic) + )); + } + + #[test] + fn spoofed_topic_origin_is_commitment_mismatch() { + let mut topic = TopicDocument::default(); + topic.inference.mode = Some(InferenceMode::Chat); + topic.inference.max_input_tokens = Some(4_096); + topic.inference.max_output_tokens = Some(256); + offer() + .serves_topic(&pin(), &topic) + .expect("token tighten ok"); + topic.inference.base_url = Some("http://evil.example/v1".into()); + assert!( + matches!( + offer().serves_topic(&pin(), &topic), + Err(OfferError::OriginMismatch) + ), + "a topic that only tightens tokens but spoofs origin must fail closed" + ); + topic.inference.base_url = Some("http://127.0.0.1:8000/v1".into()); + offer() + .serves_topic(&pin(), &topic) + .expect("matching origin is not a spoof"); + } + + #[test] + fn pin_defaults_resolve_and_topic_may_override_or_tighten() { + let mut p = pin(); + p.inference.model = "pin-model".into(); + p.inference.max_input_tokens = 4_096; + p.inference.max_output_tokens = 512; + let inherited = resolve_inference(&p, None, None, None); + assert_eq!(inherited.model, "pin-model"); + assert_eq!(inherited.mode, InferenceMode::Chat); + assert_eq!(inherited.max_input_tokens, 4_096); + assert!(!inherited.ready_to_score()); + let mut topic = TopicInference { + model: Some("topic-model".into()), + max_input_tokens: Some(2_048), + max_output_tokens: Some(128), + ..TopicInference::default() + }; + let over = resolve_inference(&p, Some(&topic), None, Some(&offer())); + assert_eq!(over.model, "topic-model"); + assert_eq!(over.max_input_tokens, 2_048); + assert_eq!(over.base_url, "http://127.0.0.1:8000/v1"); + topic.base_url = Some("http://evil.example/v1".into()); + let redirected = resolve_inference(&p, Some(&topic), None, Some(&offer())); + assert_eq!( + redirected.base_url, "http://127.0.0.1:8000/v1", + "topic URL must not redirect the committed judge origin" + ); + assert!(over.ready_to_score()); + topic.max_input_tokens = Some(8_192); + assert!(matches!( + topic.validate(&p), + Err(crate::TopicError::InferenceCeiling(..)) + )); + } + + #[test] + fn missing_model_or_origin_is_incomplete() { + let p = pin(); + assert!(p.inference.model.is_empty()); + let r = resolve_inference(&p, None, None, Some(&offer())); + assert!(r.model.trim().is_empty()); + let mut t = TopicInference { + model: Some("live-model".into()), + ..TopicInference::default() + }; + assert!(resolve_inference(&p, Some(&t), None, Some(&offer())).ready_to_score()); + t.base_url = Some("ftp://nope".into()); + assert!(matches!( + t.validate(&p), + Err(crate::TopicError::IncompleteInference) + )); + } + + #[test] + fn unknown_offer_key_is_refused() { + let err = + InferenceOffer::from_json(r#"{"offer_id":"x","api_key":"nope"}"#).expect_err("unknown"); + assert!(format!("{err}").contains("api_key"), "{err}"); + } +} diff --git a/crates/proof-task/src/lib.rs b/crates/proof-task/src/lib.rs index 20c58c26f..251f589b3 100644 --- a/crates/proof-task/src/lib.rs +++ b/crates/proof-task/src/lib.rs @@ -25,19 +25,30 @@ //! answer: this host cannot score, and nobody is paid. #![forbid(unsafe_code)] -#![allow(clippy::doc_markdown, clippy::module_name_repetitions)] +#![allow( + clippy::doc_markdown, + clippy::module_name_repetitions, + clippy::must_use_candidate +)] mod canonical; -mod holdout; +mod inference; mod pin; mod topic; pub use canonical::canonical_json; -pub use holdout::{ - contamination, holdout_commitment, synthetic_holdout, verify_holdout, HoldoutError, - HoldoutRecord, HoldoutSplit, LONGCTX_MAX_TOKENS, LONGCTX_MIN_TOKENS, +pub use inference::{ + inference_config_commitment, require_open_offer, resolve_inference, InferenceConfig, + InferenceMode, InferenceOffer, InferenceProvider, InferenceProviderKind, OfferError, + OfferStatus, PinInference, TopicInference, ALLOWED_MODES, INFERENCE_CONFIG_SCHEMA_VERSION, + INFERENCE_OFFER_COMMITMENT_ALG, MAX_INPUT_TOKENS_CEILING, MAX_OUTPUT_TOKENS_CEILING, }; pub use pin::{PinError, ProofPin}; +pub use proof_holdout::{ + contamination, holdout_commitment, synthetic_holdout, verify_holdout, HoldoutError, + HoldoutRecord, HoldoutSplit, HOLDOUT_DOMAIN, HOLDOUT_SIZE, LONGCTX_MAX_TOKENS, + LONGCTX_MIN_TOKENS, STRATUM_SIZE, +}; pub use topic::{ default_adamw, topic_signing_payload, Baseline, Constraints, DiscoverySpec, MetricDirection, MetricFamily, MetricSpec, PayoutMode, TopicDocument, TopicError, TopicStatus, ValidationSpec, @@ -58,9 +69,6 @@ pub const SCORING_VERSION: u16 = 1; /// Domain tag for task id digests. pub const TASK_ID_DOMAIN: &[u8] = b"base-proof-task-id-v1"; -/// Domain tag for per-topic holdout commitments. -pub const HOLDOUT_DOMAIN: &[u8] = b"base-proof-holdout-v1"; - /// Domain tag for eval-receipt digests. pub const RECEIPT_DOMAIN: &[u8] = b"base-proof-receipt-v1"; @@ -80,11 +88,8 @@ pub const TOPIC_DOMAIN: crypto::DomainTag = crypto::DomainTag::new(b"base-proof- /// Integer score lattice max (same scale as every other challenge). pub const SCORE_MAX: u64 = 1_000_000; -/// Base model family the proxy must belong to (same family as `relearn`). -/// -/// Family lock for the RLM judge the eval image bakes. The exact judge id -/// lives in [`ProofPin::proxy_model`] and must be a model the pinned image -/// contains. This is not a miner training proxy. +/// Research family name (documentation). This is **not** an architecture lock +/// and **not** an HF bake: the RLM judge calls a live [`InferenceOffer`]. pub const BASE_MODEL_FAMILY: &str = "Qwen/Qwen3.8"; /// Eval image repository (digest-pinned; never a floating tag in prod). @@ -119,15 +124,29 @@ pub const EPSILON_THROUGHPUT_REL_MIN: f64 = 0.05; /// Quality floor a throughput topic must keep: speed is not free. pub const QUALITY_FLOOR_NLL_MAX: f64 = 0.02; -/// Holdout records per topic. -pub const HOLDOUT_SIZE: usize = 120; - -/// Records per scored split (`HOLDOUT_SIZE / scored splits`). -pub const STRATUM_SIZE: usize = 24; - /// Slice id prefix bound into per-topic measurements. pub const HOLDOUT_SLICE_PREFIX: &str = "proof-holdout"; +pub(crate) fn is_hex64(s: &str) -> bool { + let t = s.trim(); + t.len() == 64 && t.chars().all(|c| c.is_ascii_hexdigit()) +} + +pub(crate) fn is_http_origin(url: &str) -> bool { + let u = url.trim(); + (u.starts_with("http://") || u.starts_with("https://")) + && u.len() >= 8 + && !u.contains(['\n', ' ']) +} + +pub(crate) fn is_slug(id: &str) -> bool { + let b = id.as_bytes(); + (2..=63).contains(&b.len()) + && (b[0].is_ascii_lowercase() || b[0].is_ascii_digit()) + && b.iter() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || *c == b'-') +} + #[cfg(test)] mod tests { use super::*; @@ -190,5 +209,10 @@ mod tests { 10_000 ); assert_eq!(CUSTOM_HARNESS_SUCCESS_RATE, "harness_success_rate"); + assert_eq!(INFERENCE_CONFIG_SCHEMA_VERSION, 1); + assert_eq!(INFERENCE_OFFER_COMMITMENT_ALG, "sha256"); + assert_eq!(MAX_INPUT_TOKENS_CEILING, LONGCTX_MAX_TOKENS); + assert_eq!(MAX_OUTPUT_TOKENS_CEILING, 8_192); + assert_eq!(ALLOWED_MODES.len(), 3); } } diff --git a/crates/proof-task/src/pin.rs b/crates/proof-task/src/pin.rs index 3e3cca46e..9f6105145 100644 --- a/crates/proof-task/src/pin.rs +++ b/crates/proof-task/src/pin.rs @@ -1,18 +1,22 @@ -//! Global Proof pin: eval image digest, proxy default, and the floors a topic -//! may tighten but never loosen. +//! Global Proof pin: eval image digest, inference ceilings, and the floors a +//! topic may tighten but never loosen. //! -//! There is **no topic catalog here**. Problems are operator-published signed -//! documents ([`crate::TopicDocument`]); git carries only what every topic is -//! measured against: which image may score, which key may publish, and how -//! generous a topic is allowed to be. +//! There is **no topic catalog here** and **no live InferenceOffer**. Problems +//! are operator-published signed documents ([`crate::TopicDocument`]); the +//! master's **RLM judge** backend lives in operator state. Git carries only +//! what every topic is measured against: which image may score, which key may +//! publish, which judge modes/token ceilings are legal, and how generous a +//! topic is allowed to be. No secrets. use serde::{Deserialize, Serialize}; use thiserror::Error; use crate::{ - BASE_MODEL_FAMILY, CHALLENGE_ID, EPSILON_NLL_MIN, EPSILON_THROUGHPUT_REL_MIN, - EPSILON_TOPIC_MAX_REGRESS_MIN, EVAL_IMAGE, FLOPS_BUDGET_MAX, HOLDOUT_SIZE, PROOF_GIT_URL, - QUALITY_FLOOR_NLL_MAX, SCORING_VERSION, STRATUM_SIZE, + is_hex64, is_http_origin, InferenceMode, PinInference, ALLOWED_MODES, BASE_MODEL_FAMILY, + CHALLENGE_ID, EPSILON_NLL_MIN, EPSILON_THROUGHPUT_REL_MIN, EPSILON_TOPIC_MAX_REGRESS_MIN, + EVAL_IMAGE, FLOPS_BUDGET_MAX, HOLDOUT_SIZE, INFERENCE_CONFIG_SCHEMA_VERSION, + INFERENCE_OFFER_COMMITMENT_ALG, MAX_INPUT_TOKENS_CEILING, MAX_OUTPUT_TOKENS_CEILING, + PROOF_GIT_URL, QUALITY_FLOOR_NLL_MAX, SCORING_VERSION, STRATUM_SIZE, }; /// `config/proof-pin.toml`. @@ -23,20 +27,26 @@ pub struct ProofPin { pub challenge_id: String, /// Must equal [`SCORING_VERSION`]. pub scoring_version: u16, - /// Family lock for the proxy the eval image bakes. + /// Research family name. Not an architecture lock and not an HF bake. pub base_model_family: String, - /// Default RLM judge id baked into the pinned image. Empty until it exists. - /// - /// This is the eval-image judge that scores miner submissions, not a - /// miner training proxy. A topic may name its own proxy, but only one - /// the image contains. + /// Deprecated HF proxy id. Must stay empty — Proof does not bake weights. + #[serde(default)] pub proxy_model: String, - /// Every proxy the pinned image bakes. Empty until the image exists. - /// - /// This is what makes a topic's `proxy_model` override checkable: the - /// control plane cannot look inside the image, so the pin declares its - /// contents and a topic naming anything else is a publish reject. + /// Deprecated HF proxy list. Must stay empty. + #[serde(default)] pub proxy_models: Vec, + /// Inference config schema (`1`). + pub inference_config_schema_version: u32, + /// Modes a topic / offer may name. Subset of chat, completions, embeddings. + pub allowed_modes: Vec, + /// Largest input cap a topic or offer may declare. + pub max_input_tokens_ceiling: u32, + /// Largest output cap a topic or offer may declare. + pub max_output_tokens_ceiling: u32, + /// Hash algorithm for `config_commitment` (`sha256`). + pub inference_offer_commitment_alg: String, + /// Complete provider defaults. Empty model/url is pre-launch fail-closed. + pub inference: PinInference, /// Eval image reference (no floating tag in prod). pub eval_image: String, /// `sha256:…` digest. Empty until the first green proof-eval CI image. @@ -71,6 +81,12 @@ impl Default for ProofPin { base_model_family: BASE_MODEL_FAMILY.into(), proxy_model: String::new(), proxy_models: Vec::new(), + inference_config_schema_version: INFERENCE_CONFIG_SCHEMA_VERSION, + allowed_modes: ALLOWED_MODES.to_vec(), + max_input_tokens_ceiling: MAX_INPUT_TOKENS_CEILING, + max_output_tokens_ceiling: MAX_OUTPUT_TOKENS_CEILING, + inference_offer_commitment_alg: INFERENCE_OFFER_COMMITMENT_ALG.into(), + inference: PinInference::default(), eval_image: EVAL_IMAGE.into(), eval_image_digest: String::new(), proof_git: PROOF_GIT_URL.into(), @@ -117,17 +133,42 @@ pub enum PinError { /// The only image that may score. want: &'static str, }, - /// The proxy is outside the locked family. - #[error("proxy_model {got:?} is not in the {want:?} family")] - ProxyOutsideFamily { + /// The research family name drifted. + #[error("base_model_family {got:?} is not {want:?}")] + WrongFamily { + /// What the pin said. + got: String, + /// Locked family name. + want: &'static str, + }, + /// A non-empty HF proxy would reintroduce a bake lock. + #[error("proxy_model / proxy_models must stay empty (no HF bake)")] + DeprecatedProxyBake, + /// Inference schema version drift. + #[error("inference_config_schema_version {got}, this build reads {want}")] + WrongInferenceSchema { + /// What the pin said. + got: u32, + /// What this build reads. + want: u32, + }, + /// Commitment algorithm is not sha256. + #[error("inference_offer_commitment_alg {got:?} is not {want:?}")] + WrongCommitmentAlg { /// What the pin said. got: String, - /// Locked family prefix. + /// Locked algorithm. want: &'static str, }, - /// The default proxy is not one the image is declared to contain. - #[error("proxy_model {0:?} is not listed in proxy_models")] - DefaultProxyNotBaked(String), + /// `allowed_modes` is empty or names something this build does not score. + #[error("allowed_modes must be a non-empty subset of chat, completions, embeddings")] + BadAllowedModes, + /// A token ceiling was raised above the crate lock, or is zero. + #[error("{0} = {1} must be 1..={2}")] + TokenCeiling(&'static str, u32, u32), + /// `[inference].base_url` is set but is not an http(s) origin. + #[error("inference.base_url must be empty (secret-backed) or an http(s) origin")] + BadInferenceUrl, /// The topic key is not a 64-hex sr25519 public key. #[error("topic_pubkey must be 64 hex chars (the challenges.toml `proof` row key)")] BadTopicPubkey, @@ -198,27 +239,12 @@ impl ProofPin { }); } if self.base_model_family.trim() != BASE_MODEL_FAMILY { - return Err(PinError::ProxyOutsideFamily { + return Err(PinError::WrongFamily { got: self.base_model_family.clone(), want: BASE_MODEL_FAMILY, }); } - // Empty is legal (no published image yet); a named proxy must be in - // the locked family, so a pin bump cannot quietly move the challenge - // onto another model. - let proxy = self.proxy_model.trim(); - for named in std::iter::once(proxy).chain(self.proxy_models.iter().map(String::as_str)) { - let named = named.trim(); - if !named.is_empty() && !named.starts_with(BASE_MODEL_FAMILY) { - return Err(PinError::ProxyOutsideFamily { - got: named.to_owned(), - want: BASE_MODEL_FAMILY, - }); - } - } - if !proxy.is_empty() && !self.proxy_models.is_empty() && !self.bakes_proxy(proxy) { - return Err(PinError::DefaultProxyNotBaked(self.proxy_model.clone())); - } + self.validate_inference()?; if !is_hex64(&self.topic_pubkey) { return Err(PinError::BadTopicPubkey); } @@ -270,49 +296,93 @@ impl ProofPin { Ok(()) } + fn validate_inference(&self) -> Result<(), PinError> { + if !self.proxy_model.trim().is_empty() + || self.proxy_models.iter().any(|m| !m.trim().is_empty()) + { + return Err(PinError::DeprecatedProxyBake); + } + if self.inference_config_schema_version != INFERENCE_CONFIG_SCHEMA_VERSION { + return Err(PinError::WrongInferenceSchema { + got: self.inference_config_schema_version, + want: INFERENCE_CONFIG_SCHEMA_VERSION, + }); + } + if self.inference_offer_commitment_alg.trim() != INFERENCE_OFFER_COMMITMENT_ALG { + return Err(PinError::WrongCommitmentAlg { + got: self.inference_offer_commitment_alg.clone(), + want: INFERENCE_OFFER_COMMITMENT_ALG, + }); + } + if self.allowed_modes.is_empty() { + return Err(PinError::BadAllowedModes); + } + let mut seen = Vec::new(); + for mode in &self.allowed_modes { + if seen.contains(mode) || !ALLOWED_MODES.contains(mode) { + return Err(PinError::BadAllowedModes); + } + seen.push(*mode); + } + if !self.allows_mode(self.inference.mode) { + return Err(PinError::BadAllowedModes); + } + let inf = &self.inference; + for (field, got, ceiling) in [ + ( + "max_input_tokens_ceiling", + self.max_input_tokens_ceiling, + MAX_INPUT_TOKENS_CEILING, + ), + ( + "max_output_tokens_ceiling", + self.max_output_tokens_ceiling, + MAX_OUTPUT_TOKENS_CEILING, + ), + ( + "inference.max_input_tokens", + inf.max_input_tokens, + self.max_input_tokens_ceiling, + ), + ( + "inference.max_output_tokens", + inf.max_output_tokens, + self.max_output_tokens_ceiling, + ), + ] { + if got == 0 || got > ceiling { + return Err(PinError::TokenCeiling(field, got, ceiling)); + } + } + if (!inf.base_url.trim().is_empty() && !is_http_origin(&inf.base_url)) + || inf.model.len() > 256 + { + return Err(PinError::BadInferenceUrl); + } + Ok(()) + } + /// True when a live rent is allowed (real digest pin present). /// /// An empty digest is the normal pre-launch state and the reason submits /// answer `503`. There is no sim fallback on this challenge. - #[must_use] pub fn can_rent(&self) -> bool { let d = self.eval_image_digest.trim(); d.starts_with("sha256:") && d.len() >= 71 } /// Topic-signing public key bytes, when the pin carries a well-formed one. - #[must_use] pub fn topic_pubkey_bytes(&self) -> Option<[u8; 32]> { let raw = hex::decode(self.topic_pubkey.trim()).ok()?; <[u8; 32]>::try_from(raw).ok() } - /// Whether the pinned image is declared to contain `proxy`. - #[must_use] - pub fn bakes_proxy(&self, proxy: &str) -> bool { - let want = proxy.trim(); - if want.is_empty() { - return false; - } - self.proxy_models.iter().any(|m| m.trim() == want) || self.proxy_model.trim() == want - } - - /// Proxy a topic will actually train: the topic's override, else the pin's. - #[must_use] - pub fn proxy_for(&self, topic_proxy: Option<&str>) -> String { - topic_proxy - .map(str::trim) - .filter(|s| !s.is_empty()) - .unwrap_or_else(|| self.proxy_model.trim()) - .to_owned() + /// Whether `mode` is in this pin's allowlist. + pub fn allows_mode(&self, mode: InferenceMode) -> bool { + self.allowed_modes.contains(&mode) } } -fn is_hex64(s: &str) -> bool { - let t = s.trim(); - t.len() == 64 && t.chars().all(|c| c.is_ascii_hexdigit()) -} - #[cfg(test)] mod tests { use super::*; @@ -341,6 +411,11 @@ mod tests { challenge_id = "proof" scoring_version = 1 base_model_family = "{BASE_MODEL_FAMILY}" +inference_config_schema_version = 1 +allowed_modes = ["chat", "completions", "embeddings"] +max_input_tokens_ceiling = 32768 +max_output_tokens_ceiling = 8192 +inference_offer_commitment_alg = "sha256" eval_image = "{EVAL_IMAGE}" eval_image_digest = "" topic_pubkey = "{}" @@ -355,7 +430,10 @@ stratum_size = 24 let p = ProofPin::from_toml(&body).expect("parse"); p.validate().expect("validates"); assert_eq!(p.flops_budget_max, 2_000_000_000_000_000_000); + assert_eq!(p.max_input_tokens_ceiling, 32_768); assert_eq!(p.topic_pubkey_bytes().expect("key"), [0xcd; 32]); + assert!(p.proxy_model.is_empty()); + assert!(p.proxy_models.is_empty()); } #[test] @@ -398,15 +476,48 @@ stratum_size = 24 } #[test] - fn a_proxy_outside_the_locked_family_is_refused() { + fn a_named_hf_proxy_bake_is_refused() { let mut p = pin(); - p.proxy_model = "meta-llama/Llama-3-8B".into(); + p.proxy_model = "Qwen/Qwen3.8-0.6B".into(); + assert!(matches!(p.validate(), Err(PinError::DeprecatedProxyBake))); + p.proxy_model.clear(); + p.proxy_models = vec!["Qwen/Qwen3-0.6B".into()]; + assert!(matches!(p.validate(), Err(PinError::DeprecatedProxyBake))); + p.proxy_models.clear(); + p.validate().expect("empty deprecated fields"); + } + + #[test] + fn inference_schema_and_ceilings_are_locked() { + let mut p = pin(); + p.inference_config_schema_version = 2; assert!(matches!( p.validate(), - Err(PinError::ProxyOutsideFamily { .. }) + Err(PinError::WrongInferenceSchema { .. }) )); - p.proxy_model = format!("{BASE_MODEL_FAMILY}-1.5B"); - p.validate().expect("in-family proxy"); + p = pin(); + p.inference_offer_commitment_alg = "blake3".into(); + assert!(matches!( + p.validate(), + Err(PinError::WrongCommitmentAlg { .. }) + )); + p = pin(); + p.allowed_modes.clear(); + assert!(matches!(p.validate(), Err(PinError::BadAllowedModes))); + p = pin(); + p.max_input_tokens_ceiling = MAX_INPUT_TOKENS_CEILING + 1; + assert!(matches!(p.validate(), Err(PinError::TokenCeiling(..)))); + p = pin(); + p.allowed_modes = vec![InferenceMode::Chat]; + p.validate().expect("subset is a tighten"); + assert!(p.allows_mode(InferenceMode::Chat)); + assert!(!p.allows_mode(InferenceMode::Embeddings)); + p = pin(); + p.inference.base_url = "not-a-url".into(); + assert!(matches!(p.validate(), Err(PinError::BadInferenceUrl))); + p = pin(); + p.inference.base_url = "https://example.invalid/v1".into(); + p.validate().expect("public origin may live in the pin"); } #[test] @@ -442,18 +553,28 @@ stratum_size = 24 } #[test] - fn proxy_for_prefers_the_topic_override() { - let mut p = pin(); - p.proxy_model = format!("{BASE_MODEL_FAMILY}-1.5B"); - assert_eq!(p.proxy_for(None), format!("{BASE_MODEL_FAMILY}-1.5B")); + fn default_pin_has_the_locked_inference_schema() { + let p = pin(); + assert_eq!( + p.inference_config_schema_version, + INFERENCE_CONFIG_SCHEMA_VERSION + ); assert_eq!( - p.proxy_for(Some(" ")), - format!("{BASE_MODEL_FAMILY}-1.5B") + p.inference_offer_commitment_alg, + INFERENCE_OFFER_COMMITMENT_ALG ); + assert_eq!(p.allowed_modes.as_slice(), ALLOWED_MODES.as_slice()); + assert_eq!(p.max_input_tokens_ceiling, MAX_INPUT_TOKENS_CEILING); + assert_eq!(p.max_output_tokens_ceiling, MAX_OUTPUT_TOKENS_CEILING); assert_eq!( - p.proxy_for(Some("Qwen/Qwen3.8-0.6B")), - "Qwen/Qwen3.8-0.6B".to_owned() + p.inference.provider, + crate::InferenceProviderKind::OpenaiCompatible ); + assert!(p.inference.base_url.is_empty()); + assert!(p.inference.model.is_empty()); + assert_eq!(p.inference.mode, InferenceMode::Chat); + assert_eq!(p.inference.max_input_tokens, MAX_INPUT_TOKENS_CEILING); + assert_eq!(p.inference.max_output_tokens, MAX_OUTPUT_TOKENS_CEILING); } #[test] diff --git a/crates/proof-task/src/topic.rs b/crates/proof-task/src/topic.rs index d7f47f5f4..a0282c3cf 100644 --- a/crates/proof-task/src/topic.rs +++ b/crates/proof-task/src/topic.rs @@ -27,7 +27,7 @@ use serde::{Deserialize, Serialize}; -use crate::{canonical_json, ProofPin, TOPIC_DOMAIN}; +use crate::{canonical_json, is_hex64, ProofPin, TopicInference, TOPIC_DOMAIN}; /// Only accepted `schema_version`. pub const TOPIC_SCHEMA_VERSION: u32 = 1; @@ -88,8 +88,6 @@ pub enum MetricFamily { } impl MetricFamily { - /// Wire name. - #[must_use] pub const fn as_str(self) -> &'static str { match self { Self::Nll => "nll", @@ -297,7 +295,6 @@ impl Default for Baseline { } /// The locked AdamW recipe. A topic's `adamw` baseline must be exactly this. -#[must_use] pub fn default_adamw(flops_budget: u64) -> Baseline { Baseline { optimizer: "adamw".into(), @@ -319,7 +316,6 @@ pub fn default_adamw(flops_budget: u64) -> Baseline { impl Baseline { /// Whether both seal hashes are present and well formed. - #[must_use] pub fn is_sealed(&self) -> bool { is_hex64(&self.script_sha256) && is_hex64(&self.metrics_commitment) } @@ -367,10 +363,12 @@ pub struct TopicDocument { pub flops_budget: u64, /// Absolute NLL a challenger must win by (`nll` family primary gate). pub epsilon_nll: f64, - /// Largest per-split NLL regression tolerated on any scored stratum. + /// Per-split NLL regression gate. pub epsilon_topic_max_regress: f64, - /// Proxy override. `None` means the pin default. + /// Deprecated HF proxy override. Must stay empty. pub proxy_model: Option, + /// Inference constraints this topic tightens against the pin. + pub inference: TopicInference, /// Sealed baseline recipe plus its two seal hashes. pub baseline: Baseline, /// Commitment over this topic's holdout records. @@ -407,6 +405,7 @@ impl Default for TopicDocument { epsilon_nll: crate::EPSILON_NLL_MIN, epsilon_topic_max_regress: crate::EPSILON_TOPIC_MAX_REGRESS_MIN, proxy_model: None, + inference: TopicInference::default(), baseline: default_adamw(crate::FLOPS_BUDGET_MAX), holdout_commitment: String::new(), holdout_size: crate::HOLDOUT_SIZE, @@ -533,9 +532,21 @@ pub enum TopicError { /// Pin value. want: usize, }, - /// A proxy the pinned eval image does not contain. - #[error("proxy_model {0:?} is not baked into the pinned eval image")] - ProxyNotBaked(String), + /// A proxy bake lock — retired; the RLM judge uses a live InferenceOffer. + #[error("proxy_model is deprecated; the RLM judge uses inference + a live InferenceOffer")] + DeprecatedProxyModel, + /// Inference mode not in the pin allowlist. + #[error("inference.mode {0:?} is not in the pin allowed_modes")] + InferenceModeNotAllowed(crate::InferenceMode), + /// Topic token cap is zero or above the pin inference default / ceiling. + #[error("inference.{0} = {1} must be 1..={2}")] + InferenceCeiling(&'static str, u32, u32), + /// `require_judge_offer_commitment` is not 64 hex. + #[error("inference.require_judge_offer_commitment must be 64 hex chars")] + BadOfferCommitment, + /// Open topic resolved to an incomplete provider config, or an override is unusable. + #[error("inference is incomplete or misconfigured (provider, model, mode, tokens, origin)")] + IncompleteInference, /// The validity window is inverted. #[error("valid_until_epoch {until} is before valid_from_epoch {from}")] BadWindow { @@ -558,11 +569,6 @@ pub enum TopicError { Canonicalize(String), } -fn is_hex64(s: &str) -> bool { - let t = s.trim(); - t.len() == 64 && t.chars().all(|c| c.is_ascii_hexdigit()) -} - fn approx(a: f64, b: f64) -> bool { (a - b).abs() <= 1e-12_f64.max(b.abs() * 1e-9) } @@ -783,7 +789,7 @@ impl TopicDocument { want: TOPIC_SCHEMA_VERSION, }); } - if !is_valid_id(&self.id) { + if !crate::is_slug(&self.id) { return Err(TopicError::BadId(self.id.clone())); } let statement = self.statement.trim(); @@ -817,10 +823,15 @@ impl TopicDocument { } } if let Some(proxy) = self.proxy_model.as_deref().map(str::trim) { - if proxy.is_empty() || !pin.bakes_proxy(proxy) { - return Err(TopicError::ProxyNotBaked(proxy.to_owned())); + if !proxy.is_empty() { + return Err(TopicError::DeprecatedProxyModel); } } + self.inference.validate(pin)?; + let model = crate::resolve_inference(pin, Some(&self.inference), None, None).model; + if self.status == TopicStatus::Open && model.trim().is_empty() { + return Err(TopicError::IncompleteInference); + } self.baseline.validate(&self.metric, self.flops_budget)?; if !is_hex64(&self.holdout_commitment) { return Err(TopicError::BadHoldoutCommitment); @@ -877,7 +888,6 @@ impl TopicDocument { } /// Whether this topic accepts submissions and earns emission at `epoch`. - #[must_use] pub fn is_open_at(&self, epoch: u64) -> bool { self.status == TopicStatus::Open && epoch >= self.valid_from_epoch @@ -885,39 +895,23 @@ impl TopicDocument { } /// Slice id bound into this topic's measurements. - #[must_use] pub fn slice_id(&self) -> String { format!("{}-{}", crate::HOLDOUT_SLICE_PREFIX, self.id) } } -fn is_valid_id(id: &str) -> bool { - let len = id.len(); - if !(MIN_TOPIC_ID_LEN..=MAX_TOPIC_ID_LEN).contains(&len) { - return false; - } - let mut chars = id.chars(); - let Some(first) = chars.next() else { - return false; - }; - if !first.is_ascii_lowercase() && !first.is_ascii_digit() { - return false; - } - chars.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') -} - #[cfg(test)] mod tests { use super::*; use crate::{holdout_commitment, synthetic_holdout, STRATUM_SIZE}; fn pin() -> ProofPin { - ProofPin { + let mut p = ProofPin { topic_pubkey: hex::encode(crypto::public_key_from_mini_secret(&sk()).expect("pk")), - proxy_model: "Qwen/Qwen3.8-0.6B".into(), - proxy_models: vec!["Qwen/Qwen3.8-0.6B".into(), "Qwen/Qwen3.8-1.7B".into()], ..ProofPin::default() - } + }; + p.inference.model = "master-proxy-v0".into(); + p } fn sk() -> [u8; 32] { @@ -1050,6 +1044,7 @@ mod tests { "\"metric\"", "\"baseline\"", "\"holdout_commitment\"", + "\"inference\"", "\"status\"", ] { assert!(payload.contains(field), "missing {field} in {payload}"); @@ -1322,18 +1317,79 @@ mod tests { } #[test] - fn a_proxy_the_image_does_not_bake_is_refused() { + fn a_deprecated_proxy_model_is_refused() { let p = pin(); let mut doc = nll_topic(); - doc.proxy_model = Some("Qwen/Qwen3.8-1.7B".into()); - doc.validate(&p, &[]).expect("a baked proxy is fine"); - doc.proxy_model = Some("Qwen/Qwen3.8-27B".into()); + doc.proxy_model = Some("Qwen/Qwen3.8-0.6B".into()); assert!(matches!( doc.validate(&p, &[]), - Err(TopicError::ProxyNotBaked(_)) + Err(TopicError::DeprecatedProxyModel) )); } + #[test] + fn topic_inference_may_tighten_but_never_loosen_pin_defaults() { + let p = pin(); + let mut tight = nll_topic(); + tight.inference.max_input_tokens = Some(4_096); + tight.inference.max_output_tokens = Some(256); + tight.validate(&p, &[]).expect("tighter is fine"); + + let mut loose = nll_topic(); + loose.inference.max_input_tokens = Some(crate::MAX_INPUT_TOKENS_CEILING + 1); + assert!(matches!( + loose.validate(&p, &[]), + Err(TopicError::InferenceCeiling(..)) + )); + + let mut vs_pin = nll_topic(); + let mut tight_pin = p.clone(); + tight_pin.inference.max_input_tokens = 4_096; + vs_pin.inference.max_input_tokens = Some(8_192); + assert!(matches!( + vs_pin.validate(&tight_pin, &[]), + Err(TopicError::InferenceCeiling(..)) + )); + vs_pin.inference.max_input_tokens = None; + vs_pin + .validate(&tight_pin, &[]) + .expect("inherit pin default"); + + let mut subset = pin(); + subset.allowed_modes = vec![crate::InferenceMode::Completions]; + let mut completions = nll_topic(); + completions.inference.mode = Some(crate::InferenceMode::Chat); + assert!(matches!( + completions.validate(&subset, &[]), + Err(TopicError::InferenceModeNotAllowed(_)) + )); + + let mut bad = nll_topic(); + bad.inference.require_judge_offer_commitment = Some("zz".into()); + assert!(matches!( + bad.validate(&p, &[]), + Err(TopicError::BadOfferCommitment) + )); + } + + #[test] + fn an_open_topic_without_a_model_is_incomplete() { + let mut p = pin(); + p.inference.model.clear(); + assert!(matches!( + nll_topic().validate(&p, &[]), + Err(TopicError::IncompleteInference) + )); + let mut filled = nll_topic(); + filled.inference.model = Some("topic-model".into()); + filled + .validate(&p, &[]) + .expect("topic override fills model"); + filled.status = crate::TopicStatus::Draft; + filled.inference.model = None; + filled.validate(&p, &[]).expect("draft may omit model"); + } + #[test] fn the_validity_window_gates_open() { let mut doc = nll_topic(); diff --git a/crates/proof-task/tests/committed_pin.rs b/crates/proof-task/tests/committed_pin.rs index e62d3da88..f8f6d1982 100644 --- a/crates/proof-task/tests/committed_pin.rs +++ b/crates/proof-task/tests/committed_pin.rs @@ -5,7 +5,11 @@ use std::path::{Path, PathBuf}; -use proof_task::{ProofPin, CHALLENGE_ID, EVAL_IMAGE, HOLDOUT_SIZE, STRATUM_SIZE}; +use proof_task::{ + ProofPin, ALLOWED_MODES, CHALLENGE_ID, EVAL_IMAGE, HOLDOUT_SIZE, + INFERENCE_CONFIG_SCHEMA_VERSION, INFERENCE_OFFER_COMMITMENT_ALG, MAX_INPUT_TOKENS_CEILING, + MAX_OUTPUT_TOKENS_CEILING, STRATUM_SIZE, +}; fn pin_path() -> PathBuf { Path::new(env!("CARGO_MANIFEST_DIR")) @@ -70,10 +74,36 @@ fn committed_pin_is_proof_with_a_real_eval_digest() { "{digest}" ); assert!(p.can_rent(), "pinned digest must be rentable"); - assert_eq!(p.proxy_model, "Qwen/Qwen3.8-0.6B"); - assert!(p.bakes_proxy("Qwen/Qwen3.8-0.6B")); assert_eq!(p.holdout_size, HOLDOUT_SIZE); assert_eq!(p.stratum_size, STRATUM_SIZE); + assert!(p.proxy_model.trim().is_empty(), "no HF proxy bake"); + assert!(p.proxy_models.is_empty(), "no HF proxy bake list"); + assert_eq!( + p.inference_config_schema_version, + INFERENCE_CONFIG_SCHEMA_VERSION + ); + assert_eq!( + p.inference_offer_commitment_alg, + INFERENCE_OFFER_COMMITMENT_ALG + ); + assert_eq!(p.allowed_modes.as_slice(), ALLOWED_MODES.as_slice()); + assert_eq!(p.max_input_tokens_ceiling, MAX_INPUT_TOKENS_CEILING); + assert_eq!(p.max_output_tokens_ceiling, MAX_OUTPUT_TOKENS_CEILING); + assert_eq!( + p.inference.provider, + proof_task::InferenceProviderKind::OpenaiCompatible + ); + assert!( + p.inference.base_url.trim().is_empty(), + "url is secret-backed" + ); + assert!( + p.inference.model.trim().is_empty(), + "empty model is pre-launch 503" + ); + assert_eq!(p.inference.mode, proof_task::InferenceMode::Chat); + assert_eq!(p.inference.max_input_tokens, MAX_INPUT_TOKENS_CEILING); + assert_eq!(p.inference.max_output_tokens, MAX_OUTPUT_TOKENS_CEILING); } #[test] @@ -81,19 +111,37 @@ fn topic_pubkey_matches_the_trust_root_proof_row() { assert_eq!(pin().topic_pubkey.to_ascii_lowercase(), proof_row_pubkey()); } +#[test] +fn pin_carries_complete_inference_defaults_table() { + let text = body(); + assert!(text.contains("[inference]"), "pin must declare [inference]"); + for key in [ + "provider", + "base_url", + "model", + "mode", + "max_input_tokens", + "max_output_tokens", + ] { + assert!(text.contains(key), "pin [inference] must name {key}"); + } +} + #[test] fn pin_carries_no_endpoint_secret_or_topic_catalog() { let lower = body().to_ascii_lowercase(); for banned in [ "api_key", "bearer", - "_token", + "admin_token", "mnemonic", "https://api.", "modal", "[[topics", "holdout_records", "content_sha256", + "qwen/qwen3-0.6b", + "qwen/qwen3.8-0.6b", ] { assert!(!lower.contains(banned), "pin mentions {banned:?}"); } diff --git a/deploy/env/proof-challenge.env.example b/deploy/env/proof-challenge.env.example index d3f3532f8..62beab457 100644 --- a/deploy/env/proof-challenge.env.example +++ b/deploy/env/proof-challenge.env.example @@ -44,3 +44,15 @@ PROOF_FORCE_SIM=false # Seconds the eval image gets to score one artifact on the pod. # PROOF_EVAL_TIMEOUT_SECS=5400 + +# Live RLM judge InferenceOffer (operator state, never git). The eval image +# calls this backend to score miner submissions. Miners do not bind it. +# Missing or closed → can_score=false and submits 503. Auth is a secret file, +# staged into the harvest pod as teacher.env (OPENAI_API_KEY) — never in the +# offer JSON, never on GET /v1/status. Missing key on a live open offer → 503. +# PROOF_INFERENCE_OFFER_FILE=/run/base/proof/inference_offer.json +# PROOF_INFERENCE_API_KEY_FILE=/run/base/proof/inference_api_key +# Optional secret-backed origin when pin `[inference].base_url` and the topic +# omit one. Never log. Prefer the file form on a droplet. +# PROOF_INFERENCE_BASE_URL= +# PROOF_INFERENCE_BASE_URL_FILE=/run/base/proof/inference_base_url diff --git a/deploy/secrets/README.md b/deploy/secrets/README.md index c464051fb..15fd607b0 100644 --- a/deploy/secrets/README.md +++ b/deploy/secrets/README.md @@ -35,6 +35,9 @@ chmod 0400 deploy/secrets/gateway_admin_token | `proof/holdouts.json` | proof-challenge | Per-topic holdout records (array or map keyed by `topic_id`). **Never commit.** Verified at boot against each topic's `holdout_commitment`. Mode **0400**, uid **65532** | | `proof/baselines.json` | proof-challenge | Sealed baseline measurements keyed by topic id. **Never commit.** Mode **0400**, uid **65532** | | `proof/admin_tokens` | proof-challenge | One operator bearer per line for `POST /v1/admin/proof/topics` | +| `proof/inference_offer.json` | proof-challenge | Live RLM judge `InferenceOffer` (provider kind, origin, mode, model_ref, token caps, `config_commitment`, status). Consumed by proof-eval; **not** a miner training proxy. **Never commit.** Missing/closed → `can_score=false` / 503. Mode **0400**, uid **65532** | +| `proof/inference_api_key` | proof-challenge | Provider API key for the eval image. **Never commit, never log.** Mode **0400**, uid **65532** | +| `proof/inference_base_url` | proof-challenge | Optional secret-backed origin (`PROOF_INFERENCE_BASE_URL_FILE`) when pin `[inference].base_url` and the topic omit one. **Never commit, never log.** Mode **0400**, uid **65532** | | `bounty/admin_tokens` | bounty-challenge | Operator bearer for `POST /v1/admin/adjudicate` | | `bounty/session_secret` | bounty-challenge | Pairing session HMAC secret | diff --git a/docker-compose.yml b/docker-compose.yml index 684144ee8..d2f47e967 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -256,6 +256,8 @@ services: PROOF_HOLDOUT_FILE: /run/base/proof/holdouts.json PROOF_BASELINE_FILE: /run/base/proof/baselines.json PROOF_ADMIN_TOKENS_FILE: /run/base/proof/admin_tokens + PROOF_INFERENCE_OFFER_FILE: /run/base/proof/inference_offer.json + PROOF_INFERENCE_API_KEY_FILE: /run/base/proof/inference_api_key BASE_CHALLENGE_GATEWAY_ENDPOINT: ${BASE_CHALLENGE_GATEWAY_ENDPOINT:-http://gateway:8080} env_file: - path: ./deploy/env/proof-challenge.env diff --git a/docs/COMPLETENESS.md b/docs/COMPLETENESS.md index b64a4c39e..01ee67b49 100644 --- a/docs/COMPLETENESS.md +++ b/docs/COMPLETENESS.md @@ -74,7 +74,8 @@ specs (`DESIGN_CHALLENGE.md`, `PRISM.md`) remain for `xtask` gates. Leftover | Binary (`bins/proof-challenge`) | **done** | HTTP API on `:8100`. | | Miner CLI (`bins/ctx`) | **done** | `ctx proof submit|show|status|topics`. Unpinned digest / unwired harvest / no open topic → 503. | | Compose / images | **done** | Default compose + `images.yml` target `proof-challenge`. | -| Eval pin | **done** | `config/proof-pin.toml` — `eval_image` `ghcr.io/cortexlm/proof-eval`, digest `sha256:78b614a1…` (publish-proof-eval-image run 33892650063, commit `51f937c7`). Baked RLM judge `Qwen/Qwen3.8-0.6B` (evaluates miner submissions; not a miner training proxy). Empty digest is gone; live submits still **503** until harvest is wired, a baseline is sealed, and ≥1 topic is open. Do not re-pin a guessed sha256. | +| Eval pin | **done** | `config/proof-pin.toml` — `eval_image` `ghcr.io/cortexlm/proof-eval`, digest `sha256:78b614a1…` (publish-proof-eval-image run 33892650063, commit `51f937c7`). No HF bake; `proxy_model` stays empty. Live submits still **503** until harvest is wired, a baseline is sealed, and ≥1 topic is open. Do not re-pin a guessed sha256. | +| Inference offer | **v0** | Digest-pinned RLM **judge** backend (`proof-eval` / harvest call it). Pin `[inference]` defaults plus schema v1 / ceilings / modes / commitment. `config_commitment` hashes config knobs **and** `provider.base_url`; a topic that spoofs origin is **503** before lattice. Topic `require_judge_offer_commitment` is optional and not a miner bind. Live `InferenceOffer` is operator state. Auth is `PROOF_INFERENCE_API_KEY_FILE` staged as harvest `teacher.env` (never git, never `/v1/status`). Missing/closed/judge down / missing key → `can_score=false` / 503. No baked Qwen; architecture ≠ HF stays retired. | | Topics | **done** | sr25519 under the `proof` trust-root key (`base-proof-topic-v1`). Admin `POST /v1/admin/proof/topics`. A topic must be sealed to `open`. | | Holdout | **done** | Per-topic operator file (`PROOF_HOLDOUT_FILE`). Commitment in the topic document, never in the pin. `xtask proof-holdout --topic-id`. | | Live harvest | **done** | `crates/proof-harvest` over `harvest-pod`; `PROOF_FORCE_SIM` is local-only. | @@ -111,7 +112,7 @@ Agent/operator contracts: root [`AGENTS.md`](../AGENTS.md), [`deploy/AGENTS.md`] | Component | Status | Notes | |-----------|--------|-------| | bounty HTTP / adjudicate | **done** | Internal ingest: `POST /v1/pair` (sr25519) + `POST /v1/reports`; operator bearer on `GET /v1/reports` and `POST /v1/admin/adjudicate`. Scoring **fetches** CortexLM/backend `GET /v1/bounty/public/leaderboard` + `/reports` and emits signed leaves from those rows. Unset / unreachable / unparseable `BOUNTY_BACKEND_PUBLIC_URL` → `can_score: false`, reports **503**, and an all-`NoScore(ChallengeInternal)` leaf set that pays nobody while keeping D24. `BOUNTY_FORCE_SIM` is retired and ignored. | -| proof HTTP / topics | **done** | Operator-published signed topics; `POST /v1/submissions` with `topic_id`. Empty digest / unwired harvest / unsealed baseline / empty open set → **503**. Architecture must match the baked RLM judge id. Contamination / empty manifest persist **rejected** without rent. | +| proof HTTP / topics | **done** | Operator-published signed topics; `POST /v1/submissions` with `topic_id`. Empty digest / unwired harvest / unsealed baseline / empty open set / missing RLM judge offer / missing judge API key / spoofed topic origin → **503**. Architecture ≠ HF is retired. Contamination / empty manifest persist **rejected** without rent. | | Proof Lium harvest | **done** (fail-closed) | `crates/proof-harvest` over `harvest-pod` + leftover `prism-lium*` client. Live rent refuses without a `sha256:` eval digest; `PROOF_FORCE_SIM` is CI/local only; miner BYOK never logged. | | Retired challenge products | **removed** | `relearn*`, `design`, `prism` crates/bins/compose gone. Frozen specs remain. SQL migrations for historical tables stay applied. | | Phala / agent-v1 miner path | removed | External miners use HTTP submit only ([`external-miner/`](external-miner/)). | diff --git a/docs/PROOF.md b/docs/PROOF.md index f707fca4f..02e87711b 100644 --- a/docs/PROOF.md +++ b/docs/PROOF.md @@ -2,10 +2,10 @@ Live challenge id: **`proof`**. Emission **8000 bps** (80% of the subnet; bounty is 2000). This 20%/80% lock is independent of eval digest. Eval -digest `sha256:78b614a1…` is pinned (`ghcr.io/cortexlm/proof-eval`, baked -RLM judge `Qwen/Qwen3.8-0.6B` — evaluates miner submissions; **not** a miner -training proxy). Live submits still **503** until harvest is -wired, a baseline is sealed, and ≥1 topic is open. Do not invent a sha256. +digest `sha256:78b614a1…` is pinned (`ghcr.io/cortexlm/proof-eval`). The +RLM judge backend is the live `InferenceOffer` (not a baked HF proxy). Live +submits still **503** until harvest is wired, a baseline is sealed, and ≥1 +topic is open. Do not invent a sha256. Sum across the two live rows stays 10000. Port **8100** (local probe **28100**). @@ -23,6 +23,29 @@ baseline + an open topic are on the host. without InfiniBand” are *example solutions or example topics*, never a frozen catalog in git. - A topic may tighten a floor, never loosen it. Floors live in the pin. +- The RLM **judge** lives in a digest-pinned `proof-eval` image. Harvest + boots that image and both call the live master `InferenceOffer` as the + **judge backend** (not a miner training proxy, not an HF bake). Miners + do **not** bind submit to an `offer_id` as a train target; they still + post **claim + code + FLOPs + artifact** against a topic. No baked Qwen; + architecture ≠ HF id stays retired. Pin ceilings / modes / commitment + bound the **judge** offer; missing / closed / judge down → **503**. Topic + optional field: `require_judge_offer_commitment` (not a miner-facing + bind). The pin also carries complete `[inference]` judge defaults + (`provider`, `base_url` empty = secret-backed, `model`, `mode`, + `max_input_tokens`, `max_output_tokens`) plus schema v1, `allowed_modes`, + token ceilings, and `inference_offer_commitment_alg = sha256`. A topic's + signed `inference{…}` may **override** provider/model/mode and may **only + tighten** token caps vs those pin defaults. It must **not** redirect + origin: `config_commitment` hashes config knobs **and** `provider.base_url`; + a topic that spoofs `inference.base_url` fails closed (**503**) before + lattice. `require_judge_offer_commitment` (64-hex) pins the live judge + offer's `config_commitment` — mismatch → **503**. Missing or misconfigured + judge resolve → publish **400** (open topic) / score **503**. Empty pin + `model` / `base_url` is pre-launch fail-closed (like an empty digest). + Auth is `PROOF_INFERENCE_API_KEY_FILE`, staged into the harvest pod as + `teacher.env` (`OPENAI_API_KEY`) — never git, never `/v1/status`. Missing + key on a live open offer → **503**. `proxy_model` stays empty. - A baseline must be sealed (`script_sha256` + `metrics_commitment`) to open. Nobody is paid for beating a number nobody measured. - 8000 bps is split equally across currently `open` topics. Each topic then @@ -97,12 +120,15 @@ Trust-root keygen is the throwaway owner path in ## HTTP - `GET /health`, `GET /v1/status` — `can_score`, `eval_backend`, `force_sim`, - `live_harvest_wired`, `baseline_sealed`. Never leak endpoints or records. + `live_harvest_wired`, `baseline_sealed`, public pin `inference` judge + defaults (no origin), public `inference_offer` (RLM judge backend). Never + leak origins, keys, or holdout records. - `GET /v1/proof/topics`, `GET /v1/proof/topics/{id}` - `POST /v1/admin/proof/topics` — operator bearer; verify sig/schema/floors/seal before `open` - `POST /v1/submissions` **requires** `topic_id`. Missing/unknown/not-open → - **400**. Architecture ≠ proxy → **400**. Zero open / unsealed baseline / - empty digest / agent down → **503**. Refusals must **not** persist rows. + **400**. Miners do **not** bind the judge offer. Zero open / unsealed + baseline / empty digest / missing or closed RLM judge backend / agent down + → **503**. Refusals must **not** persist rows. - Submit fields miners must send: `claim` (what the recipe achieved), `declared_flops` (≤ topic budget), `artifact_digest` of a **reproducible train/eval recipe** (code under budget, not weights-only), plus `manifest`. @@ -148,6 +174,13 @@ takes the topic. "wall_budget_s": 14400 }, "flops_budget": 2000000000000000000, + "inference": { + "provider": "openai_compatible", + "model": "master-proxy-v0", + "mode": "chat", + "max_input_tokens": 4096, + "max_output_tokens": 2048 + }, "status": "draft" } ``` @@ -174,5 +207,14 @@ this document publishes; scoring fail-closes until the harness exists. ``` These JSON bodies are documentation. Publishing requires a holdout -commitment, a sealed baseline (to open), and an sr25519 signature under the -`proof` trust-root key. +commitment, a sealed baseline, a signed `inference{…}` that does not loosen +pin **judge** defaults, and an sr25519 signature under the `proof` +trust-root key. Omitted inference fields inherit the pin; `open` requires a +complete resolved judge config (provider + model + mode + tokens). Empty pin +model with no topic `model` is **400** at publish. + +`GET /v1/status` exposes pin `inference` public judge defaults (`provider`, +`model`, `mode`, token caps) and `inference_offer` **public fields only** +(`offer_id`, `provider_kind`, `mode`, `model_ref`, token caps, +`config_commitment`, `status`). It never leaks `base_url`, API keys, or file +paths. Miners do not call this backend. diff --git a/docs/external-miner/proof.md b/docs/external-miner/proof.md index 46d52af5a..961671686 100644 --- a/docs/external-miner/proof.md +++ b/docs/external-miner/proof.md @@ -10,10 +10,12 @@ and it is **not** waiting on a digest retune to 5000/5000. **CLI:** `ctx proof topics`, then `ctx proof submit` (install: [README](./README.md)) **Pin:** [`config/proof-pin.toml`](../../config/proof-pin.toml) -**Eval image:** `ghcr.io/cortexlm/proof-eval@sha256:78b614a1f51ce5dd80076c4e343a2b31b85d6c36025e02836cb83929867e7009` -**RLM judge (eval-image InferenceOffer):** `Qwen/Qwen3.8-0.6B` — this is the -judge agent baked into the eval image that evaluates miner submissions. It is -**not** a model miners train against. +**Eval image:** `ghcr.io/cortexlm/proof-eval@sha256:78b614a1f51ce5dd80076c4e343a2b31b85d6c36025e02836cb83929867e7009` + +You submit **claim + code + FLOPs + artifact** against a topic. You do +not bind an offer id. The digest-pinned `proof-eval` image (harvest boots +it) calls the master's `InferenceOffer` as the RLM **judge** backend. +No baked Qwen; architecture is not an HF id check. Miner pays Lium (`LIUM_API_KEY` / `X-Lium-Api-Key`). @@ -38,6 +40,13 @@ the public split, and the harness fills holdout NLL / throughput. Holdout records stay sealed until after your submission digest is frozen. You never see them. +`GET /challenge/proof/v1/status` shows `can_score`, `eval_backend`, +`force_sim`, `live_harvest_wired`, `baseline_sealed`, public pin `inference` +judge defaults (provider, model, mode, token caps — never the origin), and +the public RLM judge `inference_offer` (id, kind, mode, model_ref, token +caps, commitment, status). It never leaks holdout records, teacher hosts, +origins, or keys. + Muon, token superposition, and “decentralized training without InfiniBand” are *examples* of solutions or of topics — they are not the product. @@ -57,8 +66,9 @@ curl -sS https://network.cortex.foundation/challenge/proof/v1/status `GET /challenge/proof/v1/status` shows `can_score`, `eval_backend`, `force_sim`, `live_harvest_wired`, `baseline_sealed`, `eval_image_digest`, -`proxy_model`, and `open_topics`. It never leaks holdout records or teacher -hosts. +public pin `inference` (no origin), public RLM judge `inference_offer`, and +`open_topics`. It never leaks holdout records, teacher hosts, origins, or +keys. `can_score: false` means submits **503**. Nothing is stored and nothing is rented. @@ -66,7 +76,7 @@ rented. | Status field | What it means | |--------------|----------------| | `eval_image_digest` | Must be a `sha256:…` pin (live pin is `sha256:78b614a1…`). Empty → **503** | -| `proxy_model` | Id of the RLM judge agent baked into the eval image (not a miner training proxy). Live pin: `Qwen/Qwen3.8-0.6B`. Submit `architecture` must match | +| `inference_offer` | Public RLM **judge** backend (id, kind, mode, model_ref, token caps, commitment, status). Missing/closed/misconfigured → **503**. You do not pass an offer id | | `open_topics` empty | No currently `open` signed topic with a sealed baseline → **503** | | `baseline_sealed: false` | An open topic without `script_sha256` + `metrics_commitment` → **503** | | `live_harvest_wired: false` | Live RLM harvest is not connected → **503** | @@ -146,7 +156,7 @@ code against the public split and checks the claim against those public numbers. A claim the code cannot support is `unreproduced_claim` / reject. ```bash -ctx proof status # can_score, proxy_model, eval_image_digest +ctx proof status # can_score, inference_offer, eval_image_digest ctx proof topics # pick an open topic_id; read flops_budget + payout_mode ctx proof submit \ @@ -155,7 +165,6 @@ ctx proof submit \ --artifact-digest \ --claim "beat sealed baseline holdout NLL by 0.04 at 1.2e18 FLOPs" \ --declared-flops 1500000000000000000 \ - --architecture Qwen/Qwen3.8-0.6B \ --train-dataset my-mix-v0 ``` @@ -174,7 +183,6 @@ curl -sS -X POST https://network.cortex.foundation/challenge/proof/v1/submission "artifact_digest": "", "claim": "beat sealed baseline holdout NLL by 0.04 at 1.2e18 FLOPs", "declared_flops": 1500000000000000000, - "architecture": "Qwen/Qwen3.8-0.6B", "manifest": { "train_content_hashes": [], "train_dataset_ids": ["my-mix-v0"] @@ -185,6 +193,10 @@ curl -sS -X POST https://network.cortex.foundation/challenge/proof/v1/submission `claim` and `declared_flops` are **required**. The control plane scores a claim against public numbers and checks FLOPs against the topic budget. +Poll `GET /challenge/proof/v1/submissions/{id}`. While `can_score` is +`false` (empty digest, missing/closed RLM judge backend, incomplete pin+topic +judge config, no open sealed topic), submissions answer **503**. + ### Required POST JSON `POST https://network.cortex.foundation/challenge/proof/v1/submissions` @@ -196,7 +208,6 @@ claim against public numbers and checks FLOPs against the topic budget. | `artifact_digest` | yes | SHA-256 hex of the recipe bytes | | `claim` | yes | Non-empty string: NL of what improved | | `declared_flops` | yes | `u64`, must be `≤ topic.flops_budget` | -| `architecture` | yes | Must equal the baked RLM judge id (`Qwen/Qwen3.8-0.6B`) | | `manifest.train_content_hashes` | yes (array) | Shard hashes you trained on (may be `[]` if you declare dataset ids) | | `manifest.train_dataset_ids` | yes (array) | Corpus ids you trained on (may be `[]` if you declare hashes) | | `artifact_uri` | no | Locator for the same bytes as `artifact_digest` | @@ -233,14 +244,13 @@ Refusals (**400** / **503**) do **not** persist a submission row. | **400** `topic_id is required` | Missing `topic_id` | no | no | | **400** `unknown topic` | `topic_id` not published | no | no | | **400** `topic is not open` | Draft / closed / outside epoch window | no | no | -| **400** `architecture is not the topic/pin proxy` | `architecture` ≠ baked proxy | no | no | -| **400** `proxy not baked` | Pin `proxy_model` empty | no | no | | **400** `declared_flops exceeds the topic budget` | `declared_flops > topic.flops_budget` | no | no | | **400** invalid `miner_hotkey` / `artifact_digest` | Not 64 hex | no | no | | **503** empty `eval_image_digest` | Digest not pinned | no | no | | **503** zero open sealed topics | Nothing to score against | no | no | | **503** unsealed baseline | Topic open without both seal hashes | no | no | | **503** live harvest down / unparseable agent verdict | Host cannot judge | no | no | +| **503** missing / closed RLM judge backend | Live `InferenceOffer` not scoring | no | no | | **201** `rejected` + `contamination_evidence_missing` | Empty manifest | **yes** (rejected) | **no** | | **201** `rejected` + contamination | Holdout shard / corpus id in `manifest` | **yes** (rejected) | **no** | diff --git a/docs/external-miner/troubleshoot.md b/docs/external-miner/troubleshoot.md index 9631f898d..fda93b773 100644 --- a/docs/external-miner/troubleshoot.md +++ b/docs/external-miner/troubleshoot.md @@ -33,12 +33,11 @@ Install `ctx` from [README](./README.md). Proof miners pay Lium | Symptom | Likely cause | What to check | |---------|--------------|---------------| | `400` missing / unknown / not-open `topic_id` | Topic is not currently open | `ctx proof topics`. The refusal is not a submission | -| `400` architecture / `proxy not baked` | `architecture` is not the RLM judge id the pin bakes | Copy `proxy_model` from `ctx proof status`. That id is the eval-image judge, not a miner training proxy | | `400` `declared_flops exceeds the topic budget` | `declared_flops > topic.flops_budget` | Cap declared FLOPs at the open topic's budget | | `400` invalid hotkey / `artifact_digest` | Not 64 hex | Both must be 64 hex characters | | `rejected` with `contamination_evidence_missing` | Empty `manifest` | Declare `train_content_hashes` or `train_dataset_ids` | | `rejected` with contamination / cheat code | Holdout overlap, unreproduced claim, strawman AdamW, … | Read the verdict `cheat_codes`. Contamination rejects without rent | -| HTTP 503 on submit | Empty `eval_image_digest`, zero open topics, unsealed baseline, or harvest down | `ctx proof status` → `can_score`. Live digest is `sha256:78b614a1…`. Empty digest still 503. Do not invent a digest. Nothing was rented | +| HTTP 503 on submit | Empty `eval_image_digest`, zero open topics, unsealed baseline, harvest down, or missing/closed RLM judge | `ctx proof status` → `can_score`. Live digest is `sha256:78b614a1…`. Empty digest still 503. Do not invent a digest. Nothing was rented | ## Off challenges diff --git a/eval/README.md b/eval/README.md index 8aad9aeab..fdf422c91 100644 --- a/eval/README.md +++ b/eval/README.md @@ -13,9 +13,13 @@ Harvest wrappers print `PROOF_METRICS=` and `PROOF_EVAL_OK`. non-zero with no marker. Pin the **scoring** image (`eval/Dockerfile.scoring`, CUDA + torch), never -the contract-only digest. Proxy baked into this image: `Qwen/Qwen3.8-0.6B`. -Fabric: no InfiniBand, no NVLink, no NCCL fast path, 12.5 Gbit/s cap. +the contract-only digest. The RLM **judge** is the live `InferenceOffer` +(OpenAI-compatible HTTP). Auth is `teacher.env` (`OPENAI_API_KEY` / +`PROOF_INFERENCE_API_KEY`) staged by harvest — never request.json, never +git. No HF bake into the judge path. Fabric: no InfiniBand, no NVLink, no +NCCL fast path, 12.5 Gbit/s cap. No secrets, holdout text, teacher hosts, or Modal references are baked in. -Shard bytes arrive via `PROOF_HOLDOUT_STORE/`. Proxy -weights via `PROOF_PROXY_MODEL_DIR` or `PROOF_ALLOW_MODEL_DOWNLOAD=1`. +Shard bytes arrive via `PROOF_HOLDOUT_STORE/`. Optional +local measurement weights via `PROOF_PROXY_MODEL_DIR` or +`PROOF_ALLOW_MODEL_DOWNLOAD=1`. diff --git a/eval/src/proof_eval/cli.py b/eval/src/proof_eval/cli.py index f271e3d75..09ed87918 100644 --- a/eval/src/proof_eval/cli.py +++ b/eval/src/proof_eval/cli.py @@ -28,6 +28,7 @@ marker_line, ) from .fabric import selftest as fabric_selftest +from .judge import require_judge from .request import read_request from .agent import inspect from .harness import measure, require_runtime @@ -101,7 +102,7 @@ def _selftest() -> int: def _score(request_path: Path, out: Path, *, baseline: bool) -> int: request = read_request(request_path) - require_baked(request.proxy_model) + require_judge(request) from .fabric import enforce enforce(request.constraints) diff --git a/eval/src/proof_eval/harness.py b/eval/src/proof_eval/harness.py index 7d11d8cdd..6ad754cbe 100644 --- a/eval/src/proof_eval/harness.py +++ b/eval/src/proof_eval/harness.py @@ -57,7 +57,9 @@ def measure(request: HarvestRequest, artifact_dir: str | None) -> dict[str, Any] import torch from transformers import AutoModelForCausalLM, AutoTokenizer - proxy = request.proxy_model + proxy = (request.proxy_model or request.model_ref or "").strip() + if not (artifact_dir or proxy): + raise ContractError("no model to measure") try: tok = AutoTokenizer.from_pretrained(proxy, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( diff --git a/eval/src/proof_eval/judge.py b/eval/src/proof_eval/judge.py new file mode 100644 index 000000000..9456ce7ad --- /dev/null +++ b/eval/src/proof_eval/judge.py @@ -0,0 +1,88 @@ +"""Call the live InferenceOffer as the RLM judge backend. + +Auth comes from harvest-pod `teacher.env` (`OPENAI_API_KEY` / +`PROOF_INFERENCE_API_KEY`). The key is never read from request.json and +must never be logged. +""" + +from __future__ import annotations + +import json +import os +import urllib.error +import urllib.request +from typing import Any + +from .contract import ContractError +from .request import HarvestRequest + +_KEY_ENV = ("PROOF_INFERENCE_API_KEY", "OPENAI_API_KEY") + + +def load_judge_api_key() -> str: + for name in _KEY_ENV: + value = os.environ.get(name, "").strip() + if value: + return value + raise ContractError("inference API key missing; refuse scoring") + + +def judge_url(base_url: str, mode: str) -> str: + origin = base_url.strip().rstrip("/") + if mode == "embeddings": + suffix = "/embeddings" + elif mode == "completions": + suffix = "/completions" + else: + suffix = "/chat/completions" + if origin.endswith(suffix): + return origin + return origin + suffix + + +def _payload(request: HarvestRequest) -> dict[str, Any]: + max_out = max(1, min(int(request.max_output_tokens or 16), 64)) + claim = request.claim[:2048] + if request.mode == "embeddings": + return {"model": request.model_ref, "input": claim} + if request.mode == "completions": + return {"model": request.model_ref, "prompt": claim, "max_tokens": max_out} + return { + "model": request.model_ref, + "max_tokens": max_out, + "messages": [ + { + "role": "system", + "content": "You are the Cortex Proof RLM judge. Reply with one short ack.", + }, + {"role": "user", "content": claim}, + ], + } + + +def call_judge(request: HarvestRequest, api_key: str, *, timeout_s: float = 60.0) -> None: + """POST an authenticated judge request. Fail closed on missing auth or HTTP error.""" + key = api_key.strip() + if not key: + raise ContractError("inference API key missing; refuse scoring") + url = judge_url(request.base_url, request.mode) + body = json.dumps(_payload(request), separators=(",", ":")).encode("utf-8") + req = urllib.request.Request(url, data=body, method="POST") + req.add_header("Authorization", f"Bearer {key}") + req.add_header("Content-Type", "application/json") + try: + with urllib.request.urlopen(req, timeout=timeout_s) as resp: + status = int(getattr(resp, "status", 200) or 200) + if status < 200 or status >= 300: + raise ContractError(f"judge HTTP {status}") + _ = resp.read(256) + except ContractError: + raise + except urllib.error.HTTPError as exc: + raise ContractError(f"judge HTTP {exc.code}") from None + except Exception as exc: # noqa: BLE001 + raise ContractError(f"judge request failed: {type(exc).__name__}") from None + + +def require_judge(request: HarvestRequest) -> None: + call_judge(request, load_judge_api_key()) diff --git a/eval/src/proof_eval/request.py b/eval/src/proof_eval/request.py index 60e7fca47..6c349531e 100644 --- a/eval/src/proof_eval/request.py +++ b/eval/src/proof_eval/request.py @@ -8,12 +8,17 @@ from pathlib import Path from typing import Any -from .contract import ContractError, DEFAULT_PROXY, PROOF_METRICS_SCHEMA +from .contract import ContractError, PROOF_METRICS_SCHEMA CHALLENGE_ID = "proof" HOLDOUT_DOMAIN = b"base-proof-holdout-v1" +def _is_http_origin(url: str) -> bool: + u = url.strip() + return u.startswith("http://") or u.startswith("https://") + + @dataclass(frozen=True) class Constraints: no_infiniband: bool = False @@ -49,6 +54,14 @@ class HarvestRequest: artifact_digest: str topic_id: str family: str + inference_offer_id: str + provider_kind: str + base_url: str + mode: str + model_ref: str + max_input_tokens: int + max_output_tokens: int + config_commitment: str proxy_model: str eval_image_digest: str holdout_commitment: str @@ -71,15 +84,33 @@ def from_dict(cls, raw: dict[str, Any]) -> HarvestRequest: artifact = str(raw.get("artifact_digest", "")).strip() topic = str(raw.get("topic_id", "")).strip() family = str(raw.get("family", "")).strip() - proxy = str(raw.get("proxy_model", "")).strip() or DEFAULT_PROXY + if "api_key" in raw or "openai_api_key" in raw: + raise ContractError("api_key must not appear in request.json; use teacher.env") + offer_id = str(raw.get("inference_offer_id", "")).strip() + provider_kind = str(raw.get("provider_kind", "")).strip() + base_url = str(raw.get("base_url", "")).strip() + mode = str(raw.get("mode", "")).strip() or "chat" + model_ref = str(raw.get("model_ref", "")).strip() + commitment = str(raw.get("config_commitment", "")).strip() + if not offer_id: + raise ContractError("inference_offer_id is required") + if not _is_http_origin(base_url): + raise ContractError("base_url must be an http(s) origin") + if not model_ref: + raise ContractError("model_ref is required") + if len(commitment) != 64 or any(c not in "0123456789abcdefABCDEF" for c in commitment): + raise ContractError("config_commitment must be 64 hex chars") + proxy = str(raw.get("proxy_model", "")).strip() digest = str(raw.get("eval_image_digest", "")).strip() - commitment = str(raw.get("holdout_commitment", "")).strip() + holdout_commit = str(raw.get("holdout_commitment", "")).strip() claim = str(raw.get("claim", "")).strip() if not submission or not artifact or not topic: raise ContractError("submission_digest, artifact_digest, and topic_id are required") if not digest.startswith("sha256:") or len(digest) < 71: raise ContractError("eval_image_digest is not a sha256 pin") - if len(commitment) != 64 or any(c not in "0123456789abcdefABCDEF" for c in commitment): + if len(holdout_commit) != 64 or any( + c not in "0123456789abcdefABCDEF" for c in holdout_commit + ): raise ContractError("holdout_commitment must be 64 hex chars") if not claim: raise ContractError("claim is required (recipe, not weights alone)") @@ -93,9 +124,17 @@ def from_dict(cls, raw: dict[str, Any]) -> HarvestRequest: artifact_digest=artifact, topic_id=topic, family=family, + inference_offer_id=offer_id, + provider_kind=provider_kind or "openai_compatible", + base_url=base_url, + mode=mode, + model_ref=model_ref, + max_input_tokens=int(raw.get("max_input_tokens") or 0), + max_output_tokens=int(raw.get("max_output_tokens") or 0), + config_commitment=commitment, proxy_model=proxy, eval_image_digest=digest, - holdout_commitment=commitment, + holdout_commitment=holdout_commit, constraints=Constraints.from_dict(raw.get("constraints") or {}), flops_budget=int(raw.get("flops_budget") or 0), wall_budget_s=int(raw.get("wall_budget_s") or 0), @@ -103,7 +142,7 @@ def from_dict(cls, raw: dict[str, Any]) -> HarvestRequest: holdout=holdout, ) got = holdout_commitment(req.holdout) - if got.lower() != commitment.lower(): + if got.lower() != holdout_commit.lower(): raise ContractError("holdout records do not hash to holdout_commitment") return req diff --git a/eval/tests/test_contract.py b/eval/tests/test_contract.py index d0119a64c..719dc14c1 100644 --- a/eval/tests/test_contract.py +++ b/eval/tests/test_contract.py @@ -3,14 +3,16 @@ from __future__ import annotations import json -from pathlib import Path +from http.server import BaseHTTPRequestHandler, HTTPServer +from threading import Thread import pytest from proof_eval.baked import baked_proxies, require_baked from proof_eval.contract import DEFAULT_PROXY, METRICS_MARKER, OK_MARKER from proof_eval.fabric import DT_NO_IB_GBPS, enforce -from proof_eval.request import Constraints, HarvestRequest, canonical_json +from proof_eval.judge import call_judge, load_judge_api_key, require_judge +from proof_eval.request import Constraints, HarvestRequest, canonical_json, holdout_commitment def test_markers_match_harvest_pod() -> None: @@ -59,7 +61,20 @@ def test_canonical_json_sorts_keys() -> None: assert canonical_json({"b": 1, "a": {"d": [1, 2], "c": "x"}}) == '{"a":{"c":"x","d":[1,2]},"b":1}' -def test_request_refuses_empty_claim() -> None: +def _holdout() -> list[dict]: + return [ + { + "id": 1, + "split": "web_ood", + "dataset_id": "synthetic-dev", + "content_sha256": "aa" * 32, + "token_count": 2048, + } + ] + + +def _raw(**over: object) -> dict: + holdout = over.pop("holdout", _holdout()) raw = { "schema_version": 1, "challenge_id": "proof", @@ -67,20 +82,94 @@ def test_request_refuses_empty_claim() -> None: "artifact_digest": "a", "topic_id": "dt-no-ib-v0", "family": "throughput", - "proxy_model": DEFAULT_PROXY, + "inference_offer_id": "master-v0", + "provider_kind": "openai_compatible", + "base_url": "http://127.0.0.1:8000/v1", + "mode": "chat", + "model_ref": "master-proxy-v0", + "max_input_tokens": 4096, + "max_output_tokens": 256, + "config_commitment": "ab" * 32, "eval_image_digest": "sha256:" + "ab" * 32, - "holdout_commitment": "00" * 32, + "holdout_commitment": holdout_commitment(holdout), "constraints": {}, "flops_budget": 1, "wall_budget_s": 1, - "claim": "", - "holdout": [{ - "id": 1, - "split": "web_ood", - "dataset_id": "synthetic-dev", - "content_sha256": "aa" * 32, - "token_count": 2048, - }], + "claim": "beats the sealed reference under the cap", + "holdout": holdout, } + raw.update(over) + return raw + + +def test_request_refuses_empty_claim() -> None: with pytest.raises(Exception, match="claim"): - HarvestRequest.from_dict(raw) + HarvestRequest.from_dict(_raw(claim="")) + + +def test_request_requires_judge_origin_and_ignores_no_hf_default() -> None: + req = HarvestRequest.from_dict(_raw()) + assert req.base_url == "http://127.0.0.1:8000/v1" + assert req.model_ref == "master-proxy-v0" + assert req.proxy_model == "" + with pytest.raises(Exception, match="http"): + HarvestRequest.from_dict(_raw(base_url="ftp://evil")) + with pytest.raises(Exception, match="api_key"): + HarvestRequest.from_dict(_raw(api_key="sk-should-never-be-here")) + + +def test_judge_key_comes_from_env_not_request(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + monkeypatch.delenv("PROOF_INFERENCE_API_KEY", raising=False) + with pytest.raises(Exception, match="API key"): + load_judge_api_key() + monkeypatch.setenv("OPENAI_API_KEY", "sk-from-teacher-env") + assert load_judge_api_key() == "sk-from-teacher-env" + + +class _Judge(BaseHTTPRequestHandler): + last_auth = "" + last_path = "" + last_body = b"" + + def do_POST(self) -> None: # noqa: N802 + length = int(self.headers.get("Content-Length") or 0) + _Judge.last_auth = self.headers.get("Authorization") or "" + _Judge.last_path = self.path + _Judge.last_body = self.rfile.read(length) + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"id":"x","choices":[{"message":{"content":"ok"}}]}') + + def log_message(self, fmt: str, *args: object) -> None: + return + + +def test_judge_sends_bearer_from_teacher_env(monkeypatch: pytest.MonkeyPatch) -> None: + server = HTTPServer(("127.0.0.1", 0), _Judge) + port = server.server_address[1] + thread = Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + raw = _raw(base_url=f"http://127.0.0.1:{port}/v1") + req = HarvestRequest.from_dict(raw) + monkeypatch.setenv("PROOF_INFERENCE_API_KEY", "sk-live-not-a-real-secret") + require_judge(req) + assert _Judge.last_auth == "Bearer sk-live-not-a-real-secret" + assert _Judge.last_path.endswith("/chat/completions") + body = json.loads(_Judge.last_body.decode()) + assert body["model"] == "master-proxy-v0" + assert "sk-live" not in json.dumps(body) + finally: + server.shutdown() + + +def test_judge_refuses_without_a_key(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + monkeypatch.delenv("PROOF_INFERENCE_API_KEY", raising=False) + req = HarvestRequest.from_dict(_raw()) + with pytest.raises(Exception, match="API key"): + require_judge(req) + with pytest.raises(Exception, match="API key"): + call_judge(req, " ")