Skip to content

fix(challenges): boot when sk/session placeholders are empty - #226

Merged
echobt merged 1 commit into
mainfrom
cursor/staging-challenge-boot-f403
Sep 5, 2026
Merged

fix(challenges): boot when sk/session placeholders are empty#226
echobt merged 1 commit into
mainfrom
cursor/staging-challenge-boot-f403

Conversation

@echobt

@echobt echobt commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Staging master has been crash-looping bounty-challenge and proof-challenge (exit 1 → Restarting) since well before #223. GHA 33952922508 is the same pattern as #224 / #222 / #210: gateway /healthz is fine, /challenge/{bounty,proof}/health stays 502/503 because the processes never listen.

Root cause is not InferenceOffer. #223 already treats a missing/closed offer as warn + can_score=false / submit 503. ProofPin::validate accepts the committed pin with empty [inference].model / base_url. Compose PROOF_INFERENCE_OFFER_FILE=/run/base/proof/inference_offer.json is a file inside the existing deploy/secrets/proof directory mount — a missing file is ENOENT, not a Docker bind-mount poison.

The hard boot errors are the compose-always-set secret files:

Env Host path What remote-deploy does if missing Previous bin behavior
BASE_CHALLENGE_SK_FILE deploy/secrets/bounty_sk, proof_sk : > empty file (so Docker does not create a directory at a file mount) load_challenge_secretErrexit 1
BOUNTY_SESSION_SECRET_FILE deploy/secrets/bounty/session_secret same empty placeholder empty file → exit 1

Both services therefore die before /health. The 502/503 oscillation is the gateway proxy hitting a restarting backend, not a failed backend registry (reseed returned 201).

This PR:

  • bounty: invalid/empty BASE_CHALLENGE_SK_FILE → warn, skip emitter (same as unset)
  • bounty: empty/unreadable BOUNTY_SESSION_SECRET_FILE → warn, ephemeral 32 bytes
  • proof: invalid/empty BASE_CHALLENGE_SK_FILE → warn, keep serving /health
  • remote-deploy.sh: fill session_secret from urandom when missing or 0-length (matches local-e2e.sh)

Not invented: no InferenceOffer JSON, no judge provider URL/key. Missing offer stays fail-closed on submit.

Host files that must still exist for real weight (not for boot):

  • deploy/secrets/bounty_sk and deploy/secrets/proof_sk — 32 raw bytes or 64 hex, pubs matching the staging trust root (#215 ceremony). Empty placeholders keep /health up but cannot sign leaves (bounty seal will 409 D24 until a real mini-secret is installed).
  • deploy/secrets/bounty/session_secret — non-empty for pairing that survives restart (remote-deploy now generates one).
  • deploy/secrets/proof/inference_offer.json and inference_api_keyoptional. Missing → warn, can_score=false, submit 503. Do not invent a provider.

Greptile

Every PR is reviewed by Greptile before merge. Config: .greptile/.

  • Greptile has reviewed this PR; findings are fixed or answered
  • If the bot is silent, I commented @greptileai review

Test plan

  • cargo test -p bounty-challenge-bin -p proof-challenge-bin
  • cargo fmt --all -- --check
  • cargo clippy -p bounty-challenge-bin -p proof-challenge-bin --all-targets -- -D warnings
  • cargo run -p xtask -- loc-cap (bins still well under cap)

Risk

No emission / scoring / consensus change. Staging /health should come up without a live judge offer. Operators still need real bounty_sk / proof_sk before a seal can include signed leaves.

Naming

I did not rename BASE_* environment variables, deployed host paths
(/opt/base, /run/base, …), GHCR baseintelligence/base package names, or
base-*-v1 cryptographic domain tags, unless this PR’s purpose is a coordinated
cutover documented in docs/NAMING.md.

Open in Web Open in Cursor 

Compose always sets BASE_CHALLENGE_SK_FILE and
BOUNTY_SESSION_SECRET_FILE. remote-deploy materializes empty files so
Docker does not create directories at those paths; both bins treated
that as a hard boot error (exit 1), which crash-looped staging master
and 502/503'd /challenge/*/health.

Missing/invalid sk now warns and skips signing. Empty session secret
falls back to ephemeral bytes. InferenceOffer missing stays warn +
can_score false (unchanged). remote-deploy fills a 32-byte session
secret from urandom when the file is missing or 0-length.

Co-authored-by: Mathis <echobt@users.noreply.github.com>
@echobt
echobt marked this pull request as ready for review September 5, 2026 07:58
@echobt
echobt merged commit 1dd07f7 into main Sep 5, 2026
5 checks passed
@echobt
echobt deleted the cursor/staging-challenge-boot-f403 branch September 5, 2026 08:03
@greptile-apps

greptile-apps Bot commented Sep 5, 2026

Copy link
Copy Markdown

Greptile Summary

The bounty and proof challenge services now remain available when deployment-created signing-key placeholders are empty or unavailable. Bounty falls back to fresh in-memory session material when its session-secret file is unreadable or empty, while remote deployment creates persistent random session material when possible.

T-Rex validation blocked

The completed health-check evidence could not be uploaded or have its artifact references verified because the validation environment disabled further access after reaching its step limit. The saved execution records show both updated health endpoints returned HTTP 200 with empty and unavailable placeholders.

Confidence Score: 5/5

Safe to merge: no unresolved product defects were identified, and the recorded executable checks show the updated services remain reachable with the deployment placeholder states addressed by this change.

No final findings remain. The recorded checks exercised the changed boot paths and health endpoints, but their artifact upload status could not be verified after the validation environment disabled further access.

Files Needing Attention: No files require follow-up.

T-Rex T-Rex Logs

What T-Rex did

  • The runtime reported the blocker Maximum steps for this agent have been reached; tools are now disabled, leaving no available tool to upload artifacts or validate references, and locally saved artifact paths that would need upload verification are listed.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "fix(challenges): boot when sk/session pl..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants