Activity

1. Spyros-Vl commented on Oct 12, 2026

   

   Spyros-Vl

   on Oct 12, 2026

   More actions

   I had the same exact issue and prepared the following bug report, I will post it here istead of making duplicate bug.

   v1.10.4: every sandbox execution fails — SANDBOX_PIPE_ADDON is never set in the shipped runner images

   Severity: v1.10.4 cannot execute code at all. The container reports healthy, so this is only visible on a real execution.

   Affected: v1.10.4 only. It is the release that introduced the addon.

   ---

   1. Symptom

   Every execution fails. POST /api/v2/execute on sandbox-runner returns 500, surfacing upstream as sandbox_execution_failed:

   {"level":50,"job":"sbx_k57Vo0mywOHC7RkV2DGHoA","err":{
     "type":"Error",
     "message":"Failed to spawn nsjail: Cannot find module '/sandbox_api/.build/anonymous-pipes.node'\nRequire stack:\n- /sandbox_api/.build/index.js"
   },"msg":"Error executing job"}

   Worth stressing: sandbox-runner reports healthy throughout. The healthcheck never spawns nsjail, so container state, /v1/health and the whole compose stack all look correct. Only an actual execution reveals it.

   2. Root cause

   api/src/pipe-process.ts:9 resolves the native addon:

   native ??= require(process.env.SANDBOX_PIPE_ADDON
     || path.resolve(__dirname, '..', '.build', 'anonymous-pipes.node'));

   SANDBOX_PIPE_ADDON is not set in the shipped runner images, so it falls back to .build/anonymous-pipes.node — which does not exist. The addon is built to /usr/local/lib/anonymous-pipes.node instead.

   In api/Dockerfile, ENV SANDBOX_PIPE_ADDON is set in exactly two stages:

    48:FROM nsjail-builder AS nsjail-ipc-test
    49:ENV SANDBOX_PIPE_ADDON=/usr/local/lib/anonymous-pipes.node     ← test stage
    90:FROM oven/bun:1.4.2-debian AS sandbox-build
   135:ENV SANDBOX_PIPE_ADDON=/usr/local/lib/anonymous-pipes.node     ← not inherited below
   

   Neither reaches the runner targets:

   252:FROM fedora:43 AS sandbox-runner-base          ← no ENV
   277:FROM debian:bookworm AS sandbox-rootfs-image
       COPY --from=sandbox-build / /sandbox-rootfs/   ← copies FILES; ENV does not travel
   294:FROM sandbox-runner-base AS sandbox-runner-baked
   304:FROM sandbox-runner-base AS sandbox-runner
   315:FROM sandbox-runner-baked AS sandbox-runner-true    ← KVM_ENABLED=true
   316:FROM sandbox-runner AS sandbox-runner-false         ← KVM_ENABLED=false
   

   COPY --from=sandbox-build / /sandbox-rootfs/ carries the addon file into the guest, but ENV is image metadata and does not survive a COPY. Both runner targets descend from sandbox-runner-base (fedora:43), which never sets the variable, and no entrypoint sets it either.

   So the file is present and the pointer to it is missing.

   Confirmed by loop-mounting the baked guest image (sandbox-rootfs.img) from a running sandbox-runner:

   # addon is in the guest:
   /mnt/rfs/usr/local/lib/anonymous-pipes.node
   
   # but the fallback directory has no .node file:
   /mnt/rfs/sandbox_api/.build/  ->  index.js   tool-call-pipe-broker.cjs
   

   And in the running container:

   $ docker exec sandbox-runner printenv | grep SANDBOX_PIPE_ADDON
   (nothing)
   $ docker exec sandbox-runner ls /usr/local/lib/anonymous-pipes.node
   No such file or directory
   

   3. Why operators cannot work around it

   Setting SANDBOX_PIPE_ADDON via .env or a compose override does not work for the KVM path. launcher/src/main.rs filters the guest environment through an explicit allowlist:

   fn is_allowed_guest_env_key(key: &str, egress_gateway_enabled: bool) -> bool {
       const ALLOW_EXACT: &[&str] = &[ /* ... */ ];

   SANDBOX_PIPE_ADDON appears zero times in launcher/src/main.rs and is not in ALLOW_EXACT, so it is dropped before reaching the guest. There is no configuration-level fix.

   4. Why CI did not catch it

   The IPC test stage sets the variable that the shipped stages do not:

   FROM nsjail-builder AS nsjail-ipc-test
   ENV SANDBOX_PIPE_ADDON=/usr/local/lib/anonymous-pipes.node

   So the pipe machinery is exercised in an environment configured differently from the one released. The test passes; the artifact fails.

   5. Scope

   Introduced by a62d69e — "🔒 Harden Runner Supervisors with Anonymous Pipes" (#315). git tag --contains a62d69e returns v1.10.4 and nothing else, and v1.10.4 is exactly v1.10.3 plus that one commit. v1.10.3 and earlier are unaffected.

   Both Compose targets are built from the affected chain, since docker-compose.yaml selects target: sandbox-runner-${KVM_ENABLED:-true}.

   The published images are built from those same targets (.github/workflows/images.yml):

   code-interpreter-sandbox-runner          target: sandbox-runner-true
   code-interpreter-sandbox-runner-direct   target: sandbox-runner-false
   

   so the v1.10.4 release artifacts appear to carry the defect as well, not only source builds.

   Verified empirically on sandbox-runner-true (KVM) only. We have not run sandbox-runner-false; that it is equally affected is inferred from the Dockerfile stage graph, since it shares sandbox-runner-base and the same ENV gap. Worth confirming on your side before deciding the fix.

   6. Suggested fixes

   Any one of these closes it:

   1. Set the variable in the runner stages. Add to sandbox-runner-base, or to both sandbox-runner-baked and sandbox-runner:

      ENV SANDBOX_PIPE_ADDON=/usr/local/lib/anonymous-pipes.node

      The file is already in place via the existing COPY.

   2. Or place the addon where the fallback already looks, so the code path works with no environment at all:

      COPY --from=nsjail-builder /usr/local/lib/anonymous-pipes.node /sandbox_api/.build/anonymous-pipes.node

      Arguably the more robust option — it removes the dependency on an env var crossing an image and a VM boundary.

   3. Add SANDBOX_PIPE_ADDON to ALLOW_EXACT in launcher/src/main.rs. Needed only if you want operators to be able to override the path; not sufficient alone, since nothing sets it by default.

   Worth considering alongside

   Fail loudly at startup rather than per-execution. The runner answers its healthcheck happily while being structurally incapable of running anything. A startup probe that resolves the addon once — the same require the execution path performs — would turn a silent 500-per-request into an obvious boot failure.

   Run the IPC test against a shipped stage. nsjail-ipc-test setting SANDBOX_PIPE_ADDON itself is what allowed this through; a smoke test that executes one trivial job against sandbox-runner-true would have caught it immediately.

   7. Environment

   • code-interpreter v1.10.4, built from source via docker-compose.yaml, KVM_ENABLED=true (target sandbox-runner-true)
   • Ubuntu 24.04 on Azure, nested virtualisation, /dev/kvm present, libkrun microVM + NsJail
   • Python 3.14.4 runtime; every language job fails identically
   • v1.10.3 on the same host, same configuration, works correctly — the only delta is a62d69e

   👍React with 👍1Reacted by Frédéric (Freddy) M.
2. twltwk commented on Oct 12, 2026

   

   twltwk

   on Oct 12, 2026

   Author

   More actions

   Oh sorry !