Summary
Switching between native and Pulley builds can embed the wrong hyperlight-wasm-runtime guest. A native host build can retain the Pulley runtime and reject a valid x86 AOT component:
GuestError(GuestError, "Module was compiled for architecture 'x86_64'")
Reproduction
Use a consumer feature that enables hyperlight-wasm/pulley.
- Build and run with an x86 AOT component.
- Build and run with the Pulley feature and a Pulley AOT component.
- Build and run with the x86 AOT component again.
The third run can reject the x86 component. This reproduces with hyperlight-wasm 0.15.0 and Wasmtime 36.0.14.
Root cause
hyperlight-wasm/build.rs builds the embedded runtime into a shared nested target directory:
<target>/hyperlight-wasm-runtime/x86_64-hyperlight-none/<profile>/hyperlight-wasm-runtime
Native and Pulley Cargo units have separate OUT_DIR directories. Both generated wasm_runtime_resource.rs files use include_bytes! with this shared mutable path.
The nested Pulley build overwrites the runtime binary. Cargo can rebuild the native hyperlight-wasm unit because the include_bytes! input changed while its build script remains cached. The native unit then embeds the Pulley binary.
The shared output also permits a race between concurrent normal and Pulley builds.
Suggested fix
Copy the completed nested runtime into the outer build script's OUT_DIR. Point include_bytes! at that private copy. This preserves the shared nested compilation cache.
Concurrent builds also need one of these protections:
- Hold a lock across the nested build and copy.
- Use a separate nested target directory for each runtime configuration.
The configuration key must cover pulley, gdb, trace_guest, wasmtime_latest, and the WIT environment values. The outer OUT_DIR provides complete isolation but duplicates compilation work.
Regression coverage
Build a consumer in both sequences. Verify that the final component loads:
normal -> pulley -> normal
pulley -> normal -> pulley
Concurrent native and Pulley builds should produce correctly matched embedded runtimes.
Summary
Switching between native and Pulley builds can embed the wrong
hyperlight-wasm-runtimeguest. A native host build can retain the Pulley runtime and reject a valid x86 AOT component:Reproduction
Use a consumer feature that enables
hyperlight-wasm/pulley.The third run can reject the x86 component. This reproduces with
hyperlight-wasm 0.15.0and Wasmtime36.0.14.Root cause
hyperlight-wasm/build.rsbuilds the embedded runtime into a shared nested target directory:Native and Pulley Cargo units have separate
OUT_DIRdirectories. Both generatedwasm_runtime_resource.rsfiles useinclude_bytes!with this shared mutable path.The nested Pulley build overwrites the runtime binary. Cargo can rebuild the native
hyperlight-wasmunit because theinclude_bytes!input changed while its build script remains cached. The native unit then embeds the Pulley binary.The shared output also permits a race between concurrent normal and Pulley builds.
Suggested fix
Copy the completed nested runtime into the outer build script's
OUT_DIR. Pointinclude_bytes!at that private copy. This preserves the shared nested compilation cache.Concurrent builds also need one of these protections:
The configuration key must cover
pulley,gdb,trace_guest,wasmtime_latest, and the WIT environment values. The outerOUT_DIRprovides complete isolation but duplicates compilation work.Regression coverage
Build a consumer in both sequences. Verify that the final component loads:
Concurrent native and Pulley builds should produce correctly matched embedded runtimes.