Skip to content

Build can embed the wrong runtime after switching Pulley feature #545

Description

@ludfjig

Summary

Switching between native and Pulley builds can embed the wrong hyperlight-wasm-runtime guest. A native host build can retain the Pulley runtime and reject a valid x86 AOT component:

GuestError(GuestError, "Module was compiled for architecture 'x86_64'")

Reproduction

Use a consumer feature that enables hyperlight-wasm/pulley.

  1. Build and run with an x86 AOT component.
  2. Build and run with the Pulley feature and a Pulley AOT component.
  3. Build and run with the x86 AOT component again.

The third run can reject the x86 component. This reproduces with hyperlight-wasm 0.15.0 and Wasmtime 36.0.14.

Root cause

hyperlight-wasm/build.rs builds the embedded runtime into a shared nested target directory:

<target>/hyperlight-wasm-runtime/x86_64-hyperlight-none/<profile>/hyperlight-wasm-runtime

Native and Pulley Cargo units have separate OUT_DIR directories. Both generated wasm_runtime_resource.rs files use include_bytes! with this shared mutable path.

The nested Pulley build overwrites the runtime binary. Cargo can rebuild the native hyperlight-wasm unit because the include_bytes! input changed while its build script remains cached. The native unit then embeds the Pulley binary.

The shared output also permits a race between concurrent normal and Pulley builds.

Suggested fix

Copy the completed nested runtime into the outer build script's OUT_DIR. Point include_bytes! at that private copy. This preserves the shared nested compilation cache.

Concurrent builds also need one of these protections:

  • Hold a lock across the nested build and copy.
  • Use a separate nested target directory for each runtime configuration.

The configuration key must cover pulley, gdb, trace_guest, wasmtime_latest, and the WIT environment values. The outer OUT_DIR provides complete isolation but duplicates compilation work.

Regression coverage

Build a consumer in both sequences. Verify that the final component loads:

normal -> pulley -> normal
pulley -> normal -> pulley

Concurrent native and Pulley builds should produce correctly matched embedded runtimes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions