Skip to content

Lemonade engine cannot start on Ubuntu 22.04 (glibc 2.35), but README and docs/wsl.md advertise it as supported #258

Description

@rominf

Symptom

On Ubuntu 22.04 — native or under WSL2 — anything that starts the lemonade engine fails.
rocm serve <model> --engine lemonade exits non-zero after preparing the embeddable:

Preparing Lemonade embeddable 11.5.1...
Detecting best supported Lemonade llama.cpp backend...
Error: request_failed: Lemonade server did not become ready: exit status: 1;
.../engines/lemonade/runtime/lemond: /lib/x86_64-linux-gnu/libc.so.6:
version `GLIBC_2.38' not found (required by .../engines/lemonade/runtime/lemond)

There is no workaround available to the user: no engine flag, no version pin, and no
supported way to get glibc 2.38 onto 22.04.

Root cause

The lemond binary in every published lemonade-embeddable-*-ubuntu-x64 release is
linked against symbol versions newer than 22.04 provides. Ubuntu 22.04 ships glibc 2.35
and GLIBCXX_3.4.30; the binary needs GLIBC_2.38 and GLIBCXX_3.4.32. Both are
missing, so the dynamic loader refuses to start it.

The specific imports, from readelf --dyn-syms on 11.5.1:

__isoc23_sscanf@GLIBC_2.38    __isoc23_strtoul@GLIBC_2.38
__isoc23_strtol@GLIBC_2.38    __isoc23_strtoull@GLIBC_2.38
__isoc23_strtoll@GLIBC_2.38   _ZSt21ios_base_library_initv@GLIBCXX_3.4.32

#247 first found this while chasing a CI failure and checked the 11.x releases. Extending
that check to every release that publishes an ubuntu-x64 embeddable — v10.2.0
through v11.5.2, 15 releases — gives the same answer in all of them:

release range max GLIBC max GLIBCXX
v10.2.0 … v10.10.0 (10 releases) GLIBC_2.38 GLIBCXX_3.4.32
v11.0.0, v11.5.0, v11.5.1 (current pin), v11.5.2 GLIBC_2.38 GLIBCXX_3.4.32

Verified by downloading each lemonade-embeddable-<v>-ubuntu-x64.tar.gz from the
lemonade-sdk/lemonade releases and reading the version requirements out of lemond.
So this is not a regression in the current pin, and there is no older release to pin
back to
— no published Lemonade embeddable can start on Ubuntu 22.04.

Why this is a product problem, not a CI problem

README.md states:

| Linux (x86_64) | Yes | Full support, including the live dashboard and both inference engines |

| WSL2 (x86_64) | Yes (Linux binary) | Full support, including the live dashboard; see docs/wsl.md for setup |

and docs/wsl.md lists as a prerequisite:

  1. WSL2 with Ubuntu 24.04 or Ubuntu 22.04.

Taken together, the docs promise a 22.04 user — native or WSL2 — full support for both
inference engines. One of those two engines cannot start on that release, and the WSL page
names 22.04 explicitly as a supported base. Every such user hits this the first time they
touch lemonade. CI is simply where it was noticed.

(Quotes checked against main at 4eebf1d.)

Options

Listing these for maintainers to choose between; I am not proposing one here.

  1. Narrow the documented support matrix — mark the lemonade engine as requiring
    glibc >= 2.38 (Ubuntu 24.04+), and adjust docs/wsl.md accordingly. Honest and
    immediate, but drops a currently-advertised platform.
  2. Ask upstream for a lower-baseline build — a manylinux-style or built-on-22.04
    ubuntu-x64 embeddable from lemonade-sdk/lemonade. The right fix if 22.04 is meant
    to stay supported, but it is not in this project's control and needs an upstream
    request.
  3. Ship the engine differently on old hosts — e.g. run lemond from a newer-userland
    container with /dev/kfd and /dev/dri passthrough. Heavier, and needs product
    support for containerized engines.
  4. Some combination — e.g. do 1 now as the accurate state of the world, and pursue 2
    as the durable fix.

Whichever route is taken, a clear preflight error is worth having on its own: today
the user sees a raw loader message from a subprocess. Detecting glibc < 2.38 before the
download and saying "the lemonade engine requires glibc 2.38 or newer; this host has
2.35" would turn an opaque failure into an actionable one. This is close in spirit to
#237, which is about the same class of problem for vLLM (unsupported interpreter/glibc
producing misleading advice).

Not viable

Installing glibc 2.38 on 22.04. Ubuntu publishes none for jammy (2.38 first appears in
23.10), and swapping the system libc for a newer release's breaks the ABI for every
binary on the machine. #247 has the longer write-up.

Environment

  • lemonade pinned at 11.5.1 in runtime-deps.toml; artifact
    lemonade-embeddable-11.5.1-ubuntu-x64.tar.gz
  • Affects any x86_64 Linux host with glibc < 2.38, native or WSL2
  • Ubuntu 24.04 (glibc 2.39, GLIBCXX_3.4.33) satisfies both requirements

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions