Symptom
On Ubuntu 22.04 — native or under WSL2 — anything that starts the lemonade engine fails.
rocm serve <model> --engine lemonade exits non-zero after preparing the embeddable:
Preparing Lemonade embeddable 11.5.1...
Detecting best supported Lemonade llama.cpp backend...
Error: request_failed: Lemonade server did not become ready: exit status: 1;
.../engines/lemonade/runtime/lemond: /lib/x86_64-linux-gnu/libc.so.6:
version `GLIBC_2.38' not found (required by .../engines/lemonade/runtime/lemond)
There is no workaround available to the user: no engine flag, no version pin, and no
supported way to get glibc 2.38 onto 22.04.
Root cause
The lemond binary in every published lemonade-embeddable-*-ubuntu-x64 release is
linked against symbol versions newer than 22.04 provides. Ubuntu 22.04 ships glibc 2.35
and GLIBCXX_3.4.30; the binary needs GLIBC_2.38 and GLIBCXX_3.4.32. Both are
missing, so the dynamic loader refuses to start it.
The specific imports, from readelf --dyn-syms on 11.5.1:
__isoc23_sscanf@GLIBC_2.38 __isoc23_strtoul@GLIBC_2.38
__isoc23_strtol@GLIBC_2.38 __isoc23_strtoull@GLIBC_2.38
__isoc23_strtoll@GLIBC_2.38 _ZSt21ios_base_library_initv@GLIBCXX_3.4.32
#247 first found this while chasing a CI failure and checked the 11.x releases. Extending
that check to every release that publishes an ubuntu-x64 embeddable — v10.2.0
through v11.5.2, 15 releases — gives the same answer in all of them:
| release range |
max GLIBC |
max GLIBCXX |
| v10.2.0 … v10.10.0 (10 releases) |
GLIBC_2.38 |
GLIBCXX_3.4.32 |
| v11.0.0, v11.5.0, v11.5.1 (current pin), v11.5.2 |
GLIBC_2.38 |
GLIBCXX_3.4.32 |
Verified by downloading each lemonade-embeddable-<v>-ubuntu-x64.tar.gz from the
lemonade-sdk/lemonade releases and reading the version requirements out of lemond.
So this is not a regression in the current pin, and there is no older release to pin
back to — no published Lemonade embeddable can start on Ubuntu 22.04.
Why this is a product problem, not a CI problem
README.md states:
| Linux (x86_64) | Yes | Full support, including the live dashboard and both inference engines |
| WSL2 (x86_64) | Yes (Linux binary) | Full support, including the live dashboard; see docs/wsl.md for setup |
and docs/wsl.md lists as a prerequisite:
- WSL2 with Ubuntu 24.04 or Ubuntu 22.04.
Taken together, the docs promise a 22.04 user — native or WSL2 — full support for both
inference engines. One of those two engines cannot start on that release, and the WSL page
names 22.04 explicitly as a supported base. Every such user hits this the first time they
touch lemonade. CI is simply where it was noticed.
(Quotes checked against main at 4eebf1d.)
Options
Listing these for maintainers to choose between; I am not proposing one here.
- Narrow the documented support matrix — mark the lemonade engine as requiring
glibc >= 2.38 (Ubuntu 24.04+), and adjust docs/wsl.md accordingly. Honest and
immediate, but drops a currently-advertised platform.
- Ask upstream for a lower-baseline build — a
manylinux-style or built-on-22.04
ubuntu-x64 embeddable from lemonade-sdk/lemonade. The right fix if 22.04 is meant
to stay supported, but it is not in this project's control and needs an upstream
request.
- Ship the engine differently on old hosts — e.g. run
lemond from a newer-userland
container with /dev/kfd and /dev/dri passthrough. Heavier, and needs product
support for containerized engines.
- Some combination — e.g. do 1 now as the accurate state of the world, and pursue 2
as the durable fix.
Whichever route is taken, a clear preflight error is worth having on its own: today
the user sees a raw loader message from a subprocess. Detecting glibc < 2.38 before the
download and saying "the lemonade engine requires glibc 2.38 or newer; this host has
2.35" would turn an opaque failure into an actionable one. This is close in spirit to
#237, which is about the same class of problem for vLLM (unsupported interpreter/glibc
producing misleading advice).
Not viable
Installing glibc 2.38 on 22.04. Ubuntu publishes none for jammy (2.38 first appears in
23.10), and swapping the system libc for a newer release's breaks the ABI for every
binary on the machine. #247 has the longer write-up.
Environment
- lemonade pinned at
11.5.1 in runtime-deps.toml; artifact
lemonade-embeddable-11.5.1-ubuntu-x64.tar.gz
- Affects any x86_64 Linux host with glibc < 2.38, native or WSL2
- Ubuntu 24.04 (glibc 2.39,
GLIBCXX_3.4.33) satisfies both requirements
Symptom
On Ubuntu 22.04 — native or under WSL2 — anything that starts the lemonade engine fails.
rocm serve <model> --engine lemonadeexits non-zero after preparing the embeddable:There is no workaround available to the user: no engine flag, no version pin, and no
supported way to get glibc 2.38 onto 22.04.
Root cause
The
lemondbinary in every publishedlemonade-embeddable-*-ubuntu-x64release islinked against symbol versions newer than 22.04 provides. Ubuntu 22.04 ships glibc 2.35
and
GLIBCXX_3.4.30; the binary needsGLIBC_2.38andGLIBCXX_3.4.32. Both aremissing, so the dynamic loader refuses to start it.
The specific imports, from
readelf --dyn-symson 11.5.1:#247 first found this while chasing a CI failure and checked the 11.x releases. Extending
that check to every release that publishes an
ubuntu-x64embeddable — v10.2.0through v11.5.2, 15 releases — gives the same answer in all of them:
GLIBC_2.38GLIBCXX_3.4.32GLIBC_2.38GLIBCXX_3.4.32Verified by downloading each
lemonade-embeddable-<v>-ubuntu-x64.tar.gzfrom thelemonade-sdk/lemonadereleases and reading the version requirements out oflemond.So this is not a regression in the current pin, and there is no older release to pin
back to — no published Lemonade embeddable can start on Ubuntu 22.04.
Why this is a product problem, not a CI problem
README.mdstates:and
docs/wsl.mdlists as a prerequisite:Taken together, the docs promise a 22.04 user — native or WSL2 — full support for both
inference engines. One of those two engines cannot start on that release, and the WSL page
names 22.04 explicitly as a supported base. Every such user hits this the first time they
touch lemonade. CI is simply where it was noticed.
(Quotes checked against
mainat 4eebf1d.)Options
Listing these for maintainers to choose between; I am not proposing one here.
glibc >= 2.38 (Ubuntu 24.04+), and adjust
docs/wsl.mdaccordingly. Honest andimmediate, but drops a currently-advertised platform.
manylinux-style or built-on-22.04ubuntu-x64embeddable fromlemonade-sdk/lemonade. The right fix if 22.04 is meantto stay supported, but it is not in this project's control and needs an upstream
request.
lemondfrom a newer-userlandcontainer with
/dev/kfdand/dev/dripassthrough. Heavier, and needs productsupport for containerized engines.
as the durable fix.
Whichever route is taken, a clear preflight error is worth having on its own: today
the user sees a raw loader message from a subprocess. Detecting glibc < 2.38 before the
download and saying "the lemonade engine requires glibc 2.38 or newer; this host has
2.35" would turn an opaque failure into an actionable one. This is close in spirit to
#237, which is about the same class of problem for vLLM (unsupported interpreter/glibc
producing misleading advice).
Not viable
Installing glibc 2.38 on 22.04. Ubuntu publishes none for jammy (2.38 first appears in
23.10), and swapping the system libc for a newer release's breaks the ABI for every
binary on the machine. #247 has the longer write-up.
Environment
11.5.1inruntime-deps.toml; artifactlemonade-embeddable-11.5.1-ubuntu-x64.tar.gzGLIBCXX_3.4.33) satisfies both requirements