Repository navigation
Llamacpp: the b11642 layout of llama_context_params beside the older one, and exports in host memory on a CPU-only host - #17
Open
IsuraManchanayake wants to merge 19 commits into
Conversation
baseRT 0.3.1 defines BASERT_ABI_VERSION and exports baseRT_abi_version(): two libraries with the same number drive the same way, whatever their version numbers say. The build reads the macro from the vendored header (now 0.3.1's), the loader resolves the symbol when the library has it, and the numbers decide: equal serves, unequal is refused by number. A library that states none (0.3.0 and older), or a superfluid built against a header without the macro, is judged by major.minor as before. The symbol and configuration-struct checks stay. Library carries the number it found.
…s; superfluid compat; exit 4 A probe may state the contract number it checked the runtime against (Probe.abi: baseRT's engine ABI), the worker's check report carries it, and `superfluid runtimes --json` shows it as `abi` on a ready runtime. The basert row also carries `requires`, the header version and ABI number this build was built for, whether or not an engine was found, so a host can compare numbers itself. `superfluid compat [--json]` prints the same for a release. `superfluid serve` exits 4 when the runtime for the model is not installed or cannot run here (its engine library missing or of another ABI), where a flag it got wrong exits 2, so a host can tell the two apart. The catalog finds superfluid-workerd and the shipped adapters beside the real executable, links resolved: a superfluid reached through a symlink (a package manager's bin/, a host's shim) kept running the engine in-process because the link's directory held neither.
…at.json The page names the contract a host builds on: how to start superfluid (the real path, --runtime, --basert-lib and BASERT_KERNEL_LIB, --offline, one --sessions per server, SUPERFLUID_HOME), how to ask for the compatibility verdict before serving, how to choose a release from the compat.json each release now carries beside its tarball, and the exit codes. The CLI reference, the runtimes page and the changelog follow.
…he unnumbered line; the engine install chooses by ABI number The vendored header is baseRT 0.4.0's, so the fallback for a library that states no ABI number can no longer be "the header's minor": that would refuse 0.3.0, which carries ABI 1's functions and layouts without stating it. The line that does so is named instead, 0.3.x for ABI 1, and a library without a number serves only if it is of that line. No later ABI has such libraries. A library of 0.4.0 or later is asked for the model configuration by size (baseRT_get_config_into), into a zeroed struct of these bindings' own size, so the struct may grow or shrink between them without a change of ABI number; an older library is read by value with the size check as before. Every read of the configuration goes through config_of(). `superfluid runtime install basert` chooses by the same rule the loader applies: a release's ABI number, read from the basert-engine-abi-<N>.txt asset baseRT publishes beside its bundles, or a pin's, with the 0.3.x line accepted without one; another engine installs only with --untested. `superfluid compat` and the runtimes report name the unnumbered line too.
The pack step wrote dist/<name>.compat.json, but the artifact the release job downloads carried only the tarball and its digest, so the sidecar the docs promise beside the tarball never reached the release.
`superfluid compat` and the release's compat.json described basert alone. Now an adapter declares its part through Runtime::compat, its worker answers `compat --json`, and the daemon merges the answers of every adapter it ships: for basert the header version, the ABI number, the line accepted without one and the engines it installs on its own; for llama.cpp the upstream builds it was built and tested against; for MLX the versions it installs and the Python they run on. The llama.cpp and MLX adapters are not linked into the daemon, so asking their workers is what keeps the daemon free of them.
…without credentials The report inserted the linked basert adapter by name and then asked the shipped workers, so a source build with --features llamacpp and no standalone worker left llamacpp out. Every linked adapter now answers in process first. The shipped worker's compat is run through exec::without_secrets, as check, plan, install and serving are, so a token in the caller's environment does not reach it.
… install here `superfluid runtime versions <id> [--json]` lists the versions an adapter installs on its own, newest first: which install on this machine and as what, and which are installed. `superfluid runtimes` gains an INSTALLABLE column, the newest such version here, and `installable` in its JSON. The adapter kit answers through a new `Runtime::versions` method, by default each tested version `compat` names planned for the host, and a worker command `versions`; both from the recipe alone, since the table asks on every run. The basert adapter lists its pins only, with the short ABI verdict for one this build does not drive, because its `plan` consults the release listing when a pin fails.
…efore the linked adapter The listing went to the linked adapter whenever this build links one, so the worker override that runtime execution and runtime install honour was ignored, and the INSTALLABLE column could name versions the selected installer does not provide. A named worker is asked first.
…a.cpp's parameters An adapter declares the options its runtime takes on serve (Runtime::options), each with the shape of its value; the daemon checks a value against the declaration before a worker starts and forwards it as --<name> <value>, the way --pull-<name> already works. superfluid runtime options <id> lists them and superfluid runtimes --json carries them. llama.cpp gets its model and context parameters as --llamacpp-<name>: n-gpu-layers, main-gpu, split-mode, tensor-split, load-mode, lazy-mode, repack, threads, threads-batch, batch-size, ubatch-size, flash-attn, cache-type-k, cache-type-v, kv-offload, op-offload, swa-full, the RoPE and YaRN settings, and lib. MLX gets venv, prefill-chunk and cache-limit-mb. A knob every runtime could take stays a flag of serve.
…ced by its cache types, and the options reach every worker start Four findings of the review: - The state identity folds in the RoPE, YaRN and cache-type settings, so a store written under one configuration is refused under another instead of continuing from keys that mean something else. With none of them set the key is the weights alone, as before. - A KV cell is priced by the cache types chosen, by ggml's row sizes, in the descriptor and in the sizing before a load; f16 remains the default. - The options are set for the process as given, before the runtime is required, and ride in the worker's prefix from the first probe on, so a library or environment option counts for availability and sizing. When the probe fails and the worker answers without the options, the option at fault is named as before. - In-process workers get their arguments through the worker's own parser, so --venv lands in its field and the MLX environment option is honoured in a linked daemon and at a fleet node.
…on's own tokenizer sees the llama.cpp library Two more findings of the review: the worker's pull now takes the runtime's options apart from the pull's own arguments, and both pull paths pass them, so a pull by id runs in the environment the check passed and the serve will use. The llama.cpp tokenizer library the daemon loads in process finds libllama as the worker does, by SUPERFLUID_LLAMA_LIB, which the daemon sets from --llamacpp-lib before the first load.
…s beside the older one llama.cpp b11642 inserted size_t moe_cache_size into llama_context_params before embeddings, which moved every field after it; the adapter's check refused the build, rightly, since it would have driven it through the wrong fields. The adapter now knows both layouts: it tells them apart from the defaults the library returns, reads and hands back the struct in the library's own layout, and fills either with the same assignments. b11284 keeps working as before and b11642 joins the tested builds. Found by the nightly runtime job on a fork: #9
…as b11642's header lays it out The v2 struct had the field after abort_callback_data. The defaults could not tell: all three fields default to zero, and the adapter sets none of them, so no tested build misread. A caller setting the abort callback through the public struct would have written it into moe_cache_size. The layout test now pins every v2 offset to what offsetof gives on b11642's include/llama.h.
2 of 3 tasks
…n on the CPU does The export budget took the host-memory path only for n_gpu_layers 0 and otherwise read the offload device's free memory, which a host without one reports as 0; 0 granted nothing, so a CPU-only host kept no state beyond the pool. The nightly's Linux leg, the first run of the adapter's tests without an accelerator, failed two export tests on it for this. The decision is one function keyed on the device the loader resolved, with a unit test; the messages are unchanged.
IsuraManchanayake
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #15: its diff includes #10, #11, #14 and #15 until they merge; this PR's own commit is the last one. I will rebase it onto
rc-0.1.3as they land.What changed
llama.cpp b11642 inserted one field,
size_t moe_cache_size, intollama_context_paramsbeforeembeddings;llama_model_paramsdid not change. Every field after it moved by eight bytes, so the adapter's layout check sawoffload_kqvandsamplersat the wrong places and refused the build. The refusal was right: setting the context through the old layout would have written into the wrong fields.The adapter now keeps both layouts:
llama_context_params_v2is the struct with the new field;llama_context_default_paramsandllama_init_from_modelare bound a second time under that type (the function-pointer table takes an alias for a symbol now).ContextLayoutis told from the defaults the library returns, read at the offsets of each layout with the same expected values as before; the library remembers which layout it has. A build that fits neither is refused with the message as before.append).llama.cpp inserts fields like this every few weeks; a new layout is one more struct and one more arm, and a generation can be retired when no tested build uses it.
Second, with the model on the CPU because the host has no accelerator, the prefix cache keeps state beyond the pool in host memory within 8 GiB, as it already did with the model on the CPU by choice (
--llamacpp-n-gpu-layers 0). It kept nothing: the export budget took the host-memory path only forn_gpu_layers0 and otherwise read the free memory of an offload device the host does not have, which is 0, and 0 granted nothing. The decision is now one function,export_budget, keyed on the device the loader resolved; the messages are unchanged.Why
The fork's nightly job found b11642 refused: IsuraManchanayake#9. Without this the newest upstream builds stay uninstallable and the recipe's tested list stops at b11284. The second change is what the nightly's Linux leg found once b11643 loaded: a CPU-only runner, the first place these tests ever ran without an accelerator (CI's adapter job is macOS only), failed the two export tests for this and not for anything in the build (https://github.com/IsuraManchanayake/superfluid/actions/runs/38725440780). Reviewed in the fork as IsuraManchanayake#11 and #13.
How it was tested
cargo clippy --all-targets -- -D warnings: clean.cargo test(CI at this head on macos-14, ubuntu-24.04 and ubuntu-24.04-arm): the unit tests read a synthetic v1 image as V1, the same image with the inserted field as V2, an older image as neither, and a v1 image read as v2 misplaces exactly the fields b11642 showed.Against the real libraries, locally on macOS arm64: the layout test reports b11284 as V1 and b11642 as V2; the adapter's 23 real-model tests pass on both with Qwen3-0.6B-Q4_K_M; the worker's
checkon b11642 reads the model.Not run: a full
superfluid serveon b11642, and the CUDA or Vulkan builds of it; the worker'scheckand the adapter's tests on the macOS build are the proof.The review on the fork (llamacpp: the adapter drives the b11642 layout of llama_context_params beside the older one IsuraManchanayake/superfluid#11) found
moe_cache_sizedeclared after the abort callback pair, which the zero defaults masked; fixed in 6eadeb1, the v2 offsets pinned to the header's, and the runs above repeated on both builds.The export budget: a unit test on
export_budget(no device grants the pool's cells within 8 GiB; a device with no free memory grants nothing and names the headroom). End to end against a CPU-only llama.cpp b11643 built from source without Metal: before the fix the real-model suite fails the two export tests withup to 0 tokens keptlogged, after it all 23 pass; the fork's nightly Linux leg passes with it (https://github.com/IsuraManchanayake/superfluid/actions/runs/38751331332). Unchanged on Metal: the adapter's full tests on b11284 and b11642 with the fix in place.cargo clippy --all-targets -- -D warningscargo testChanges a wire protocol, the WAL format or the tick ABI (says so above, version bumped)