Conversation
Adds pkg/tokens/ wrapping pkoukk/tiktoken-go with three exported functions (Count, CountText, EncodingFor) and a small alias table. Promotes tiktoken-go from indirect to direct dependency. Counts are exact for OpenAI families (cl100k_base, o200k_base) and approximate (within ~5%) for other model families — accurate enough for compression-trigger thresholds and cost prediction, not for billing. Implements RFC mudler#9537. Does not change behavior of any existing endpoint; purely additive utility code under MIT. References: mudler#9537 Assisted-by: Claude:claude-opus-4-7
Foundation for the upcoming context-compression middleware (RFC mudler#9534). This commit adds two pieces: 1. CompressionConfig in core/config/model_config.go — per-model YAML schema mirroring FunctionsConfig / MCPConfig style. Six fields: enabled, trigger_at_ratio, keep_tail_tokens, max_summary_tokens, compressor_model, on_post_compression_overflow. Opt-in, default off, absent block changes nothing. 2. pkg/compress/ — algorithmic core, decoupled from HTTP layer: - Compress entrypoint with Summarizer interface - partition() splits messages into a head (compress) and tail (keep) by token count, preserving keep_tail_tokens - drop-oldest-summary overflow recovery (default) and error mode - 14 Ginkgo specs covering passthrough, compression, model fallback, summarizer error propagation, and overflow recovery The middleware that wires this into /v1/chat/completions follows in a subsequent commit. References: mudler#9534 Assisted-by: Claude:claude-opus-4-7
Adds the HTTP-layer glue for the compression feature introduced in the previous commit: * core/http/middleware/compression.go — Echo middleware that runs after SetOpenAIRequest, reads the parsed *schema.OpenAIRequest and *config.ModelConfig from context, and calls pkg/compress.Compress with a backend-backed Summarizer when CompressionConfig.Enabled is set. Compression failures fall through to passthrough rather than aborting the request — best-effort optimisation must never break the underlying chat completion. * CompressionMiddlewareWithSummarizer is exported as a test seam so Ginkgo specs can inject a stub Summarizer without standing up an Application. * The Summarizer implementation invokes the configured compressor model via core/backend.ModelInferenceFunc with a fixed compression prompt; empty predInput relies on the model's chat template (operators must set use_jinja: true on the compressor model). * Registers the middleware on /v1/chat/completions (routes/openai.go) and /v1/mcp/chat/completions (routes/localai.go). * 10 Ginkgo specs cover passthrough, compression-applied path, summarizer-error fallthrough, and default-injection behaviour. Sensible defaults applied when only `enabled: true` is set: trigger_at_ratio=0.75, keep_tail_tokens=8000, max_summary_tokens=2048, on_post_compression_overflow=drop_oldest_summary. Compression result is attached to the Echo context via CONTEXT_LOCALS_KEY_COMPRESSION_RESULT so a future commit can surface usage.compression_meta in the response body. References: mudler#9534 Assisted-by: Claude:claude-opus-4-7
User-facing documentation for the optional context-compression feature introduced in the previous two commits. Covers: - when to enable - full YAML field reference with defaults - compressor-model requirements (use_jinja: true) - how the middleware partitions and summarises - best-effort failure semantics - backward compatibility (additive, opt-in) - known limitations (approximate counts, single-pass, no cross-request memory) - troubleshooting common scenarios References: mudler#9534 Assisted-by: Claude:claude-opus-4-7
Operator-level kill-switch that disables the context-compression middleware globally, overriding any per-model `compression.enabled: true` settings without requiring YAML edits. Mirrors the pattern of LOCALAI_DISABLE_AGENTS / LOCALAI_DISABLE_MCP. When set, the middleware is replaced with a passthrough at chain construction time, so the kill-switch has zero per-request overhead. References: mudler#9534 Assisted-by: Claude:claude-opus-4-7
…on-middleware Conflicts (resolved additively — both sides are independent feature adds, neither overrides the other): - core/config/model_config.go: keep our Compression field + type AND upstream's new PII/Router/Proxy/MITM/Limits fields and types. - core/http/routes/openai.go (chat middleware chain): keep upstream's RouteModel → AdmissionControl → PII redaction chain, with our CompressionMiddleware appended as the innermost step. Order rationale: RouteModel resolves the served model first, AdmissionControl gates on the served model's limits, PII redaction runs before any LLM (including the compression summariser) sees content, Compression operates last on the PII-redacted request using the served model's CompressionConfig.
# Conflicts: # core/config/model_config.go # core/http/routes/openai.go
The fork sat on llama.cpp 3466812d while the llama-cpp backend builds 9cffdcc. That is 22 commits apart, and it made every cross-backend comparison ambiguous: an agent-path measurement of an FP4 model against a normal one was measuring two llama.cpp vintages as much as two quantizations. The eight fork commits cherry-pick onto 9cffdcc without a single conflict, and the only MMQ-area movement in those 22 commits is 16 deleted lines (mmq-vec-dot.cuh -11, mmq.cuh -5) — nothing the FP4 kernels had to be re-expressed against. Branch walcz-de/llama.cpp-ROCmFP4:rocmfp4-rebase-9cffdcc, head 32284d40b7ddac920b06c4587e50c01a3bf38ac1. The rocmfp4 branch and the pin the open PR carries are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PHcZ8CTpZEoK3qgroNnkB4
…ate system turns Tokenizer chat templates such as Qwen3.8 / Qwen3.8-Flash-Next raise 'System message must be at the beginning' for system-role messages that appear after the leading system block, while agent frameworks (cogito tool selection and adjustment prompts) legitimately append system instructions mid-conversation. Every such request failed with a 500 (48 errors in one 10-task agent run). New per-model option template.system_messages_after_first: merge fold late system turns into the leading system message user forward them as user-role turns at their original position Default (unset) keeps the current pass-through behaviour. Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
…Finish idempotent (double finish panicked the agent) Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
…ot/cache reuse after parallel burst on gfx1151 with 67672dc5 (nonce probe red 3/5); revert patch c7d87229 alone no longer sufficient; upstream report pending Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
github.com/mudler/localrecall v0.6.3 destroys the hybrid-search ranking: the RRF fusion divides the weight parameters by a bigint rank, so PostgreSQL infers them as bigint and the division is integer division — every weight below 61 yields a score of exactly 0. All candidates tie, ORDER BY similarity DESC orders by nothing, and the collection search endpoint returns the join order instead of a ranking. HYBRID_SEARCH_BM25_WEIGHT and HYBRID_SEARCH_VECTOR_WEIGHT therefore never had any effect. Measured on the live tax-law collection (650 chunks, four questions with a verifiable answer), correct document in the top three: 2/4 before, 4/4 after. A verbatim excerpt of a stored document did not retrieve its own chunk at rank 1, while a direct pgvector KNN on the same query was correct — embeddings and index were healthy all along. Replace points at walcz-de/LocalRecall branch fix/rrf-integer-division (also carries the BM25 text_config knob from 2026-07 and the deferred search_vector follow-up). Upstream PR against mudler/localrecall still to be opened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # backend/cpp/llama-cpp/Makefile
# Conflicts: # Makefile # docs/content/features/backends.md
…zt verlangt Der Upstream-Merge brachte LocalAI-Code, der eine neuere LocalAGI-Schnittstelle ruft (state.PoolLimits, conversations.RetentionPolicy, scheduler.RetentionPolicy/CreationPolicy, state.SkillsProvider). Unsere `replace`-Ersetzung hielt den Fork-Stand vom 2026-09-08 fest, der diese Typen nicht kennt — `make build` brach in core/services/agentpool ab. Das ist die in CLAUDE.md beschriebene Kopplung, nur andersherum als dort notiert: nicht wir bumpen eigenmaechtig, sondern upstream zieht seinen Pin (jetzt d93d478e42f1) und unser Override bleibt zurueck. Abstand waren 20 Commits. Unsere sechs Commits sind konfliktfrei auf d93d478e42f1 aufgesetzt und leben unveraendert weiter: Selbstkorrektur bei unbekannten Werkzeugen, Grounding-Tor (zwei Stufen), Reihenfolge im Nachrichtenstrom, Idempotenz von JobResult.Finish, 10-Minuten-Rueckfall bei unlesbarer Laufzeit. Neuer Fork-Zweig fix/walcz-patches-on-d93d478 (716a840) statt Force-Update des alten — so muss die neue Push-Sperre nicht uebergangen werden, und die alte Linie bleibt als Beleg stehen. Geprueft: core/ des Forks uebersetzt, core/types gruen, core/agent laeuft in die Frist — auf dem REINEN upstream-Stand d93d478 ebenso, also vorbestehend (die Suite braucht eine laufende LocalAI). core/services/agentpool von LocalAI uebersetzt mit der neuen Ersetzung. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Paradigma (Stefan 2026-09-12): Upstream ist der Standard, eine abweichende Pin nur wenn upstream BEWIESEN kaputt ist. Fuer 465e49b9 gab es diesen Beweis nie — der eigene Bisect-Plan weist ihn als offen aus: die schlechte Evidenz war EIN warmer Prozess, der 3 Proben in 5 Minuten fehlschlug, waehrend dasselbe Binaer davor 7x sauber war, und die in §1 verlangte Baseline wurde nie gefahren. Ohne Beweis geht der Pin nach oben: 465e49b9 -> df03399b. Damit fallen zwei Patches ERSATZLOS weg, weil upstream sie selbst uebernommen hat: · backend/cpp/llama-cpp/patches/0001-revert-c7d87229-hip-integrated-crossover.patch · backend/cpp/rocmfp4/patches/0001-revert-c7d87229-hip-integrated-crossover.patch Upstream d4389a4dd revertiert c7d87229 selbst (#28604), enthalten ab df03399b. Dazu kommt #28326 (iGPU-Lazy-Loading aus) und #28302 (Checkpoint-Verdraengung) — dieselbe Fehlerklasse, oben behandelt. Die uebrigen zwei llama-cpp-Patches (score, tts) wurden am neuen Pin geprueft und werden weiter gebraucht — in der Reihenfolge, in der prepare.sh sie faehrt: tts setzt auf score auf und meldet einzeln geprueft faelschlich "passt nicht". Genau dieser Messfehler steckte auch im Tor und ist behoben; ebenso, dass es Patches fremder Backends gegen den llama.cpp-Pin geprueft hat. Register: von 10 Abweichungen auf 7. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backend upgrade check gates on `downloader.URI(...).LooksLikeOCI()` and then passes the
gallery URI unchanged to `oci.GetImageDigest`, which calls `name.ParseReference`. That parser
does not accept an `oci://` scheme, so the lookup fails with "could not parse reference" and
the upgrade is skipped — visible only as a WARN at startup.
LooksLikeOCI() recognises an image either by a fixed host list (quay.io, ghcr.io, docker.io)
or by the explicit oci:// scheme. Users pulling backends from those three hosts write no
scheme and never hit this. Users running their own registry MUST set the scheme to be
recognised at all — and therefore lose backend upgrade detection entirely, silently.
Observed on a self-hosted gallery, every start, for all four locally built backends:
WARN Failed to get remote OCI digest for upgrade check backend="..."
error=could not parse reference: oci://registry.example:5556/localai-backends:...-vllm
while gallery backends from quay.io report upgrades normally in the same run.
Dropping the scheme from the gallery entry is not a workaround: without it LooksLikeOCI()
no longer treats the registry as an image source and the backends cannot be installed.
Stripping happens at the call site because pkg/downloader imports pkg/oci — the constant
cannot be referenced from inside GetImageDigest without an import cycle. This matches the
two existing call sites that strip the prefix themselves (pkg/downloader/uri.go,
core/gallery/importers/llama-cpp.go).
Verified: `go build ./core/gallery/...` and `go test ./core/gallery/` pass.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
ROCMFP4_VERSION 32284d40 -> 5f91425d. Der Fork stand 216 Commits und 13 Tage hinter
LLAMA_VERSION; das Wrapper-Makefile loescht vor dem Bauen die fuer upstream-llama.cpp
vendorierten Patches und setzt damit voraus, dass der Fork auf genau diesem Pin sitzt.
Der Rebase war kein Rebase, sondern ein Port: upstream 5a4d0fecae hat den
FlashAttention-Vec-Dispatch ersetzt. Statt handgepflegter Instanzliste plus Makro-Kaskade
gibt es jetzt einen CMake-Generator (ggml_cuda_fattn_vec_instances) und eine Registry
(ggml_cuda_get_fattn_vec_case), die ueber generierte GGML_CUDA_FA_<K>_<V>-Makros
entscheidet, was kompiliert wird.
Unsere FP4-Typen sind deshalb dorthin gewandert, wo upstream sie erwartet:
ggml/cmake/common.cmake FA_TYPES += q4_0_rocmfp4 q4_0_rocmfp4_fast
q3_0_rocmfpx q6_0_rocmfpx q8_0_rocmfpx
ggml/CMakeLists.txt FA_QUANTS-Vorgabe += die fuenf symmetrischen Paare
ggml/src/ggml-cuda/fattn.cu Registry-Zeilen + kv_type_supported
Zwei Dinge sind dabei WENIGER geworden, und das ist der Punkt der Uebung:
· "register the FA instances with the HIP backend's own source list" ist ERSATZLOS
entfallen — upstream ruft den Generator jetzt selbst in ggml-hip/CMakeLists.txt auf
und liest dieselbe Auswahl. Sieben Commits statt acht.
· Die TurboQuant-Paare habe ich beim Portieren zuerst mitgenommen und dann wieder
entfernt: der naechste Commit der Serie verwirft sie ausdruecklich (Linkfehler,
"dieses Backend bedient ROCmFP4-GEWICHTE"). Sie stehen jetzt weder in FA_TYPES noch
in der Auswahl noch in der Registry; die TYPEN bleiben deklariert, damit die
ggml-Typnummern zu veroeffentlichten GGUFs passen.
Gegengeprueft: jede Kombination der FA_QUANTS-Vorgabe hat ihre Instanzdatei (11/11) —
sonst bricht der neue Generator hart ab. GGML_CUDA_FA_QUANTS="all" ist auf diesem Fork
NICHT benutzbar (196 Kombinationen, nur symmetrische Instanzen); das ist im common.cmake
vermerkt.
Fork-Zweig: walcz-de/llama.cpp-ROCmFP4 rocmfp4-rebase-38a5b42d
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1HR5SwH4eT33fELQ2yvHG
…pgrade record a digest mudler#12138 stripped the oci:// scheme in the upgrade check, but the two paths that record the digest after an install (core/gallery/backends.go) and after an upgrade (core/gallery/upgrade.go) still hand the raw gallery URI to GetImageDigest, which fails with "could not parse reference: oci://…". No digest is ever stored for a self-hosted gallery, so the check compares the registry digest against nothing and reports "upgrade available (new build)" forever; every click re-installs the same image and the flag comes back. Normalise once inside GetImageDigest so every caller behaves the same, and cover it with a unit test that also asserts the raw oci:// form is unparseable. Observed on LocalAI v4.10.0-263 with an oci:// gallery on a private registry: five backends flagged permanently, log "Failed to get OCI image digest after upgrade … could not parse reference: oci://registry0…:5556/…". Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> Assisted-by: Claude:claude-fable-5-1
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #12138. That change strips the
oci://scheme in the upgrade check, but the two places that record a digest still hand the raw gallery URI tooci.GetImageDigest:core/gallery/backends.go— after an installcore/gallery/upgrade.go— after an upgradename.ParseReferencecannot parse the scheme, so the call fails withcould not parse reference: oci://…and no digest is ever written tometadata.json. For a self-hosted gallery (which must use theoci://scheme to be recognised as OCI) the check then compares the registry digest against nothing and reports "upgrade available (new build)" forever; every click re-installs the same image and the flag comes straight back.This PR normalises the reference once inside
GetImageDigest, so install, upgrade and check all behave the same, and adds a unit test that also asserts the rawoci://form is unparseable (otherwise the normalisation would be moot).Observed on LocalAI
v4.10.0-263with a private gallery pointing atoci://registry:5556/…: five backends flagged permanently, log lineFailed to get OCI image digest after upgrade … could not parse reference: oci://….Notes for Reviewers
TrimPrefixis a no-op there).go test ./pkg/oci/andgo vet ./pkg/oci/; a fullmake buildof the tree runs in our nightly.Signed commits
🤖 Generated with Claude Code
https://claude.ai/code/session_01PHcZ8CTpZEoK3qgroNnkB4