Can't compute multiple embeddings in a single call

[jeberger]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Running this code:

model = llama_cpp.Llama ("mxbai-embed-xsmall-v1-q8_0.gguf", embedding = True)
embeddings = model.embed (["Hello", "World"])

used to work in v0.3.14

Current Behavior

The code raises an exception RuntimeError: llama_decode returned -1. The following messages are printed to the console:

init: invalid seq_id[3][0] = 1 >= 1
encode: failed to initialize batch

Environment and Context

llama-cpp-python was compiled in CUDA mode

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import llama_cpp
>>> model = llama_cpp.Llama ("../models/mxbai-embed-xsmall-v1-q8_0.gguf", embedding = True)
...
>>> embeddings = model.embed (["Hello", "World"])
decode: cannot decode batches with this context (calling encode() instead)
init: invalid seq_id[3][0] = 1 >= 1
encode: failed to initialize batch
llama_decode: failed to decode, ret = -1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../site-packages/llama_cpp/llama.py", line 1108, in embed
    decode_batch(s_batch)
  File ".../site-packages/llama_cpp/llama.py", line 1045, in decode_batch
    self._ctx.decode(self._batch)
  File ".../site-packages/llama_cpp/_internals.py", line 327, in decode
    raise RuntimeError(f"llama_decode returned {return_code}")
RuntimeError: llama_decode returned -1

[codingl2k1]

Same issue, I am using the CPU build: CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

[iamlemec]

Yep, this is due to recent(?) changes in how the KV cache works. Here's how the embedding example in llama.cpp handles it:

// if the number of prompts that would be encoded is known in advance, it's more efficient to specify the
//   --parallel argument accordingly. for convenience, if not specified, we fallback to unified KV cache
//   in order to support any number of prompts
if (params.n_parallel == 1) {
    LOG_INF("%s: n_parallel == 1 -> unified KV cache is enabled\n", __func__);
    params.kv_unified = true;
}

So in the context params, you either have to set kv_unified = true or set n_seq_max to some reasonable maximum (n_parallel gets mapped to n_seq_max). If you use kv_unified=true the max number of sequences will be LLAMA_MAX_SEQ=64. Unfortunately, neither of these is settable from llama-cpp-python.

I think just replicating the logic above is a good solution. I can submit a PR today.

[astrowonk]

I'd love to see the PR #2058 that fixes this merged / released soon… this bug as it is breaks demonstration code I was planning to show on how to use llamacpp to generate multiple embeddings.

[JamePeng]

I refactored and extracted embedding logic to LlamaEmbedding class, Rerank support and fix parallel batching
See:
https://github.com/JamePeng/llama-cpp-python?tab=readme-ov-file#embeddings--reranking-gguf