Skip to content

fix(vllm): apply Options[] engine flags before engine init - #11147

Open
localai-bot wants to merge 1 commit into
masterfrom
fix/vllm-options-engine-args
Open

fix(vllm): apply Options[] engine flags before engine init#11147
localai-bot wants to merge 1 commit into
masterfrom
fix/vllm-options-engine-args

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Fixes #11130

Problem

CLI-style flags in a model's options: array were silently discarded:

backend: cuda13-vllm
options:
  - --quantization:gptq_marlin
  - --enable-prefix-caching
  - --kv-cache-dtype:fp8_e5m2
  - --reasoning-parser:qwen3

The vLLM backend read Options[] only for tool_parser/reasoning_parser, and read it after AsyncLLMEngine.from_engine_args() - past the point where anything it set could still reach the engine.

Fix

apply_options_to_engine_args() in backend/python/common/vllm_utils.py maps -- prefixed options onto the AsyncEngineArgs dataclass, applied in LoadModel before the engine is constructed.

  • Only -- prefixed entries are engine flags, so tool_parser:, reasoning_parser:, vad_only and friends keep their meaning. Both --flag:value (LocalAI convention) and --flag=value (vLLM CLI convention) are accepted.
  • Dashes become underscores (--enable-prefix-caching -> enable_prefix_caching); values are coerced to the target field's type; a bare --flag sets a boolean field to true.
  • Unknown or uncoercible flags warn on stderr and are skipped rather than failing the load - unlike engine_args:, Options[] is a shared bag that carries entries this mapping knows nothing about. Unknown flags get a closest-field hint.
  • --tool-parser/--reasoning-parser are never reported as unknown: LocalAI consumes them itself, and whether the engine dataclass also carries the field varies by vLLM version.

Type inference reads the annotation's base, not a substring. The first cut matched substrings and misread vLLM's dtype: Literal["auto", "float16", "bfloat16"] as a float, silently dropping --dtype:bfloat16; there is a regression test for it. It also handles PEP 563 string annotations, which is how vLLM's arg_utils is written.

Precedence: typed proto fields -> options: -> engine_args:. Since engine_args: lands last, applyEngineArgDefaults in core/config/hooks_vllm.go now skips seeding a production default whose key the user already set as an option - otherwise the seeded enable_prefix_caching: true would silently beat a user's --enable-prefix-caching:false.

normalize_option_key also makes the parser lookup accept the CLI spelling, so --reasoning-parser:qwen3 selects LocalAI's reasoning parser and not just the engine field.

Tests

  • backend/python/common/vllm_utils_test.py - 20 stdlib-only specs covering coercion (bool/int/float/dict/str), bare flags, = separator, unknown-flag warn-and-skip with hint, uncoercible values, Literal-typed fields, PEP 563 string annotations, and non-flag options being left alone.
  • core/config/hooks_test.go - two new specs: a default is not seeded when the user set that key via options:, and unrelated options still get the defaults.
  • New make test-python-helpers target, wired into the lint workflow's dependency-light job. These shared python helper tests previously ran nowhere in CI.

make lint clean; core/config suite green (409 specs).

Not covered

  • vllm-omni builds Omni(**kwargs) rather than an AsyncEngineArgs dataclass, so it needs a different mapping and is untouched here.
  • The dataclass-reflection path is exercised against stand-in dataclasses mirroring vLLM's field shapes (no vLLM wheel in the dev environment), not against vLLM itself. The issue's "deploy and check the engine log" step still wants a run on a GPU host.

Docs

docs/content/features/text-generation.md gains a "CLI-style engine flags in options" section stating the rules and the precedence; .agents/vllm-backend.md records the ordering trap for future work.

CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.

Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.

Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.

The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 golangci-lint
@mudler

mudler commented Jul 27, 2026

Copy link
Copy Markdown
Owner

@localai-org-maint-bot CI is failing, pick this up and fix the branch

Repository owner deleted a comment from localai-org-maint-bot Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vllm: backend Options array not applied to AsyncEngineArgs before engine init

2 participants