LocalAI version:
docker.io/localai/localai:latest-gpu-hipblas (v4.10.0 (7ad0cbf))
Environment, CPU architecture, OS, and Version:
Linux, amd64, PikaOS (Debian Testing-based), latest
Describe the bug
The API endpoint /v1/models/capabilities returns 4096 for a model that has no context_size configured. I would expect Settings -> Performance -> Default Context Size to be used.
To Reproduce
- configure a model with no context_size set in its YAML config
- Go to
hostname/v1/models/capabilities
Expected behavior
Default context size is used when a model is not configured
Logs
not relevant
Additional context
🐈
LocalAI version:
docker.io/localai/localai:latest-gpu-hipblas (v4.10.0 (7ad0cbf))
Environment, CPU architecture, OS, and Version:
Linux, amd64, PikaOS (Debian Testing-based), latest
Describe the bug
The API endpoint
/v1/models/capabilitiesreturns 4096 for a model that has nocontext_sizeconfigured. I would expectSettings -> Performance -> Default Context Sizeto be used.To Reproduce
hostname/v1/models/capabilitiesExpected behavior
Default context size is used when a model is not configured
Logs
not relevant
Additional context
🐈