Skip to content

LCORE-0000: Switch default OpenAI model from gpt-4o-mini to gpt-5.6-luna #3027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jrobertboos wants to merge 1 commit into from
Closed

LCORE-0000: Switch default OpenAI model from gpt-4o-mini to gpt-5.6-luna #3027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/e2e_tests_providers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ jobs:
# Expected default LLM for Behave (matches tests/e2e/configs/run-<env>.yaml).
# | environment | model_id | provider_id |
# |-------------|-------------------------------------|---------------|
# | azure | gpt-4o-mini | azure |
# | azure | gpt-5.6-luna | azure |
# | vertexai | google/gemini-2.5-flash | google-vertex |
# | watsonx | meta-llama/llama-3-3-70b-instruct | watsonx |
# | bedrock | deepseek.v3-v1:0 | aws-bedrock |
include:
- environment: azure
e2e_default_model: gpt-4o-mini
e2e_default_model: gpt-5.6-luna
e2e_default_provider: azure
- environment: vertexai
e2e_default_model: google/gemini-2.5-flash
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ start-llama-stack-container: build-llama-stack-image ## Start llama-stack contai
-e OPENAI_API_KEY \
-e BRAVE_SEARCH_API_KEY \
-e TAVILY_SEARCH_API_KEY \
-e E2E_OPENAI_MODEL=$${E2E_OPENAI_MODEL:-gpt-4o-mini} \
-e E2E_OPENAI_MODEL=$${E2E_OPENAI_MODEL:-gpt-5.6-luna} \
-e TENANT_ID=$${TENANT_ID:-} \
-e CLIENT_ID=$${CLIENT_ID:-} \
-e CLIENT_SECRET \
Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ __Note__: Support for individual models is dependent on the specific inference p
| OpenAI | gpt-3.5-turbo, gpt-4 | No | remote::openai | |
| RHOAI (vLLM) | meta-llama/Llama-3.2-1B-Instruct | Yes | remote::vllm | [1](tests/e2e-prow/rhoai/configs/run.yaml) |
| RHEL AI (RHAIIS/vLLM) | meta-llama/Llama-3.1-8B-Instruct | Yes | remote::vllm | [1](tests/e2e/configs/run-rhelai.yaml) [2](tests/e2e/configs/run-rhaiis.yaml) |
| Azure | gpt-5, gpt-5-mini, gpt-5-nano, gpt-4o-mini, o3-mini, o4-mini, o1 | Yes | remote::azure | [1](examples/azure-run.yaml) |
| Azure | gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.6-luna, o3-mini, o4-mini, o1 | Yes | remote::azure | [1](examples/azure-run.yaml) |
| Azure | gpt-5-chat, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, o1-mini | No or limited | remote::azure | |
| VertexAI | google/gemini-2.0-flash, google/gemini-2.5-flash, google/gemini-2.5-pro [^1] | Yes | remote::vertexai | [1](examples/vertexai-run.yaml) |
| WatsonX | meta-llama/llama-3-3-70b-instruct | Yes | remote::watsonx | [1](examples/watsonx-run.yaml) |
Expand Down Expand Up @@ -1193,12 +1193,12 @@ will be returned.
"model_type": "embedding"
},
{
"identifier": "openai/gpt-4o-mini",
"identifier": "openai/gpt-5.6-luna",
"metadata": {},
"api_model_type": "llm",
"provider_id": "openai",
"type": "model",
"provider_resource_id": "gpt-4o-mini",
"provider_resource_id": "gpt-5.6-luna",
"model_type": "llm"
},
{
Expand Down
2 changes: 1 addition & 1 deletion docker-compose-library.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ services:
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
# OpenAI
- OPENAI_API_KEY=${OPENAI_API_KEY}
- E2E_OPENAI_MODEL=${E2E_OPENAI_MODEL:-gpt-4o-mini}
- E2E_OPENAI_MODEL=${E2E_OPENAI_MODEL:-gpt-5.6-luna}
# Azure Entra ID credentials (AZURE_API_KEY is obtained dynamically in Python)
- TENANT_ID=${TENANT_ID:-}
- CLIENT_ID=${CLIENT_ID:-}
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ services:
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
# OpenAI
- OPENAI_API_KEY=${OPENAI_API_KEY}
- E2E_OPENAI_MODEL=${E2E_OPENAI_MODEL:-gpt-4o-mini}
- E2E_OPENAI_MODEL=${E2E_OPENAI_MODEL:-gpt-5.6-luna}
# Azure Entra ID credentials (AZURE_API_KEY is passed via provider_data at request time)
- TENANT_ID=${TENANT_ID:-}
- CLIENT_ID=${CLIENT_ID:-}
Expand Down
2 changes: 1 addition & 1 deletion docs/config.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1016,7 +1016,7 @@ <h2 id="inferenceconfiguration">InferenceConfiguration</h2>
<tr class="odd">
<td>context_windows</td>
<td>object</td>
<td>Map of fully-qualified model identifier (e.g., &#x201C;openai/gpt-4o-mini&#x201D;)
<td>Map of fully-qualified model identifier (e.g., &#x201C;openai/gpt-5.6-luna&#x201D;)
to context window size in tokens. Used by the conversation compaction
trigger to decide when older turns must be summarized before the input
exceeds the window. Models absent from this map have no registered
Expand Down
2 changes: 1 addition & 1 deletion docs/config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -885,7 +885,7 @@
"minimum": 0,
"type": "integer"
},
"description": "Map of fully-qualified model identifier (e.g., \"openai/gpt-4o-mini\") to context window size in tokens. Used by the conversation compaction trigger to decide when older turns must be summarized before the input exceeds the window. Models absent from this map have no registered window — callers fall back to their own default or skip the token-based trigger.",
"description": "Map of fully-qualified model identifier (e.g., \"openai/gpt-5.6-luna\") to context window size in tokens. Used by the conversation compaction trigger to decide when older turns must be summarized before the input exceeds the window. Models absent from this map have no registered window — callers fall back to their own default or skip the token-based trigger.",
"title": "Per-model context window sizes (tokens)",
"type": "object"
},
Expand Down
2 changes: 1 addition & 1 deletion docs/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@ Inference configuration.
|------------------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| default_model | string | Identification of default model used when no other model is specified. |
| default_provider | string | Identification of default provider used when no other model is specified. |
| context_windows | object | Map of fully-qualified model identifier (e.g., "openai/gpt-4o-mini") to context window size in tokens. Used by the conversation compaction trigger to decide when older turns must be summarized before the input exceeds the window. Models absent from this map have no registered window — callers fall back to their own default or skip the token-based trigger. |
| context_windows | object | Map of fully-qualified model identifier (e.g., "openai/gpt-5.6-luna") to context window size in tokens. Used by the conversation compaction trigger to decide when older turns must be summarized before the input exceeds the window. Models absent from this map have no registered window — callers fall back to their own default or skip the token-based trigger. |
| providers | array | Unified-mode synthesis input (Decision S5): a high-level, backend-agnostic list of inference providers the synthesizer expands into Llama Stack provider entries. Lives at the configuration root so it survives a future backend change. A non-empty list signals unified mode. Empty (the default) leaves legacy/remote modes unaffected. The sibling default_model / default_provider keep their query-time routing meaning and are independent of this list. |


Expand Down
2 changes: 1 addition & 1 deletion docs/container_orchestration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ See [OKP Guide](okp_guide.md) for detailed setup instructions.

#### Other Configuration

- `E2E_OPENAI_MODEL`: OpenAI model for E2E tests (default: `gpt-4o-mini`)
- `E2E_OPENAI_MODEL`: OpenAI model for E2E tests (default: `gpt-5.6-luna`)
- `LLAMA_STACK_LOGGING`: Enable debug logging in llama-stack
- `FAISS_VECTOR_STORE_ID`: FAISS vector store identifier
- `LITELLM_DROP_PARAMS`: Drop unsupported params in LiteLLM (default: `true`)
Expand Down
6 changes: 3 additions & 3 deletions docs/deployment_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,7 +535,7 @@ cp examples/lightspeed-stack-lls-library.yaml lightspeed-stack.yaml
DEBUG Set provider/model configuration for openai/openai/gpt-4o to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-4o-2024-08-06 to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-4o-audio-preview to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-4o-mini to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-5.6-luna to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/o1 to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/o1-mini to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/o3-mini to 0 utils.py:45
Expand Down Expand Up @@ -1137,12 +1137,12 @@ Please note that actual response from the service is larger. It was stripped dow
"model_type": "llm"
},
{
"identifier": "openai/gpt-4o-mini",
"identifier": "openai/gpt-5.6-luna",
"metadata": {},
"api_model_type": "llm",
"provider_id": "openai",
"type": "model",
"provider_resource_id": "gpt-4o-mini",
"provider_resource_id": "gpt-5.6-luna",
"model_type": "llm"
},
{
Expand Down
2 changes: 1 addition & 1 deletion docs/design/conversation-compaction/conversation-compaction-spike.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,7 @@ To verify: check that the docs site renders correctly and OpenAPI spec validates

# PoC results

A proof-of-concept was built in lightspeed-stack and tested against a real Llama Stack + OpenAI (gpt-4o-mini) setup.
A proof-of-concept was built in lightspeed-stack and tested against a real Llama Stack + OpenAI (gpt-5.6-luna) setup.

## What the PoC does

Expand Down
6 changes: 3 additions & 3 deletions docs/design/conversation-compaction/conversation-compaction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ Context window sizes are configured per model in YAML:

``` yaml
inference:
default_model: openai/gpt-4o-mini
default_model: openai/gpt-5.6-luna
context_windows:
openai/gpt-4o-mini: 128000
openai/gpt-5.6-luna: 128000
openai/gpt-4o: 128000
```

Expand Down Expand Up @@ -369,7 +369,7 @@ Example config files go in `examples/`.
| Trigger turn | 1 LLM call (or 413) | 2 LLM calls (summarize + respond) |
| Post-trigger turn | 1 LLM call | 1 LLM call (no change) |

Compaction adds latency only on the trigger turn. In PoC testing, compaction turns took 14-40 seconds vs 9-20 seconds for normal turns (gpt-4o-mini).
Compaction adds latency only on the trigger turn. In PoC testing, compaction turns took 14-40 seconds vs 9-20 seconds for normal turns (gpt-5.6-luna).

# Open Questions for Future Work

Expand Down
6 changes: 3 additions & 3 deletions docs/design/llama-stack-config-merge/llama-stack-config-merge-spike.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ also stay under `llama_stack.config` whenever they ship as high-level keys
Literal values (`openai`, `azure`, `sentence_transformers`, `vertexai`,
`watsonx`, `vllm_rhaiis`, `vllm_rhel_ai`). They are vendor identifiers
that both Llama Stack (`provider_type: remote::openai`) and Pydantic AI
(model-string prefixes such as `openai:gpt-4o-mini`) recognise. Each
(model-string prefixes such as `openai:gpt-5.6-luna`) recognise. Each
backend-specific synthesizer translates the canonical LCORE vocabulary to
its target shape; we do not adopt either backend's surface verbatim.

Expand Down Expand Up @@ -920,7 +920,7 @@ Summary of validation:
from production design").
- **Vacuous safety-shield validation in the library-mode PoC**. The
`native_override` used during PoC validation registered `llama-guard`
with `provider_shield_id: openai/gpt-4o-mini` — an OpenAI chat model,
with `provider_shield_id: openai/gpt-5.6-luna` — an OpenAI chat model,
not a Llama Guard checkpoint. The "`native_override` took effect"
evidence row above only shows that the key landed in the synthesized
output; it does **not** show that a real safety shield gated any
Expand Down Expand Up @@ -1163,7 +1163,7 @@ curl -sSL -o /tmp/lcore-836-poc/lightspeed-stack-unified-library.yaml \

# 1. Start LCORE in library mode with the unified config
export OPENAI_API_KEY=<your-key>
export E2E_OPENAI_MODEL=gpt-4o-mini
export E2E_OPENAI_MODEL=gpt-5.6-luna
uv run lightspeed-stack \
-c /tmp/lcore-836-poc/lightspeed-stack-unified-library.yaml

Expand Down
10 changes: 5 additions & 5 deletions docs/design/llama-stack-config-merge/llama-stack-config-merge.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,12 +243,12 @@ Top-level high-level sections plus a sub-section under the existing
# with a `providers:` list (Decision S5). Backend-agnostic: it stays at the
# root, not under `llama_stack`, so it survives a future backend change.
inference:
default_model: gpt-4o-mini # existing — query-time default routing
default_model: gpt-5.6-luna # existing — query-time default routing
default_provider: openai # existing — query-time default routing
providers: # NEW — high-level provider setup (synthesis input)
- type: openai # mapped to remote::openai
api_key_env: OPENAI_API_KEY
allowed_models: [gpt-4o-mini]
allowed_models: [gpt-5.6-luna]
- type: sentence_transformers

llama_stack:
Expand Down Expand Up @@ -560,7 +560,7 @@ providers:
provider_type: remote::openai
config:
api_key: ${env.OPENAI_API_KEY}
allowed_models: ["${env.E2E_OPENAI_MODEL:=gpt-4o-mini}"]
allowed_models: ["${env.E2E_OPENAI_MODEL:=gpt-5.6-luna}"]
# ... more ...
```

Expand Down Expand Up @@ -601,7 +601,7 @@ llama_stack:
provider_type: remote::openai
config:
api_key: ${env.OPENAI_API_KEY}
allowed_models: ["${env.E2E_OPENAI_MODEL:=gpt-4o-mini}"]
allowed_models: ["${env.E2E_OPENAI_MODEL:=gpt-5.6-luna}"]
# ... rest of run.yaml content under native_override ...
# ... rest of lightspeed-stack.yaml content ...
```
Expand All @@ -624,7 +624,7 @@ providers:
provider_type: remote::openai
config:
api_key: ${env.OPENAI_API_KEY}
allowed_models: ["${env.OPENAI_MODEL:=gpt-4o-mini}"]
allowed_models: ["${env.OPENAI_MODEL:=gpt-5.6-luna}"]
- provider_id: sentence-transformers
provider_type: inline::sentence-transformers
# ... the rest is the same shape as a working run.yaml ...
Expand Down
2 changes: 1 addition & 1 deletion docs/e2e_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
| `E2E_LLAMA_PORT` | `8321` | Port of the Llama Stack service. |
| `E2E_LLAMA_STACK_URL` | — | Full base URL for Llama Stack (overrides host/port if set). Used by shield helpers. |
| `E2E_LLAMA_STACK_API_KEY` | `xyzzy` | API key for Llama Stack client (e.g. shield API). |
| `E2E_DEFAULT_MODEL_OVERRIDE` | — | Override default LLM model id (e.g. `gpt-4o-mini`). |
| `E2E_DEFAULT_MODEL_OVERRIDE` | — | Override default LLM model id (e.g. `gpt-5.6-luna`). |
| `E2E_DEFAULT_PROVIDER_OVERRIDE` | — | Override default provider id (e.g. `openai`). |
| `FAISS_VECTOR_STORE_ID` | — | Vector store id for FAISS-related scenarios. |
| `RUNNING_PROW` | — | Set in Prow/OpenShift; enables Prow config paths and pod/container ops. |
Expand Down
2 changes: 1 addition & 1 deletion docs/getting_started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ authentication:
DEBUG Set provider/model configuration for openai/openai/gpt-4o to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-4o-2024-08-06 to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-4o-audio-preview to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-4o-mini to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/gpt-5.6-luna to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/o1 to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/o1-mini to 0 utils.py:45
DEBUG Set provider/model configuration for openai/openai/o3-mini to 0 utils.py:45
Expand Down
2 changes: 1 addition & 1 deletion docs/models/compaction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Attributes:
produced. Kept as a string (not datetime) to match the
cache schema convention used elsewhere in the codebase.
model_used: Fully-qualified model identifier used for the
summarization LLM call (e.g., ``"openai/gpt-4o-mini"``).
summarization LLM call (e.g., ``"openai/gpt-5.6-luna"``).
Preserved for audit and for diagnostics when summary
quality varies between models.

Expand Down
2 changes: 1 addition & 1 deletion docs/models/responses_succ.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,7 +593,7 @@ Inference configuration.
|-------|------|-------------|
| default_model | string | Identification of default model used when no other model is specified. |
| default_provider | string | Identification of default provider used when no other model is specified. |
| context_windows | object | Map of fully-qualified model identifier (e.g., "openai/gpt-4o-mini") to context window size in tokens. Used by the conversation compaction trigger to decide when older turns must be summarized before the input exceeds the window. Models absent from this map have no registered window — callers fall back to their own default or skip the token-based trigger. |
| context_windows | object | Map of fully-qualified model identifier (e.g., "openai/gpt-5.6-luna") to context window size in tokens. Used by the conversation compaction trigger to decide when older turns must be summarized before the input exceeds the window. Models absent from this map have no registered window — callers fall back to their own default or skip the token-based trigger. |


## InfoResponse
Expand Down
Loading
Loading