Skip to content

feat(relay): report answer fallback in routing marks - #612

Draft
afourniernv wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
afourniernv:codex/relay-answer-fallback-telemetry
Draft

feat(relay): report answer fallback in routing marks#612
afourniernv wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
afourniernv:codex/relay-answer-fallback-telemetry

Conversation

@afourniernv

@afourniernv afourniernv commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Stacked on #604. Until it lands, review the incremental commit. This branch will then be rebased onto main.

Why

The routing decision currently records the model Switchyard selected, while the Relay LLM span records the model that eventually answered. If the selected model fails and Switchyard falls back, neither record explains the difference, and the failed answer attempt is not visible.

This makes the complete serving path visible without changing routing, retries, fallback, responses, or metrics.

What changed

Mark Change
switchyard.routing.decision Adds nullable served_model and fallback_used fields.
switchyard.routing.llm_call Emits one mark for each answer candidate, including call order, model, outcome, and latency.

Trace shape

Before, fallback has to be inferred from the selected model and the outer response:

agent-run
├── openai.chat_completions              response from model/strong
├── mark:switchyard.routing.requested    algorithm=random
├── mark:switchyard.routing.overhead
└── mark:switchyard.routing.decision     selected=model/weak

After, both answer attempts and the final serving path are explicit:

agent-run
├── openai.chat_completions               response from model/strong
├── mark:switchyard.routing.requested     algorithm=random
├── mark:switchyard.routing.llm_call      answer #1, model/weak, error
├── mark:switchyard.routing.llm_call      answer #2, model/strong, ok
├── mark:switchyard.routing.overhead
└── mark:switchyard.routing.decision      model/weak → model/strong, fallback=true

With an agent scope, Relay parents the LLM span and Switchyard marks into one trace. The smoke harness used for these screenshots had no agent scope, so Phoenix displays them as separate roots.

Live Phoenix result

Before

The decision records model/weak, while the successful LLM span reports model/strong. There is no fallback field or failed-attempt mark.

Before: the decision mark records model/weak

Before: model/strong served the response

After

The decision reports the selected and served models and fallback_used = true. Two answer-call marks show model/weak failing before model/strong succeeds.

After: the decision records selected model, served model, and fallback

Answer-candidate attempt details
First candidate Fallback candidate
The model/weak call failed The model/strong call succeeded

Contract notes

  • These are additive fields and values on the version-1 schemas introduced in feat(relay): add schemas to routing marks #604.
  • served_model and fallback_used are null when serving metadata is unavailable.
  • For streaming answers, outcome = ok and latency cover creation of the upstream stream. Later stream failures continue through switchyard.routing.error.
  • No provider response body or free-form provider error is added to telemetry.

Testing

  • Focused plugin tests, formatting, and Clippy pass; GitHub CI runs the full suite.
  • Live smoke test: model/weak returned HTTP 503, model/strong succeeded, and the resulting marks were verified in Phoenix.
  • A direct successful call reports the same selected and served model with fallback_used = false.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv
afourniernv force-pushed the codex/relay-answer-fallback-telemetry branch from 2876b35 to 9713c0f Compare September 3, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant