fix: PR #4 post-merge review follow-ups - #5
Draft
radudobrinescu wants to merge 3 commits into
Draft
radudobrinescu wants to merge 3 commits into
radudobrinescu wants to merge 3 commits into
Conversation
… RBAC) The readiness probe (added in #4) swallowed ALL ApiException statuses and returned 200, so a 403 — the ServiceAccount can't list the CRs it exists to reconcile, e.g. a partial/rolled-back RBAC apply (exactly what #4's new bedrockmodels rule can trigger) — reported healthy while the single-replica controller registered nothing. Now only 404 (CRD absent: kro=false / Bedrock-only install) is treated as healthy; 403/401/other API errors and transport failures return 503, so a controller that can reconcile nothing surfaces a failing signal instead of a green probe.
radudobrinescu
marked this pull request as draft
September 18, 2026 09:17
register_model was monotonic (add only when no DB row), so editing a live BedrockModel's spec.inputCostPerToken/outputCostPerToken/bedrockEndpoint (or a serving model's params) was accepted by k8s but silently ignored by LiteLLM — cost tracking stayed wrong with no signal. Now, when a row already exists, compare the CR's litellm_params against the live ones and re-register (delete+add under the lock) on a genuine change. Drift detection is conservative so it cannot churn: it only compares keys LiteLLM echoes back in /model/info (a key absent from the live params is treated as 'can't compare -> not drift', degrading to the old no-hot-apply behavior for that field rather than re-registering every reconcile), excludes the redacted api_key, and uses a tolerance for float costs. Steady state never triggers it; only a real edit does.
…ot a guess token_prices() inferred per-1K vs per-1M from free-text substrings and DEFAULTED to per-1K, so a per-1,000,000 Price List dimension with a bare 'tokens' unit was divided by 1,000 -> a per-token cost 1000x too high, baked into the committed CR and driving LiteLLM/Langfuse cost tracking + budgets. Read the scale structurally (_tokens_per_unit): match the numeric magnitude (1,000,000 before 1,000) in the unit/description, and return None when ambiguous so the dimension is skipped (fall back to --input-cost/--output-cost or LiteLLM's map) rather than guessed. Add a sanity clamp that rejects implausibly high per-token values (> $1000/1M) as a unit-scale backstop. Adds tests/test_bedrock_pricing.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up fixes from the post-merge review of #4. Draft while the batch accumulates; validated together, merged once.
litellm-sync readiness probe: fail on non-404 API errors. PR #3 review fixes (batch) #4 broadened the probe to swallow all
ApiExceptionstatuses (200), so a 403 (broken/rolled-back RBAC — which PR #3 review fixes (batch) #4's newbedrockmodelsrule can trigger) reported healthy while the single-replica controller registered nothing. Now only 404 (CRD absent: kro=false / Bedrock-only) is healthy; 403/401/other + transport failures return 503.litellm-sync: re-register a model when its CR params drift. Editing a live
BedrockModel's price/endpoint (or a serving model's params) was accepted by k8s but silently ignored by LiteLLM (monotonic registration). Nowregister_modeldiffs the CR'slitellm_paramsagainst the live registration and re-registers (delete+add under the lock) on a genuine change. Conservative so it cannot churn: compares only keys LiteLLM echoes in/model/info(missing key ⇒ not drift), excludes redactedapi_key, tolerances float costs.new-model: derive Bedrock per-token price scale from the unit, not a guess.
token_prices()inferred per-1K vs per-1M from free text and defaulted to per-1K, so a per-1,000,000 dimension with a baretokensunit was recorded 1000x too high into the committed CR (wrong spend tracking + budgets). Now reads the scale structurally (_tokens_per_unit: numeric magnitude, 1M before 1K), returns None when ambiguous (skip → fall back to--input-cost/--output-costor LiteLLM's map), and adds a sanity clamp rejecting implausibly high per-token values. Addstests/test_bedrock_pricing.py.Pre-merge note: fix #2 (drift-reconcile) should get a live smoke-test — edit an enrolled Bedrock model's price and confirm LiteLLM picks it up — since LiteLLM's
/model/infoparam echo can't be verified statically. It fails safe (no churn) if a field isn't echoed.86 unit tests pass.