Skip to content

fix(pulse): map adapter models to inference levels that exist - #1648

Open
elhoim wants to merge 1 commit into
danielmiessler:mainfrom
elhoim:fix/adapter-inference-levels
Open

fix(pulse): map adapter models to inference levels that exist#1648
elhoim wants to merge 1 commit into
danielmiessler:mainfrom
elhoim:fix/adapter-inference-levels

Conversation

@elhoim

@elhoim elhoim commented Jul 26, 2026

Copy link
Copy Markdown

The problem

AdapterRunner.ts's modelToLevel() could only return "fast", "smart" or "standard". Those names were removed from the inference layer on 2026-06-10. Inference.ts now validates against low | medium | high | max and throws on anything else:

[Inference] unknown level 'smart' — use low | medium | high | max

So every level the adapter could produce was rejected. Nothing caught it: callInference and runAdapter have no catch, AdapterCli.ts top-level-awaits runAdapter so the process dies, and RebuildAll.ts uses Promise.allSettled, printing each rejection as an unlabelled "unhandled" row. Only the cache-hit, no-sources and missing-prompt early returns avoid inference, so 100% of actual adapter builds failed.

Why it compiled

This is the part worth fixing properly. Inference.ts declared its own copy of the level names:

export type InferenceLevel = 'low' | 'medium' | 'high' | 'max';   // a copy

models.ts already had EffortLevel as the real source of truth. Two hand-maintained lists of the same names means a rename in one leaves the other stale and still type-checking — which is exactly what happened. The type mismatch didn't compile-error because the adapter was typed against a copy that agreed with itself.

The fix

Bind both ends to one source.

export type InferenceLevel = EffortLevel;                              // alias, not a copy
const VALID_LEVELS: readonly InferenceLevel[] = Object.keys(EFFORT_MODEL) as InferenceLevel[];

The type now IS models.ts's level type, and the runtime validator derives from EFFORT_MODEL's keys. Rename a level in models.ts and every stale call site fails at the type level rather than at runtime. The error message and the CLI's own validation also read from VALID_LEVELS instead of repeating the literals a third and fourth time.

Mapping, extracted to adapters/model-level.ts and typed InferenceLevel:

model family level
haiku low
sonnet medium
opus high
fable max
anything else medium

Cheap models map to cheap tiers, and an unrecognised model gets a sane middle default rather than the most expensive one.

Failures stop being unlabelled. callInference catches a rejection and returns the same structured failure shape as every other error path, so one bad page reports as a named page with a real message and the batch keeps going.

Tests

test/pulse/adapters/model-level.test.ts, 8 cases:

8 pass
0 fail
49 expect() calls

Separately verified that every level the mapping can emit is one the inference layer actually accepts, checked against EFFORT_MODEL's real keys rather than a hardcoded list:

VALID_LEVELS (from EFFORT_MODEL): max, high, medium, low
✓ claude-haiku-4-5       → low
✓ claude-sonnet-5        → medium
✓ claude-opus-5          → high
✓ claude-fable-5         → max
✓ gpt-5                  → medium
✓ <empty>                → medium
✓ weird-unknown-model    → medium

No API calls are made by the tests or the check.

AdapterRunner.modelToLevel returned "fast" / "standard" / "smart" — names
Inference.ts removed on 2026-06-10. normalizeLevel() rejects all three, so
every level the mapping could produce was refused and 100% of adapter builds
that reached inference threw: AdapterCli died on its top-level await, and
RebuildAll's Promise.allSettled printed an unlabelled "unhandled" row.

- Derive the mapping from models.ts EFFORT_MODEL instead of restating level
  names (new adapters/model-level.ts): haiku->low, sonnet->medium, opus->high,
  fable->max. A tie resolves to the cheaper rung and a non-Claude model falls
  back to medium, so a manifest never silently buys a costlier rung than the
  model it names. Previously fable fell through to the sonnet rung.
- Bind InferenceLevel to models.ts EffortLevel and derive VALID_LEVELS from
  EFFORT_MODEL's keys, so the type, the validator, and the CLI all follow one
  source of truth rather than three hand-kept copies.
- Catch a rejecting inference() in callInference: a bad request now yields a
  structured inference-failed result, written to the data plane like any other
  failure, instead of escaping runAdapter.
- Report a rejected page by id and reason in RebuildAll and AdapterCli.
- Add test/pulse/adapters/model-level.test.ts (8 tests, no API calls) covering
  every model family plus the mapping/inference drift guard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant