feat: add decision-model-backed tool discovery - #522
danieljvdm wants to merge 4 commits into
Conversation
danieljvdm
changed the title
Bundle size
Minified ESM for es2022, browser target, including Effect and other dependencies. Gzip is measured per chunk. Initial includes statically imported shared chunks; deferred is the remaining output. New exports have no prior baseline. Chunks, module analysis, and exact bytes for |
![effect-agent[bot]](https://avatars.githubusercontent.com/in/4652591?s=60&v=4){kind=small}
There was a problem hiding this comment.
Effect Agent review
Tip
No actionable findings.
| Scope | Files | New findings |
|---|---|---|
| Full diff | 14 reviewed · 1 ignored | ✅ None |
Summary
No concrete defects found in the supplied change.
6 model calls · 123,057 input (1,374 uncached · 91,407 cached · 30,276 cache write; 74.3% cache reads) / 738 output tokens · ≈ $1.04 at GPT-6 Astra rates · $20.591580 spending ceiling · inspected at b2c24c4 · 4 automatic reviews remain
There was a problem hiding this comment.
Effect Agent review
Tip
No actionable findings.
| Scope | Files | New findings |
|---|---|---|
| Incremental | 8 reviewed | ✅ None |
Summary
No concrete defects found in the supplied change. Earlier findings remain open unless explicitly verified as addressed; an incremental review does not establish that merging is safe.
6 model calls · 81,889 input (1,374 uncached · 61,812 cached · 18,703 cache write; 75.5% cache reads) / 473 output tokens · ≈ $0.6660 at GPT-6 Astra rates · $20.333980 spending ceiling · inspected at 0f3b0e4 · 3 automatic reviews remain
There was a problem hiding this comment.
Effect Agent review
Tip
No actionable findings.
| Scope | Files | New findings |
|---|---|---|
| Incremental | 6 reviewed | ✅ None |
Summary
No concrete defects found in the supplied change. Earlier findings remain open unless explicitly verified as addressed; an incremental review does not establish that merging is safe.
5 model calls · 56,736 input (1,145 uncached · 41,757 cached · 13,834 cache write; 73.6% cache reads) / 344 output tokens · ≈ $0.4867 at GPT-6 Astra rates · $20.179660 spending ceiling · inspected at e1876a9 · 2 automatic reviews remain
There was a problem hiding this comment.
Effect Agent review
Tip
No actionable findings.
| Scope | Files | New findings |
|---|---|---|
| Incremental | 0 reviewed | ✅ None |
Summary
No pull-request files changed since the last completed review.
No model call · $0.000000 spending ceiling · inspected at 5041735 · 1 automatic review remains
5041735 to
31001ef
Compare
There was a problem hiding this comment.
Effect Agent review
Tip
No actionable findings.
| Scope | Files | New findings |
|---|---|---|
| Full diff | 17 reviewed · 1 ignored | ✅ None |
Note
Automatic reviews are paused for this pull request.
Further pushes will not start another review. Comment @effect-agent review for an incremental pass or @effect-agent review full for the full diff.
Summary
No concrete defects found in the supplied change.
11 model calls · 313,425 input (2,519 uncached · 273,082 cached · 37,824 cache write; 87.1% cache reads) / 777 output tokens · ≈ $1.62 at GPT-6 Astra rates · $20.887870 spending ceiling · inspected at 31001ef
1 participant
When an agent's initial shortlist misses a capability,
discover_toolscan now rank the eligible catalogue semantically through the same provider-neutralDecisionModelused by initial selection. This PR is stacked on #520.sequenceDiagram participant Agent participant Discovery as discover_tools participant Model as DecisionModel participant Runtime Agent->>Discovery: Capability query, optional namespace Discovery->>Model: Bounded query + eligible metadata Model-->>Discovery: Independent relevance probabilities Discovery-->>Runtime: Bounded tool documentation + native names Runtime-->>Agent: Activate selected schemas next turnToolSelectorand semantic discovery share relevance ranking and stable tie-breaking. Discovery projects only its query and optional namespace, excludes itself, and retains the existing authority checks, schema/documentation budgets, pins, and next-turn replacement semantics. Empty catalogues skip evaluation; no matches return an empty selection. Provider errors and declared observer failures remain typed, and the host owns deadlines and separately billed decision usage. Keyword search remains available throughToolDiscovery.make; semantic search does not silently fall back to it.The opt-in benchmark now compares the fixed/JEV initial-selector × keyword/JEV discovery matrix with full-catalogue controls and experimental stable-catalogue OpenAI transport. The latter keeps all 50 business definitions plus discovery in a fixed order and changes native
allowed_tools; runtime exposure checks still enforce the active subset. Separate variants append persistent availability messages, preserving earlier messages when the callable subset changes. This transport remains benchmark-only; it is not a new framework API.Measured 560 agent attempts: seven synthetic tasks × ten configurations × four repetitions × two context sizes. The tasks include paraphrases, two deliberate initial omissions, and a four-record dependent chain. All shortlist configurations start with eight business tools plus discovery. The reference context adds about 3,342 tokens of identical synthetic archive material; business handlers remain local, read-only fixtures with uniform schemas. All attempts, including policy-limit failures, contribute to costs and latency.
The model is
gpt-6-astra, low reasoning, default service tier,store=false, 2,048 maximum output tokens, no retries. Agent arms share unsalted prefixes and run serially in rotated order. “Warm” means repetitions 2–4 after every task/configuration has run once; it does not mean an isolated cache namespace. Short-context availability variants were added after observing raw-stable failures and ran later than the controls; their latency comparisons are exploratory.Reference context: all configurations, costs, latency and warm-up sensitivity
Short context: all configurations, costs, latency and warm-up sensitivity
For reference-context JEV selection/discovery, stable definitions plus availability messages change all-attempt cost from $0.0332 to $0.0266, but warmed cost changes from $0.0203 to $0.0259. Repeated physical subsets can reuse earlier prefixes: the all-attempt result is not a steady-state savings claim. Raw
allowed_toolsconfigurations also frequently looped instead of discovering a missing capability; explicit availability messages changed that behavior. A representative rare-tool attempt called the customer-profile tool six times, while the availability variant discovered and executed the required retention-exception lookup.A separate cache probe isolates transport from agent behavior: 48 independently salted trials / 192 OpenAI requests, all passing. Each trial starts with zero cached-read tokens, then makes four identical shipping requests. A/A/A/A holds eight physical definitions fixed; A/B/C/A changes the callable eight while keeping the required shipping tool present. Costs below include the cold request and cache writes. No handlers or JEV calls run in the probe.
The experiment made 2,381 OpenAI calls and 461 JEV calls. Returned usage accounts for an estimated $18.1426, including $0.0853 JEV; no usage or cache-write counters are missing. Estimates use OpenAI standard pricing and JEV input pricing, including cache writes, reads and all failed attempts. Four repetitions per synthetic task do not establish production reliability or tail latency. Fresh conversations with heterogeneous tools, remote business-handler latency, native deferred loading and per-turn reranking remain unmeasured.
Measured clean source revisions:
0f3b0e47170575be0db82a725e380e9619d04e29,e1876a95426b0896f6d9db6e426eaf8c30abaed1(Darwin arm64, Bun 1.4.2). Every paid request has a start/completion checkpoint and captured synthetic request/usage data. The controlled probes substantiate cache preservation; the agent tables measure the additional behavioral and billing tradeoffs.