Skip to content

Iter 7: Eval framework, architecture extraction, and tool behavior refactor #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
echo-xiao wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Iter 7: Eval framework, architecture extraction, and tool behavior refactor #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b8e6550
Iter 7: Architecture extraction, tool behavior refactor, Claude judge…
Jun 8, 2026
06890e0
Cleanup eval scripts, add executive report
Jun 8, 2026
641eb50
Move executive-report.md to logs/
Jun 8, 2026
e495187
Add eval framework (layer0/1 baseline+tool eval) and update gemini an…
Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 22 additions & 177 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,38 @@

## Answer Rules

1. **Always include specific file paths** in your answer (e.g., `apps/meteor/app/lib/server/functions/sendMessage.ts`). Every key file in the chain must be listed with its role.
2. **Start from the entry point**, not the middle. For architecture questions, trace the full chain from the top-level entry to the final destination.
3. **Keep tool calls efficient.** Use `search` → `graph` first, then `implement` only at key points (max 2-3 calls). Do NOT keep searching if you already have enough information — write your answer.
1. **ALWAYS call at least one tool.** Never answer from memory alone — your training data has outdated file paths. Use tools to get real paths.
2. **Always include specific file paths** in your answer (e.g., `apps/meteor/app/lib/server/functions/sendMessage.ts`). Every key file in the chain must be listed with its role.
3. **Start from the entry point**, not the middle. For architecture questions, trace the full chain from the top-level entry to the final destination.
4. **List the call chain explicitly** in your answer: `Entry → Step 1 → Step 2 → ... → Final`.
5. **Follow the tool order: search → graph → implement.** You MUST call `search` or `graph` before `implement`. The system enforces this — `implement` will be rejected if you haven't searched first.

## Tools

Three tools only. All other file/shell tools are disabled.

| Tool | When to use |
|------|-------------|
| `search(query, layer?, question?)` | Find entry point by symbol or keyword |
| `graph(symbol, direction, depth?, edgeTypes?, question?)` | Traverse dependency edges from a known symbol |
| `implement(symbol, filename)` | Read full source of one specific symbol — use when you need implementation details |
| Tool | When to use | Cost |
|------|-------------|------|
| `search(query, layer?)` | Find entry point by symbol or keyword | Cheap (~200 tokens) |
| `graph(query, direction?, depth?, layer?, mode?, edgeTypes?)` | Traverse dependency edges from a known symbol | Cheap (~300 tokens) |
| `implement(symbolName, filename)` | Read source of a specific symbol. For classes: returns method signatures — use `implement("Class.method", file)` to read a specific method. | Expensive (1K-5K tokens) |

**`implement` is expensive (returns full source code). Use it at layer boundaries or to confirm key details (max 2-3 calls). Prefer `search` + `graph` for navigation — they return file paths and symbols without consuming excessive tokens. When you have enough information, stop calling tools and write your answer.**
**Strategy: use `search` + `graph` to map the territory first (cheap), then `implement` only at 1-2 key points (expensive). `graph(down)` already shows what a function calls — you don't need `implement` just to see the call chain.**

---

## Navigation Rules

**Default flow for any architectural question:**
**Mandatory flow — follow this order every time:**
```
search → graph(down) → implement only at boundaries
Step 1: search(entry_symbol) → find files + symbols
Step 2: graph(symbol, "down") → map the call chain (cheap, gives you the full picture)
Step 3: implement(symbol, file) → read source ONLY at 1-2 key points
Step 4: STOP and write your answer → include all file paths from steps 1-3
```

**Do NOT skip to implement.** `graph(down)` gives you the same call chain information for 1/10th the token cost. Use `implement` only when you need to see the actual logic inside a function.

**Pick direction:**
- `graph(down)` — what does X invoke? (trace a flow forward)
- `graph(up)` — what calls X? (find callers, assess impact)
Expand All @@ -42,178 +48,17 @@ search → graph(down) → implement only at boundaries
- Component tree: `edgeTypes=['jsx']`
- Full routing: `edgeTypes=['call','event_listen','pubsub_subscribe']`

**If `search` or `graph` returns nothing:** the symbol may be dynamically registered — check the Dynamic Patterns section below before retrying.

---

## Question Type → Entry Strategy

| Type | Strategy |
|------|----------|
| Architecture / Call chain | Check Architecture section for entry point → `search(entry)` → `graph(down)` |
| Locate | `search(keyword)` → `implement` top result |
| Pattern | `search` existing instance → `implement` — skip `graph` |
| Routing | Check Architecture section → `search(dispatcher)` → `graph(down, edgeTypes=[...])` |
| Impact | `search(target)` → `graph(up)` → `implement` top callers |

---

## Architecture

### Client Message Sending
```
RoomBody → ComposerContainer → ComposerMessage → MessageBox
↓ onSend
chat.flows.sendMessage()
↓
sdk.call('sendMessage') ← DDP boundary
```
Entry: `search('MessageBox', layer='client')` → `graph(down)`

Cross DDP boundary: `sdk.call('sendMessage')` → virtual node `'sendMessage'` → server handler (see Dynamic Patterns §A)

---

### Server Message Sending
```
Meteor.methods({ sendMessage }) ← DDP entry (virtual node 'sendMessage')
↓
executeSendMessage ← permission check
↓
sendMessage → Messages.insertOne ← DB write
↓
afterSaveMessage callbacks ← event_emit (see Dynamic Patterns §B)
```
Entry: `search('executeSendMessage', layer='server')` → `graph(down)`

---

### Push Notifications
```
afterSaveMessage → sendMessageNotifications → sendNotification (per user)
↓
shouldNotifyMobile/Desktop/Email
↓
NotificationQueue → PushNotification → APN / FCM
```
Entry: `search('sendNotificationsOnMessage')` → `graph(down)`

---

### REST API
```
ApiClass → authenticationMiddleware → permissionsMiddleware → rate limiter → Route Handler
```
Entry: `search('ApiClass')` or search the specific route path → `graph(down)`

---

### DDP Subscription / Real-time Sync
```
Meteor.subscribe('X') → Meteor.publish('X', fn) → StreamerCentral → DDP push to client
↓
Streamer Client → React re-render
```
Entry: `search('StreamerCentral')` → `graph(down)`

---

### Apps Engine
```
AppManager → AppListenerManager → executeListener()
↓
Bridge layer (adapts core ↔ App)
↓
App hook return value applied to core flow
```
Entry: `search('AppListenerManager')` → `graph(down)`

---

### Authentication
```
Meteor.loginWithPassword/LDAP/OAuth
↓
Accounts.registerLoginHandler → credential validation → { id, token }
↓ (subsequent requests)
x-auth-token header → authenticationMiddleware → Users.findOneByIdAndLoginToken
```
Entry: `search('registerLoginHandler')` → `graph(down)`

---

### Webhook Routing
```
POST /hooks/:integrationId/:token → authenticatedRoute → executeIntegrationRest → processWebhookMessage
```
Entry: `search('executeIntegrationRest')` → `graph(down)`

---

## Dynamic Patterns

These patterns are **not visible via import edges**. The graph connects them via virtual nodes — but only if the dispatch target is a string literal in source.

### A. DDP Method Dispatch
```
sdk.call('sendMessage') → virtual node 'sendMessage'
Meteor.methods({ sendMessage: fn }) → virtual node 'sendMessage' → fn
```
`graph('sendMessage', up)` shows the client caller. `graph('sendMessage', down)` shows the server handler.

### B. Callbacks Event System
```
callbacks.run('afterSaveMessage') → virtual node 'afterSaveMessage'
callbacks.add('afterSaveMessage', handler) → virtual node 'afterSaveMessage' → handler
```
Use `graph('afterSaveMessage', down, edgeTypes=['event_listen'])` to find all registered handlers.

### C. Meteor Pub/Sub
```
Meteor.subscribe('roomMessages') → virtual node 'roomMessages'
Meteor.publish('roomMessages', fn) → virtual node 'roomMessages' → fn
```

### D. core-services Bus
Services do NOT call each other directly — they go through a broker.
```
ServiceName.method(args) → proxify('ServiceName') → LocalBroker → ServiceClass instance
```
If you can't find a service implementation via `graph`, search for the `ServiceClass` with `name = 'ServiceName'`.

### E. Message Rendering (data pipeline, not a call chain)
```
message.msg → parse() → Root AST → <Markup /> → <GazzodownText /> → <MessageContentBody />
```
`graph` cannot traverse this. Use `implement` on each step directly.

### F. Blaze → React (legacy portals)
Some pages use HTML/Blaze templates. React mounts into them via `createPortal`. If you find a `.html` template, look for the React counterpart in a nearby `portals/` or `views/` directory.

### G. Fuselage components
`<Box>`, `<Button>`, `<TextInput>` etc. are from `@rocket.chat/fuselage`. Do NOT traverse into Fuselage for business logic questions.

### H. Message Composer
The composer uses a native textarea + ComposerAPI, **not Slate.js**. Do not search for Slate.
```
MessageBox → ComposerAPI (setText/insertText) → onSend({ value }) → chat.flows.sendMessage()
```

---

## Subsystem Entry Points

| Subsystem | Entry Symbol | File |
|-----------|-------------|------|
| Authorization | `hasPermission` | `apps/meteor/app/authorization/server/functions/hasPermission.ts` |
| Slash commands | `slashCommands` | `apps/meteor/app/utils/server/slashCommand.ts` |
| File upload | `uploadFiles` | `apps/meteor/client/lib/chats/flows/uploadFiles.ts` |
| E2E encryption | `Rocketchate2e` | `apps/meteor/client/lib/e2ee/rocketchat.e2e.ts` |
| Livechat widget | `api` | `packages/livechat/src/api.ts` |
| Livechat routing | `RoutingManager` | `apps/meteor/app/livechat/server/lib/RoutingManager.ts` |
| Federation | `FederationMatrix` | `ee/packages/federation-matrix/src/FederationMatrix.ts` |
| Room service | `RoomService` | `apps/meteor/server/services/room/service.ts` |
| Messages model | `MessagesRaw` | `packages/models/src/models/Messages.ts` |
| Architecture / Call chain | `search(entry)` → `graph(down)` |
| Locate | `search(keyword)` → `graph(down)` → `implement` only if needed |
| Pattern | `search` existing instance → `graph(down)` → `implement` one example |
| Routing | `search(dispatcher)` → `graph(down, edgeTypes=[...])` |
| Impact | `search(target)` → `graph(up, mode="impact")` |

---

Expand Down
23 changes: 12 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@ An offline indexer builds a typed dependency graph (11 edge kinds). Three MCP to
Source (.ts/.tsx)
→ hasher.ts incremental MD5, skip unchanged
→ skeleton.ts AST parse: signatures + 11 typed edges
→ embedder.ts Gemini API: symbol → float32[768]
→ GLOBAL_INDEX symbols · callGraph · fileDependents · embeddings
→ GLOBAL_INDEX symbols · callGraph · fileDependents
↓
AGENTS.md (navigation rules) + MCP tools → LLM
↓
Expand All @@ -43,8 +42,11 @@ git clone https://github.com/RocketChat/Rocket.Chat.git

cd Agentic.Code.Analyzer
npm install
export GEMINI_API_KEY=your_key
npm start

# For eval scripts only:
export GEMINI_API_KEY=your_key
npm run eval:agent
```

The analyzer expects `Rocket.Chat` as a sibling directory by default. To use a different path:
Expand Down Expand Up @@ -96,9 +98,9 @@ This is a standard MCP server. Add to your client's MCP config:

| Tool | Description |
|------|-------------|
| `search(query, layer?, question?)` | Fuzzy symbol search reranked by embedding similarity (0.4 × fuzzy + 0.6 × cosine). Supports `client`/`server` layer filter. |
| `graph(symbol, direction, depth?, edgeTypes?, question?)` | BFS downstream or upstream. When `question` is provided, applies semantic pruning — edges with cosine similarity < 0.1 are dropped. |
| `implement(symbol, filename)` | Full source + up to 5 callee skeletons. Capped at 3 calls per question. |
| `search(query, layer?)` | Fuzzy symbol search with path-aware scoring. Supports `client`/`server` layer filter. |
| `graph(query, direction?, depth?, layer?, mode?, edgeTypes?)` | BFS upstream or downstream traversal. `mode=impact` shows blast radius layer-by-layer. |
| `implement(symbolName, filename)` | Full source + up to 5 callee skeletons. |

## Question types → tool strategy

Expand Down Expand Up @@ -132,14 +134,13 @@ src/
index.ts scan, prewarm, build GLOBAL_INDEX
skeleton.ts AST parse → signatures + 11 edge types
hasher.ts incremental MD5 change detection
embedder.ts Gemini embedding API
state.ts GLOBAL_INDEX type definitions
local-db.ts index persistence to disk
eval/ evaluation framework
session-recorder.ts record agy sessions (tool calls + AI output)
evaluator.ts score sessions against 5 metrics
testcases.json ground truth: questions + expected files/symbols
claude_answers.md Claude baseline (for comparison)
layer0-baseline-eval.ts Layer 0: Gemini without tools (control group)
layer1-tool-eval.ts Layer 1: deterministic tool recall/reachability
layer2-agent-eval.ts Layer 2: Gemini + tools end-to-end scoring
testcases.json ground truth: questions + expected files/symbols
config.ts paths and constants
AGENTS.md navigation rules (auto-loaded by agy)
.agents/ workspace MCP config (auto-loaded by agy)
Expand Down
Loading