Skip to content

feat: add regression eval for finding Edge Function console output #232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
claude wants to merge 14 commits into
base: main
Choose a base branch
from
Open

feat: add regression eval for finding Edge Function console output #232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b6e5f60
feat: add regression eval for finding Edge Function console output
claude Aug 24, 2026
aac44c2
chore: refresh eval results
github-actions[bot] Aug 24, 2026
7199bc6
fix: score the console-output eval from real query_logs calls
claude Aug 25, 2026
4900b09
chore: refresh eval results
github-actions[bot] Aug 25, 2026
302bb6c
fix: gate the console-output eval on query_logs result evidence
claude Aug 25, 2026
4e047d4
chore: refresh eval results
github-actions[bot] Aug 25, 2026
4c76b92
fix: harden console log result evidence
barryroodt Aug 25, 2026
782f302
chore: refresh eval results
github-actions[bot] Aug 25, 2026
6146668
fix: read log rows through the untrusted-data boundary
claude Aug 25, 2026
6da2e05
docs: fix stale helper name in row-extractor comment
claude Aug 25, 2026
00c17bf
chore: refresh eval results
github-actions[bot] Aug 25, 2026
9ec8ba8
chore: refresh eval results
github-actions[bot] Aug 25, 2026
432161c
Merge main into feat/eval-edge-function-console-output
claude Aug 26, 2026
a3316e0
chore: refresh eval results
github-actions[bot] Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/framework/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@
"version": "0.0.1",
"type": "module",
"scripts": {
"check": "pnpm typecheck && pnpm test:framework && pnpm test:vercel-runner",
"check": "pnpm typecheck && pnpm test:framework && pnpm test:vercel-runner && pnpm test:eval-scorers",
"eval": "node --env-file=../../.env --import tsx/esm harness/run-eval.ts",
"eval:dry": "node --env-file=../../.env --import tsx/esm harness/run-eval.ts --dry",
"eval:smoke": "node --env-file=../../.env --import tsx/esm harness/run-eval.ts --smoke",
"eval:vercel": "node --env-file=../../.env --import tsx/esm scripts/run-vercel-evals.ts",
"typecheck": "tsc --noEmit",
"test:framework": "node --env-file-if-exists=../../.env --import tsx/esm scripts/smoke-framework.ts",
"test:vercel-runner": "vitest run scripts/run-vercel-evals.test.ts lib/cli-args.test.ts lib/sample-sets.test.ts",
"test:eval-scorers": "vitest run --root ../.. evals/investigate-functions-002-edge-function-console-output/result-evidence.test.ts",
"export-results": "node --import tsx/esm scripts/export-results.ts",
"demo:mcp": "node --env-file=../../.env --import tsx/esm scripts/mcp-demo.ts",
"demo:executor": "node --env-file=../../.env --import tsx/esm scripts/executor-demo.ts"
Expand Down
328 changes: 328 additions & 0 deletions apps/web/src/data/regression-eval-results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -620,6 +620,153 @@
"attempts": 1,
"sourcePath": "claude-code-sonnet-5/investigate-functions-001-546-resource-limit.json"
},
{
"experiment": "claude-code-sonnet-5",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "Reported actual checkout-quote console output, including the expired SPRING24 coupon being dropped and pricing-gateway timing out after retries with tax falling back to zero."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 8 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [
"supabase",
"supabase-postgres-best-practices"
],
"loaded": [
"supabase"
]
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"run": 1,
"sourcePath": "claude-code-sonnet-5/investigate-functions-002-edge-function-console-output/run-1/result.json"
},
{
"experiment": "claude-code-sonnet-5",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "It quoted function console logs showing SPRING24 was expired and dropped, and the pricing-gateway timed out after retries with tax falling back to zero."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 7 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [
"supabase",
"supabase-postgres-best-practices"
],
"loaded": [
"supabase"
]
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"run": 2,
"sourcePath": "claude-code-sonnet-5/investigate-functions-002-edge-function-console-output/run-2/result.json"
},
{
"experiment": "claude-code-sonnet-5",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "Reported specific checkout-quote console output: SPRING24 was expired and dropped, and pricing-gateway timed out after retries with tax falling back to zero."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 6 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [
"supabase",
"supabase-postgres-best-practices"
],
"loaded": [
"supabase"
]
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"run": 3,
"sourcePath": "claude-code-sonnet-5/investigate-functions-002-edge-function-console-output/run-3/result.json"
},
{
"experiment": "claude-code-sonnet-5",
"experimentSuite": "regression",
Expand Down Expand Up @@ -1244,6 +1391,55 @@
"attempts": 1,
"sourcePath": "claude-code-sonnet-5/resolve-storage-001-upsert-missing-update-policy.json"
},
{
"experiment": "claude-code-sonnet-5-mcp-0-11",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "Reported actual checkout-quote console output, including expired SPRING24 discount removal and pricing-gateway timeout with zero-tax fallback."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 5 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [
"supabase",
"supabase-postgres-best-practices"
],
"loaded": [
"supabase"
]
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"attempts": 1,
"sourcePath": "claude-code-sonnet-5-mcp-0-11/investigate-functions-002-edge-function-console-output.json"
},
{
"experiment": "claude-code-sonnet-5-no-skills",
"experimentSuite": "regression",
Expand Down Expand Up @@ -1711,6 +1907,138 @@
"attempts": 1,
"sourcePath": "claude-code-sonnet-5-no-skills/investigate-functions-001-546-resource-limit.json"
},
{
"experiment": "claude-code-sonnet-5-no-skills",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "Reported specific console output: SPRING24 was expired and dropped, and pricing-gateway timed out after retries with tax falling back to zero."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 7 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [],
"loaded": []
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"run": 1,
"sourcePath": "claude-code-sonnet-5-no-skills/investigate-functions-002-edge-function-console-output/run-1/result.json"
},
{
"experiment": "claude-code-sonnet-5-no-skills",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "Reported console output showing SPRING24 expired and was dropped, and pricing-gateway timed out after retries with tax falling back to zero."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 7 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [],
"loaded": []
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"run": 2,
"sourcePath": "claude-code-sonnet-5-no-skills/investigate-functions-002-edge-function-console-output/run-2/result.json"
},
{
"experiment": "claude-code-sonnet-5-no-skills",
"experimentSuite": "regression",
"experimentDisplay": {
"agent": "claude-code",
"modelProvider": "anthropic",
"modelId": "claude-sonnet-5",
"reasoningEffort": "high"
},
"eval": "investigate-functions-002-edge-function-console-output",
"stage": "investigate",
"product": [
"edge-functions"
],
"topic": [
"observability"
],
"suite": "regression",
"interface": "mcp",
"passed": true,
"checks": [
{
"name": "surfaced the function console output",
"passed": true,
"judgeNotes": "It quoted console output showing SPRING24 was expired and dropped, and the pricing-gateway timed out after retries with tax falling back to zero."
},
{
"name": "read the function console output from the logs",
"passed": true,
"notes": "ran a source-discovery query (select distinct source) before reading logs; 5 query_logs call(s); console-row markers returned: pricing-gateway, timed out after 3 retries, cart_8f21ac, SPRING24"
}
],
"skills": {
"available": [],
"loaded": []
},
"docs": {
"calls": []
},
"prompt": "I pushed a new version of our `checkout-quote` edge function this morning and it has a bunch of `console.log` lines in it, but I can't find any of that output anywhere in the dashboard.\n\nCan you dig the output out of the project logs and tell me what the function is actually printing?",
"promptSourcePath": "evals/investigate-functions-002-edge-function-console-output/PROMPT.md",
"run": 3,
"sourcePath": "claude-code-sonnet-5-no-skills/investigate-functions-002-edge-function-console-output/run-3/result.json"
},
{
"experiment": "claude-code-sonnet-5-no-skills",
"experimentSuite": "regression",
Expand Down
Loading