feat: add @supabase/middleware regression eval - #273
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
mandarini
added
the
run-evals-changed
mandarini
removed
the
run-evals-changed
barryroodt
left a comment
barryroodt
left a comment
There was a problem hiding this comment.
One scorer issue to fix before merge: the prompt permits defining the API-key middleware in a helper module, but the index-only source check rejects that valid layout. This fails the whole eval even when all HTTP checks pass. Could we inspect imported local modules before rejecting the implementation?
| const src = await ctx.readFile(path).catch(() => ''); | ||
| const importsPackage = | ||
| /(?:from|import)\s*\(?\s*['"](?:npm:|jsr:)?@supabase\/middleware(?:@[^'"/]+)?(?:\/[^'"]*)?['"]/.test( | ||
| src | ||
| ); | ||
| const definesMiddleware = /\bdefineMiddleware\s*[<(]/.test(src); |
There was a problem hiding this comment.
A valid helper-module implementation fails this gate. If notes-api/index.ts imports pipeline from @supabase/middleware and composes withApiKey exported by ../_shared/api-key.ts, the defineMiddleware call lives in the helper. The prompt permits this, but sourceChecks checks only the entrypoint, so the whole eval fails even when all HTTP checks pass. Could we inspect the imported local modules too?
2 participants
Adds a regression eval for
@supabase/middleware(https://github.com/supabase/middleware), the sibling of the@supabase/serverpair: the prompt names the package and asks for an Edge Function that composes the built-in CORS middleware with a hand-writtendefineMiddlewareAPI-key gate. The scorer drives the key gate through the local gateway (no key and wrong key both401, right key200with the key id read fromctx), checksVary: Originas the CORS signal, and gates on the source importing the package and callingdefineMiddleware.Two details worth knowing: the local Kong answers preflights and rewrites
Access-Control-Allow-Originto*on/functions/v1/*, soVary: Originis the only CORS behavior observable through it; and the seededfunctions/.envneeded a.gitignoreexception, following the existing one. Refreshed results are included: 6 of 6 runs pass (three with skills, three without), the expected baseline for a usability canary.