Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .oxlintrc.effect.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"!packages/api/**",
"!packages/config/**",
"!packages/stack/**",
"!apps/cli-e2e/**",
"!apps/cli/**",
"!apps/docs/**",
"!packages/cli-test-helpers/**",
Expand Down
68 changes: 46 additions & 22 deletions apps/cli-e2e/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,20 +61,30 @@ the running tests exercise. The implications:
### `testBehaviour` — the default for all success and error tests

```typescript
import { testBehaviour } from "./test-context.ts";

testBehaviour("does the thing", async ({ run, projectRef, orgId, apiUrl, workspace }) => {
const result = await run(["command", `--remote=${projectRef}`]);
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain("Expected output");
});
import { Effect } from "effect";
import { runE2e, testBehaviour } from "./test-context.ts";

testBehaviour("does the thing", ({ run, projectRef, orgId, apiUrl, workspace, signal }) =>
runE2e(
Effect.gen(function* () {
const result = yield* run(["command", `--remote=${projectRef}`]);
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain("Expected output");
}),
signal,
),
);
```

The `workspace` fixture is what triggers scenario setup — if you destructure `workspace` or `run` (which depends on `workspace`), the scenario is automatically started before the test and cleaned up after.

Always return `runE2e(...)` from the body: Vitest does not run a returned Effect, so the harness fails a test that requests `run` or `runNoProjectId` and passes without executing it. A custom runner added with `testBehaviour.extend` gets the same check by wrapping its CLI Effect in `runGuard.track(...)`, as `stackRun` does. Pass the test's `signal` so a Vitest timeout interrupts the body.

This workspace is under the Effect lint (`.oxlintrc.effect.json`, warnings are errors), which rejects `async` test bodies, global `fetch`, `process.env` and `JSON.parse` in Effect code. Decode JSON stdout with `decodeJsonOutput(schema)(result.stdout)`, which reports a stdout excerpt on failure.

Available fixtures:

- `run(cmd)` — executes the CLI via the relay server
- `run(cmd)` — returns an Effect that executes the CLI via the relay server
- `projectRef` — 20-char project ref (real in record mode, default in replay)
- `orgId` — org slug (real in record mode, default in replay)
- `apiUrl` — the relay server base URL, for direct `/_ctrl/` calls
Expand All @@ -85,15 +95,23 @@ Available fixtures:
Inject errors via `/_ctrl/error-all` before the `run` call:

```typescript
testBehaviour("exits non-zero on 401", async ({ run, apiUrl }) => {
await fetch(`${apiUrl}/_ctrl/error-all`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ status: 401, body: { message: "Invalid token" } }),
});
const result = await run(["command", `--remote=${PROJECT_REF}`]);
expect(result.exitCode).not.toBe(0);
});
import { Effect } from "effect";
import { PROJECT_REF } from "./env.ts";
import { postControl, runE2e, testBehaviour } from "./test-context.ts";

testBehaviour("exits non-zero on 401", ({ run, apiUrl, signal }) =>
runE2e(
Effect.gen(function* () {
yield* postControl(`${apiUrl}/_ctrl/error-all`, {
status: 401,
body: { message: "Invalid token" },
});
const result = yield* run(["command", `--remote=${PROJECT_REF}`]);
expect(result.exitCode).not.toBe(0);
}),
signal,
),
);
```

Error injection tests can use `PROJECT_REF` directly (not `projectRef`) because no real API call reaches staging — the error is injected before the relay server checks the fixture. Their scenario fixture is always `[]` (empty).
Expand All @@ -103,15 +121,21 @@ Error injection tests can use `PROJECT_REF` directly (not `projectRef`) because
If a test depends on timing (e.g. deleting a project that was just created won't work because it's not yet provisioned), use `skipIf(isRecording)` and write a static fixture instead:

```typescript
import { Effect } from "effect";
import { isRecording } from "./env.ts";
import { runE2e, testBehaviour } from "./test-context.ts";

testBehaviour.skipIf(isRecording)(
"deletes project with --yes flag",
async ({ run, projectRef }) => {
const result = await run(["projects", "delete", projectRef, "--yes"]);
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain("Deleted project");
},
({ run, projectRef, signal }) =>
runE2e(
Effect.gen(function* () {
const result = yield* run(["projects", "delete", projectRef, "--yes"]);
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain("Deleted project");
}),
signal,
),
);
```

Expand Down
5 changes: 4 additions & 1 deletion apps/cli-e2e/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@
"types:check": "tsc --noEmit"
},
"dependencies": {
"@supabase/cli-test-helpers": "workspace:*"
"@effect/platform-bun": "catalog:",
"@effect/platform-node": "catalog:",
"@supabase/cli-test-helpers": "workspace:*",
"effect": "catalog:"
},
"devDependencies": {
"@tsconfig/bun": "catalog:",
Expand Down
69 changes: 44 additions & 25 deletions apps/cli-e2e/scripts/replay-serve.ts
Original file line number Diff line number Diff line change
@@ -1,36 +1,55 @@
import { Config, Console, Data, Effect, Fiber, Schema } from "effect";
import { startReplayServer } from "../src/server/replay-server.ts";

const FIXTURES_DIR = new URL("../fixtures", import.meta.url).pathname;

function parsePort(argv: readonly string[]): number {
class ReplayServeError extends Data.TaggedError("ReplayServeError")<{ readonly message: string }> {}

const parsePort = (argv: readonly string[]) => {
const index = argv.indexOf("--port");
if (index === -1) return 0;
if (index === -1) return Effect.succeed(0);
const port = Number(argv[index + 1]);
if (!Number.isInteger(port) || port < 0 || port > 65535) {
throw new Error(`--port expects an integer between 0 and 65535, got "${argv[index + 1]}"`);
return Effect.fail(
new ReplayServeError({
message: `--port expects an integer between 0 and 65535, got "${argv[index + 1]}"`,
}),
);
}
return port;
}

// Recording mode wipes the committed fixtures, so a standalone server never runs in it.
if (process.env["RECORD"] === "true") {
throw new Error("replay-serve does not support RECORD; unset it to serve the fixtures");
}

const server = await startReplayServer({
fixturesDir: FIXTURES_DIR,
hostname: "127.0.0.1",
port: parsePort(process.argv.slice(2)),
return Effect.succeed(port);
};

const stopSignal = Effect.callback<void>((resume) => {
const stop = () => resume(Effect.void);
process.on("SIGTERM", stop);
process.on("SIGINT", stop);
return Effect.sync(() => {
process.off("SIGTERM", stop);
process.off("SIGINT", stop);
});
});

let stopping = false;
const shutdown = async () => {
if (stopping) return;
stopping = true;
await server.stop();
process.exit(0);
};
process.on("SIGTERM", shutdown);
process.on("SIGINT", shutdown);
const main = Effect.gen(function* () {
// Recording mode wipes the committed fixtures, so a standalone server never runs in it.
if ((yield* Config.String("RECORD").pipe(Config.withDefault(""))) === "true") {
return yield* new ReplayServeError({
message: "replay-serve does not support RECORD; unset it to serve the fixtures",
});
}

const server = yield* startReplayServer({
fixturesDir: FIXTURES_DIR,
hostname: "127.0.0.1",
port: yield* parsePort(process.argv.slice(2)),
record: false,
stagingUrl: undefined,
});

const stopped = yield* stopSignal.pipe(Effect.forkChild({ startImmediately: true }));
yield* Console.log(
yield* Schema.encodeEffect(Schema.fromJsonString(Schema.Unknown))({ url: server.url }),
);
yield* Fiber.join(stopped);
});

console.log(JSON.stringify({ url: server.url }));
await Effect.runPromise(Effect.scoped(main));
133 changes: 86 additions & 47 deletions apps/cli-e2e/src/server/fixture-loader.ts
Original file line number Diff line number Diff line change
@@ -1,59 +1,86 @@
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { join } from "node:path";

export interface FixtureRequest {
method: string;
path: string;
query: Record<string, string>;
headers: Record<string, string>;
body: unknown;
}
import { Data, Effect, FileSystem, Formatter, Path, Schema } from "effect";

export interface FixtureResponse {
status: number;
headers: Record<string, string>;
body: unknown;
}
/* Hand-authored fixtures may omit headers (served as none) and the response body (served empty). */
const FixtureHeaders = Schema.Record(Schema.String, Schema.String).pipe(
Schema.withDecodingDefaultKey(Effect.sync(() => ({}))),
);

export interface FixtureEntry {
request: FixtureRequest;
response: FixtureResponse;
}
const FixtureRequest = Schema.Struct({
method: Schema.String,
path: Schema.mutableKey(Schema.String),
query: Schema.Record(Schema.String, Schema.String),
headers: FixtureHeaders,
body: Schema.Unknown,
});

const FixtureResponse = Schema.Struct({
status: Schema.Finite,
headers: FixtureHeaders,
body: Schema.optionalKey(Schema.Unknown),
});

const FixtureEntry = Schema.Struct({ request: FixtureRequest, response: FixtureResponse });

export type FixtureRequest = typeof FixtureRequest.Type;
export type FixtureResponse = typeof FixtureResponse.Type;
export type FixtureEntry = typeof FixtureEntry.Type;

/** All fixtures indexed by their key (e.g. "GET_v1_projects"), each holding
* an ordered queue of entries (for sequential calls to the same endpoint). */
export type FixtureStore = Map<string, FixtureEntry[]>;

/* Mirrors `existsSync`: any failure to reach the path counts as absent. */
export const pathExists = (fs: FileSystem.FileSystem, path: string) =>
fs.exists(path).pipe(Effect.orElseSucceed(() => false));

/** A fixture file is missing or does not hold valid JSON. */
class FixtureFileError extends Data.Error<{ readonly message: string; readonly cause?: unknown }> {}

/** Load an ordered scenario from scenarios/<name>/interactions.json.
* Returns null if the scenario file does not exist — caller decides whether
* to fail loudly or fall back to per-endpoint fixtures. */
export function loadScenario(scenariosDir: string, name: string): FixtureEntry[] | null {
const scenarioFile = join(scenariosDir, name, "interactions.json");
if (!existsSync(scenarioFile)) return null;
return parseFixtureFile<FixtureEntry[]>(scenarioFile);
}
export const loadScenario = Effect.fn("FixtureLoader.loadScenario")(function* (
scenariosDir: string,
name: string,
) {
const fs = yield* FileSystem.FileSystem;
const path = yield* Path.Path;
const scenarioFile = path.join(scenariosDir, name, "interactions.json");
if (!(yield* pathExists(fs, scenarioFile))) return null;
const entries: FixtureEntry[] = yield* parseFixtureFile(
scenarioFile,
Schema.mutable(Schema.Array(FixtureEntry)),
);
return entries;
});

/** Load all fixture pairs from the recorded/ directory into a FixtureStore.
* Fails fast with a descriptive error if any fixture file is malformed. */
export function loadFixtures(fixturesDir: string): FixtureStore {
const recordedDir = join(fixturesDir, "recorded");
export const loadFixtures = Effect.fn("FixtureLoader.loadFixtures")(function* (
fixturesDir: string,
) {
const fs = yield* FileSystem.FileSystem;
const path = yield* Path.Path;
const recordedDir = path.join(fixturesDir, "recorded");
const store: FixtureStore = new Map();

if (!existsSync(recordedDir)) {
if (!(yield* pathExists(fs, recordedDir))) {
return store;
}

const keys = readdirSync(recordedDir, { withFileTypes: true })
.filter((d) => d.isDirectory())
.map((d) => d.name);
const keys: string[] = [];
for (const name of yield* fs.readDirectory(recordedDir)) {
const info = yield* fs.stat(path.join(recordedDir, name));
if (info.type === "Directory") keys.push(name);
}

for (const key of keys) {
const keyDir = join(recordedDir, key);
const keyDir = path.join(recordedDir, key);
const entries: FixtureEntry[] = [];

// Collect numbered pairs: 1.request.json/1.response.json, 2.request.json, ...
// Also accept "default" as an alias for "1".
const files = readdirSync(keyDir).sort();
const files = (yield* fs.readDirectory(keyDir)).sort();
const indices = new Set<string>();

for (const file of files) {
Expand All @@ -62,11 +89,11 @@ export function loadFixtures(fixturesDir: string): FixtureStore {
}

for (const index of [...indices].sort(compareIndices)) {
const reqFile = join(keyDir, `${index}.request.json`);
const resFile = join(keyDir, `${index}.response.json`);
const reqFile = path.join(keyDir, `${index}.request.json`);
const resFile = path.join(keyDir, `${index}.response.json`);

const request = parseFixtureFile<FixtureRequest>(reqFile);
const response = parseFixtureFile<FixtureResponse>(resFile);
const request = yield* parseFixtureFile(reqFile, FixtureRequest);
const response = yield* parseFixtureFile(resFile, FixtureResponse);
entries.push({ request, response });
}

Expand All @@ -75,19 +102,31 @@ export function loadFixtures(fixturesDir: string): FixtureStore {
}
}

yield* Effect.annotateCurrentSpan("fixture.key_count", store.size);
return store;
}

function parseFixtureFile<T>(filePath: string): T {
if (!existsSync(filePath)) {
throw new Error(`Missing fixture file: ${filePath}`);
}
try {
return JSON.parse(readFileSync(filePath, "utf8")) as T;
} catch (cause) {
throw new Error(`Malformed fixture file: ${filePath}`, { cause });
});

const parseFixtureFile = Effect.fnUntraced(function* <S extends Schema.Constraint>(
filePath: string,
schema: S,
) {
const fs = yield* FileSystem.FileSystem;
if (!(yield* pathExists(fs, filePath))) {
return yield* new FixtureFileError({ message: `Missing fixture file: ${filePath}` });
}
}
const malformed = (cause: unknown) =>
new FixtureFileError({ message: `Malformed fixture file: ${filePath}`, cause });
const bytes = yield* fs.readFile(filePath).pipe(Effect.mapError(malformed));
const text = new TextDecoder("utf-8", { ignoreBOM: true }).decode(bytes);
return yield* Schema.decodeEffect(Schema.fromJsonString(schema))(text).pipe(
Effect.mapError(
(error) =>
new FixtureFileError({
message: `Malformed fixture file: ${filePath}\n ${error.message}: ${Formatter.format(text.slice(0, 200))}`,
}),
),
);
});

/** Sort indices so "default" comes first, then numerically. */
function compareIndices(a: string, b: string): number {
Expand Down
Loading
Loading