test(cli): temporary full benchmark dry run (do not merge) by Prashansa-K · Pull Request #7415 · supabase/cli

Prashansa-K · 2026-10-30T12:28:52Z

Do not merge. This is a temporary test pull request for #7412 and will be closed without merging. It replaces #7413.

It adds one commit on top of ci/cli-bench-stack that makes cli-bench-full.yml also run on a push to this branch, as if it were dispatched with workloads: all, runtime: both, and base: merge-base. This exercises the full benchmark end to end before merge, including its comments on this pull request. Dispatch, /bench full, and the schedule run only from develop, so they are not covered here.

🤖 Generated with Claude Code

The benchmark comment holds a Lightweight Commands and an Integration Heavy Commands section. Each writer replaces only its own section, verifies it landed, and retries when another writer overwrote it. Previous runs record the result each section replaced. /bench full dispatches the stack benchmark workflow on the pull request's branch and refreshes the lightweight section only when it is behind the head commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ner counts The benchmark harness runs workloads that need a local stack: untimed setup and teardown shared per run or per block of launches, a stack switch, a runtime, fixtures, a shared native artifact cache, per-workload run counts, and tracing limits. Reports add container operation counts, a runtime column, a section name, shard packing, and a report merge. Every launch disables product telemetry. The stack manifest and fixtures are generated and committed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Runs from the Actions UI, the /bench full command, and a weekly schedule against the latest stable release. Plans shards, builds both binaries once, measures each shard on one runner, and reports to the job summary and the pull request comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… outputs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep escaped table pipes through sanitizing, rebuild a comment another writer changed before writing it, keep each section's newest run and reallocate unused space when truncating, refresh a failed lightweight section on /bench full, skip launches when a setup step fails, and stage the migration fixture after setup for the migration workloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The lightweight and integration-heavy benchmarks each keep their own pull request comment, identified by its own hidden marker and carrying its own history and re-run note. This removes the shared-comment section markers, cross-section size budgeting and the read-back retry loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-30T12:51:24Z

CLI benchmark (Integration Heavy Commands): 5 things to check

Commit b38e45f against develop at 94511ff · 5-10 runs each · report only, never blocks merging

Command	Runtime	Before	After	Change
`supabase start`	docker	19087 ms	18729 ms	no real change
`supabase stop --no-backup`	docker	4616 ms	4654 ms	no real change
`supabase db start`	docker	10764 ms	10853 ms	no real change
`supabase status`	docker	236 ms	405 ms	slower by 169 ms (+71%)
`supabase db query --local select 1`	docker	215 ms	250 ms	slower by 35 ms (+17%)
`supabase db reset`	docker	17512 ms	16790 ms	no real change
`supabase test db`	docker	657 ms	697 ms	no real change (+6%, within normal variation)
`supabase gen types --lang typescript --local`	docker	233 ms	231 ms	no real change
`supabase gen types --lang typescript --local`	docker	294 ms	282 ms	no real change
`supabase gen types --lang typescript --local`	docker	272 ms	293 ms	no real change (+8%, within normal variation)
`supabase db diff --use-pg-delta`	docker	7065 ms	7837 ms	no real change (+11%, within normal variation)
`supabase db diff --use-pg-delta`	docker	1723 ms	2008 ms	no real change (+17%, within normal variation)
`supabase db dump --local`	docker	609 ms	552 ms	no real change (-9%, within normal variation)
`supabase db lint --local`	docker	298 ms	230 ms	faster by 68 ms (-23%)
`supabase migration up --local`	docker	312 ms	324 ms	no real change
`supabase db push --local`	docker	313 ms	289 ms	no real change (-8%, within normal variation)
`supabase migration list --local`	docker	370 ms	340 ms	no real change (-8%, within normal variation)
`supabase stack start --runtime docker --eager`	stack, docker	12151 ms	12322 ms	no real change
`supabase stack status`	stack, docker	247 ms	337 ms	slower by 89 ms (+36%)
`supabase stack stop`	stack, docker	7375 ms	8144 ms	no real change (+10%, within normal variation)
`supabase db query --local select 1`	stack, docker	463 ms	453 ms	no real change
`supabase db reset`	stack, docker	16813 ms	18475 ms	no real change (+10%, within normal variation)
`supabase stack start --runtime native --eager`	stack, native	6090 ms	5703 ms	no real change (-6%, within normal variation)
`supabase stack status`	stack, native	246 ms	242 ms	no real change
`supabase stack stop`	stack, native	3267 ms	3247 ms	no real change
`supabase db query --local select 1`	stack, native	270 ms	227 ms	faster by 43 ms (-16%)
`supabase db reset`	stack, native	10102 ms	10039 ms	no real change

API calls: none, before or after, for all 27 commands.

Container operations: unchanged for all 17 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, container operations, slowest phases

Timing ranges

Command	Before range	After range	Change	Verdict
`supabase start`	16552-19687 ms	16261-20854 ms	-357 ms (-2%)	within noise
`supabase stop --no-backup`	4457-4754 ms	4627-4713 ms	+38 ms (+1%)	within noise
`supabase db start`	10571-11153 ms	10572-11186 ms	+89 ms (+1%)	within noise
`supabase status`	197-276 ms	222-729 ms	+169 ms (+71%)	slower
`supabase db query --local select 1`	203-238 ms	216-255 ms	+35 ms (+17%)	slower
`supabase db reset`	16019-19196 ms	16130-19869 ms	-721 ms (-4%)	within noise
`supabase test db`	579-752 ms	635-734 ms	+41 ms (+6%)	within noise
`supabase gen types --lang typescript --local`	212-250 ms	218-248 ms	-2 ms (-1%)	within noise
`supabase gen types --lang typescript --local`	254-352 ms	261-306 ms	-11 ms (-4%)	within noise
`supabase gen types --lang typescript --local`	256-309 ms	264-356 ms	+21 ms (+8%)	within noise
`supabase db diff --use-pg-delta`	5667-9571 ms	7243-12334 ms	+772 ms (+11%)	within noise
`supabase db diff --use-pg-delta`	1459-3677 ms	1382-2297 ms	+286 ms (+17%)	within noise
`supabase db dump --local`	421-823 ms	493-846 ms	-58 ms (-9%)	within noise
`supabase db lint --local`	240-427 ms	201-273 ms	-68 ms (-23%)	faster
`supabase migration up --local`	277-365 ms	286-355 ms	+12 ms (+4%)	within noise
`supabase db push --local`	218-337 ms	257-379 ms	-24 ms (-8%)	within noise
`supabase migration list --local`	277-476 ms	277-435 ms	-31 ms (-8%)	within noise
`supabase stack start --runtime docker --eager`	7316-13241 ms	8618-13724 ms	+171 ms (+1%)	within noise
`supabase stack status`	211-312 ms	271-449 ms	+89 ms (+36%)	slower
`supabase stack stop`	6861-8204 ms	7649-9161 ms	+769 ms (+10%)	within noise
`supabase db query --local select 1`	403-596 ms	397-597 ms	-10 ms (-2%)	within noise
`supabase db reset`	14279-19533 ms	15543-22195 ms	+1662 ms (+10%)	within noise
`supabase stack start --runtime native --eager`	5685-6435 ms	5594-5881 ms	-387 ms (-6%)	within noise
`supabase stack status`	216-286 ms	217-279 ms	-4 ms (-2%)	within noise
`supabase stack stop`	3191-3372 ms	3226-3469 ms	-20 ms (-1%)	within noise
`supabase db query --local select 1`	253-313 ms	217-266 ms	-43 ms (-16%)	faster
`supabase db reset`	8678-10579 ms	9692-14247 ms	-64 ms (-1%)	within noise

API calls per command

Command	Before	After	Endpoint changes
`supabase start`	0	0
`supabase stop --no-backup`	0	0
`supabase db start`	0	0
`supabase status`	0	0
`supabase db query --local select 1`	0	0
`supabase db reset`	0	0
`supabase test db`	0	0
`supabase gen types --lang typescript --local`	0	0
`supabase gen types --lang typescript --local`	0	0
`supabase gen types --lang typescript --local`	0	0
`supabase db diff --use-pg-delta`	0	0
`supabase db diff --use-pg-delta`	0	0
`supabase db dump --local`	0	0
`supabase db lint --local`	0	0
`supabase migration up --local`	0	0
`supabase db push --local`	0	0
`supabase migration list --local`	0	0
`supabase stack start --runtime docker --eager`	0	0
`supabase stack status`	0	0
`supabase stack stop`	0	0
`supabase db query --local select 1`	0	0
`supabase db reset`	0	0
`supabase stack start --runtime native --eager`	0	0
`supabase stack status`	0	0
`supabase stack stop`	0	0
`supabase db query --local select 1`	0	0
`supabase db reset`	0	0

Container operations per command

Command	Runtime	Operation	Before	After
`supabase start`	docker	`cp`	3	3
`supabase start`	docker	`create`	12	12
`supabase start`	docker	`image inspect`	17	17
`supabase start`	docker	`network create`	1	1
`supabase start`	docker	`network inspect`	2	2
`supabase start`	docker	`run`	3	3
`supabase start`	docker	`start`	12	12
`supabase start`	docker	`volume create`	3	3
`supabase start`	docker	`volume inspect`	1	1
`supabase stop --no-backup`	docker	`container prune`	1	1
`supabase stop --no-backup`	docker	`network prune`	1	1
`supabase stop --no-backup`	docker	`ps`	1	1
`supabase stop --no-backup`	docker	`stop`	12	12
`supabase stop --no-backup`	docker	`volume ls`	1	1
`supabase stop --no-backup`	docker	`volume prune`	1	1
`supabase db start`	docker	`cp`	1	1
`supabase db start`	docker	`create`	1	1
`supabase db start`	docker	`image inspect`	4	4
`supabase db start`	docker	`network create`	1	1
`supabase db start`	docker	`network inspect`	1	1
`supabase db start`	docker	`run`	3	3
`supabase db start`	docker	`start`	1	1
`supabase db start`	docker	`volume create`	1	1
`supabase db start`	docker	`volume inspect`	1	1
`supabase status`	docker	`ps`	1	1
`supabase db reset`	docker	`container rm`	1	1
`supabase db reset`	docker	`cp`	1	1
`supabase db reset`	docker	`create`	1	1
`supabase db reset`	docker	`exec`	1	1
`supabase db reset`	docker	`image inspect`	4	4
`supabase db reset`	docker	`network inspect`	1	1
`supabase db reset`	docker	`restart`	4	4
`supabase db reset`	docker	`run`	3	3
`supabase db reset`	docker	`start`	1	1
`supabase db reset`	docker	`stop`	4	4
`supabase db reset`	docker	`volume create`	1	1
`supabase db reset`	docker	`volume rm`	1	1
`supabase test db`	docker	`image inspect`	1	1
`supabase test db`	docker	`run`	1	1
`supabase gen types --lang typescript --local`	docker	`container inspect`	1	1
`supabase gen types --lang typescript --local`	docker	`container inspect`	1	1
`supabase gen types --lang typescript --local`	docker	`container inspect`	1	1
`supabase db diff --use-pg-delta`	docker	`cp`	3	3
`supabase db diff --use-pg-delta`	docker	`create`	1	1
`supabase db diff --use-pg-delta`	docker	`image inspect`	4	4
`supabase db diff --use-pg-delta`	docker	`network inspect`	1	1
`supabase db diff --use-pg-delta`	docker	`rm`	1	1
`supabase db diff --use-pg-delta`	docker	`run`	3	3
`supabase db diff --use-pg-delta`	docker	`start`	2	2
`supabase db diff --use-pg-delta`	docker	`stop`	1	1
`supabase db diff --use-pg-delta`	docker	`cp`	2	2
`supabase db diff --use-pg-delta`	docker	`create`	1	1
`supabase db diff --use-pg-delta`	docker	`image inspect`	1	1
`supabase db diff --use-pg-delta`	docker	`network inspect`	1	1
`supabase db diff --use-pg-delta`	docker	`rm`	1	1
`supabase db diff --use-pg-delta`	docker	`start`	1	1
`supabase db dump --local`	docker	`image inspect`	1	1
`supabase db dump --local`	docker	`run`	1	1

Slowest phases

Command	Span	Change	Before -> after
`supabase status`	`cli.run`	+12 ms	53 -> 65 ms
`supabase status`	`command.status`	+10 ms	36 -> 47 ms
`supabase status`	`status`	+10 ms	35 -> 46 ms
`supabase db query --local select 1`	`DbConfig.resolveConnection`	+8 ms	18 -> 26 ms
`supabase db query --local select 1`	`CliConfigValues.load`	+7 ms	14 -> 21 ms
`supabase db query --local select 1`	`db.query.runLocal`	-5 ms	18 -> 13 ms
`supabase db lint --local`	`cli.run`	-14 ms	105 -> 92 ms
`supabase db lint --local`	`command.db.lint`	-11 ms	77 -> 65 ms
`supabase db lint --local`	`db.lint`	-11 ms	75 -> 64 ms
`supabase stack status`	`cli.run`	+23 ms	66 -> 89 ms
`supabase stack status`	`command.stack.status`	+19 ms	49 -> 68 ms
`supabase stack status`	`experimental.stack.status`	+19 ms	48 -> 67 ms
`supabase db query --local select 1`	`db.query`	-6 ms	55 -> 49 ms
`supabase db query --local select 1`	`command.db.query`	-5 ms	56 -> 50 ms
`supabase db query --local select 1`	`DbConfig.resolveConnection`	-5 ms	40 -> 36 ms

Workflow run

Comment /bench full to run this benchmark again on the latest commit, which also refreshes the lightweight comment if it is behind. /bench runs only the lightweight benchmark.

Previous runs

769b1f8 · 2026-10-30 14:25 UTC · 5 things to check · run
84f81c8 · 2026-10-30 13:06 UTC · 2 things to check · run

github-actions · 2026-10-30T12:54:09Z

CLI benchmark (Lightweight Commands): no changes found

Commit b38e45f against develop at 94511ff · 10 runs each · report only, never blocks merging

Command	Before	After	Change
`supabase --version`	168 ms	172 ms	no real change
`supabase --help`	188 ms	188 ms	no real change
`supabase init`	173 ms	171 ms	no real change
`supabase migration new bench`	188 ms	179 ms	no real change (-5%, within normal variation)
`supabase gen types --lang typescript --project-id replayfixtureprojref`	216 ms	206 ms	no real change
`supabase projects list`	211 ms	202 ms	no real change
`supabase functions list --project-ref replayfixtureprojref`	220 ms	226 ms	no real change

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command	Before range	After range	Change	Verdict
`supabase --version`	157-214 ms	156-229 ms	+4 ms (+3%)	within noise
`supabase --help`	161-236 ms	158-220 ms	0 ms (0%)	within noise
`supabase init`	160-202 ms	168-185 ms	-2 ms (-1%)	within noise
`supabase migration new bench`	159-249 ms	163-236 ms	-9 ms (-5%)	within noise
`supabase gen types --lang typescript --project-id replayfixtureprojref`	185-240 ms	190-257 ms	-10 ms (-5%)	within noise
`supabase projects list`	170-258 ms	169-242 ms	-9 ms (-4%)	within noise
`supabase functions list --project-ref replayfixtureprojref`	200-258 ms	190-265 ms	+6 ms (+3%)	within noise

API calls per command

Command	Before	After
`supabase --version`	0	0
`supabase --help`	0	0
`supabase init`	0	0
`supabase migration new bench`	0	0
`supabase gen types --lang typescript --project-id replayfixtureprojref`	2	2
`supabase projects list`	1	1
`supabase functions list --project-ref replayfixtureprojref`	2	2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

Previous runs

769b1f8 · 2026-10-30 14:11 UTC · no changes found · run
84f81c8 · 2026-10-30 12:54 UTC · no changes found · run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prashansa-K and others added 11 commits October 30, 2026 15:33

docs(cli): document the stack benchmark

339eb13

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ci(cli): harden benchmark comment sanitizing and dispatch ref

1abdfce

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore: merge develop

7179549

ci(cli): rename the stack benchmark tier to full and drop unused plan…

9b05727

… outputs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore(cli): format benchmark files

691bf23

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(cli): temporarily run the full benchmark on push (do not merge)

b1be316

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prashansa-K added the do not merge Approve to apply; do not merge. label Oct 30, 2026

test(cli): start a fresh full benchmark run (do not merge)

84f81c8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prashansa-K and others added 6 commits October 30, 2026 19:37

ci(cli): label the lightweight benchmark comment

8c858cd

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore: merge ci/cli-bench-stack

769b1f8

ci(cli): count API calls in the full benchmark

c1855b4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore: merge ci/cli-bench-stack

d53447b

chore: merge develop

16b0c34

chore: merge ci/cli-bench-stack

b38e45f

Prashansa-K mentioned this pull request Oct 30, 2026

ci(cli): add a full benchmark for integration-heavy commands #7412

Open

Navigation Menu

Repository navigation

test(cli): temporary full benchmark dry run (do not merge) - #7415

test(cli): temporary full benchmark dry run (do not merge)#7415
Prashansa-K wants to merge 18 commits into
developfrom
ci/cli-bench-full-test

Prashansa-K commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Timing ranges

API calls per command

Container operations per command

Slowest phases

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Timing ranges

API calls per command

Slowest phases

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Prashansa-K commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI benchmark (Integration Heavy Commands): 5 things to check

Timing ranges

API calls per command

Container operations per command

Slowest phases

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI benchmark (Lightweight Commands): no changes found

Timing ranges

API calls per command

Slowest phases

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading