Skip to content

test(cli): temporary stack benchmark dry run (do not merge) - #7413

Closed
Prashansa-K wants to merge 7 commits into
developfrom
ci/cli-bench-stack-test
Closed

test(cli): temporary stack benchmark dry run (do not merge)#7413
Prashansa-K wants to merge 7 commits into
developfrom
ci/cli-bench-stack-test

Conversation

@Prashansa-K

@Prashansa-K Prashansa-K commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Do not merge. This is a temporary test pull request for #7412 and will be closed without merging.

It adds one commit on top of ci/cli-bench-stack that makes cli-bench-stack.yml also run on a push to this branch, as if it were dispatched with workloads: all, runtime: both, and base: merge-base. This exercises the stack benchmark end to end before merge, including the comment on this pull request. Dispatch, /bench full, and the schedule run only from develop, so they are not covered here.

🤖 Generated with Claude Code

Prashansa-K and others added 7 commits October 30, 2026 15:33
@Prashansa-K @claude
ci(cli): add benchmark comment sections and the /bench full command
ad897e4 
The benchmark comment holds a Lightweight Commands and an Integration Heavy Commands section.
Each writer replaces only its own section, verifies it landed, and retries when another writer
overwrote it. Previous runs record the result each section replaced. /bench full dispatches the
stack benchmark workflow on the pull request's branch and refreshes the lightweight section only
when it is behind the head commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K @claude
feat(cli): benchmark stack workloads with setup, teardown, and contai…
88db0a9 
…ner counts

The benchmark harness runs workloads that need a local stack: untimed setup and teardown shared
per run or per block of launches, a stack switch, a runtime, fixtures, a shared native artifact
cache, per-workload run counts, and tracing limits. Reports add container operation counts, a
runtime column, a section name, shard packing, and a report merge. Every launch disables product
telemetry. The stack manifest and fixtures are generated and committed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K @claude
ci(cli): add the stack benchmark workflow
6eed9c2 
Runs from the Actions UI, the /bench full command, and a weekly schedule against the latest
stable release. Plans shards, builds both binaries once, measures each shard on one runner, and
reports to the job summary and the pull request comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K @claude
docs(cli): document the stack benchmark
339eb13 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K @claude
ci(cli): harden benchmark comment sanitizing and dispatch ref
1abdfce 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K
chore: merge develop
7179549
@Prashansa-K @claude
test(cli): temporarily run the stack benchmark on push (do not merge)
75d5d27 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K Prashansa-K added the do not merge Approve to apply; do not merge. label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

CLI benchmark

Lightweight Commands: no changes found

Commit 75d5d27 against develop at db5ee12 · 10 runs each · report only, never blocks merging

Command Before After Change
supabase --version 164 ms 178 ms no real change (+8%, within normal variation)
supabase --help 210 ms 214 ms no real change
supabase init 199 ms 222 ms no real change (+12%, within normal variation)
supabase migration new bench 183 ms 184 ms no real change
supabase gen types --lang typescript --project-id replayfixtureprojref 207 ms 209 ms no real change
supabase projects list 184 ms 184 ms no real change
supabase functions list --project-ref replayfixtureprojref 180 ms 186 ms no real change

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Before range After range Change Verdict
supabase --version 150-227 ms 149-239 ms +13 ms (+8%) within noise
supabase --help 155-247 ms 172-263 ms +4 ms (+2%) within noise
supabase init 165-233 ms 169-265 ms +23 ms (+12%) within noise
supabase migration new bench 159-223 ms 165-266 ms +1 ms (+1%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref 186-249 ms 190-223 ms +2 ms (+1%) within noise
supabase projects list 176-229 ms 172-206 ms 0 ms (0%) within noise
supabase functions list --project-ref replayfixtureprojref 172-210 ms 171-217 ms +6 ms (+3%) within noise

API calls per command

Command Before After Endpoint changes
supabase --version 0 0
supabase --help 0 0
supabase init 0 0
supabase migration new bench 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref 2 2
supabase projects list 1 1
supabase functions list --project-ref replayfixtureprojref 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Integration Heavy Commands: 4 things to check

Commit 75d5d27 against develop at db5ee12 · 5-10 runs each · report only, never blocks merging

⚠️ supabase gen types --lang typescript --local exited with an error before; its timing is not compared.

⚠️ supabase gen types --lang typescript --local exited with an error after; its timing is not compared.

⚠️ supabase db push --local exits with different codes before and after, so its timing is not compared.

⚠️ supabase db push --local exited with an error before; its timing is not compared.

Command Runtime Before After Change
supabase start docker 16875 ms 16598 ms no real change
supabase stop --no-backup docker 4570 ms 4528 ms no real change
supabase db start docker 10537 ms 10952 ms no real change
supabase status docker 259 ms 206 ms no real change (-21%, within normal variation)
supabase db query --local select 1 docker 210 ms 236 ms no real change (+12%, within normal variation)
supabase db reset docker 16687 ms 17337 ms no real change
supabase test db docker 568 ms 640 ms no real change (+13%, within normal variation)
supabase gen types --lang typescript --local docker 211 ms 219 ms not compared
supabase gen types --lang typescript --local docker 270 ms 292 ms no real change (+8%, within normal variation)
supabase gen types --lang typescript --local docker 286 ms 284 ms no real change
supabase db diff --use-pg-delta docker 6238 ms 6226 ms no real change
supabase db diff --use-pg-delta docker 1573 ms 1763 ms no real change (+12%, within normal variation)
supabase db dump --local docker 395 ms 429 ms slower by 33 ms (+8%)
supabase db lint --local docker 257 ms 255 ms no real change
supabase migration up --local docker 403 ms 344 ms no real change (-15%, within normal variation)
supabase db push --local docker 254 ms 240 ms not compared
supabase migration list --local docker 280 ms 226 ms no real change (-19%, within normal variation)
supabase stack start --runtime docker --eager stack, docker 8325 ms 8273 ms no real change
supabase stack status stack, docker 224 ms 214 ms no real change
supabase stack stop stack, docker 7752 ms 8173 ms no real change (+5%, within normal variation)
supabase db query --local select 1 stack, docker 278 ms 267 ms no real change
supabase db reset stack, docker 15112 ms 16807 ms no real change (+11%, within normal variation)
supabase stack start --runtime native --eager stack, native 6229 ms 6847 ms slower by 618 ms (+10%)
supabase stack status stack, native 210 ms 211 ms no real change
supabase stack stop stack, native 3343 ms 3343 ms no real change
supabase db query --local select 1 stack, native 279 ms 252 ms no real change (-10%, within normal variation)
supabase db reset stack, native 9740 ms 9198 ms no real change (-6%, within normal variation)

API calls were not counted.

Container operations: unchanged for all 17 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, container operations, slowest phases

Timing ranges

Command Before range After range Change Verdict
supabase start 16491-17538 ms 16449-16999 ms -276 ms (-2%) within noise
supabase stop --no-backup 4527-4700 ms 4501-4630 ms -42 ms (-1%) within noise
supabase db start 10430-11461 ms 10668-11324 ms +415 ms (+4%) within noise
supabase status 206-469 ms 198-217 ms -53 ms (-21%) within noise
supabase db query --local select 1 191-277 ms 208-262 ms +26 ms (+12%) within noise
supabase db reset 16405-21027 ms 16528-18480 ms +650 ms (+4%) within noise
supabase test db 554-677 ms 582-676 ms +71 ms (+13%) within noise
supabase gen types --lang typescript --local 175-254 ms 197-249 ms +8 ms (+4%) not comparable
supabase gen types --lang typescript --local 238-363 ms 241-331 ms +22 ms (+8%) within noise
supabase gen types --lang typescript --local 264-509 ms 272-305 ms -2 ms (-1%) within noise
supabase db diff --use-pg-delta 5551-6400 ms 5647-8026 ms -11 ms (0%) within noise
supabase db diff --use-pg-delta 1498-2132 ms 1620-2472 ms +190 ms (+12%) within noise
supabase db dump --local 345-424 ms 386-462 ms +33 ms (+8%) slower
supabase db lint --local 228-275 ms 234-281 ms -2 ms (-1%) within noise
supabase migration up --local 251-607 ms 224-586 ms -59 ms (-15%) within noise
supabase db push --local 209-345 ms 228-340 ms -14 ms (-5%) not comparable
supabase migration list --local 206-479 ms 208-261 ms -54 ms (-19%) within noise
supabase stack start --runtime docker --eager 8071-8668 ms 8001-8835 ms -52 ms (-1%) within noise
supabase stack status 195-296 ms 193-251 ms -10 ms (-5%) within noise
supabase stack stop 7651-8331 ms 7463-8728 ms +421 ms (+5%) within noise
supabase db query --local select 1 225-308 ms 253-300 ms -10 ms (-4%) within noise
supabase db reset 14045-19279 ms 14162-21090 ms +1695 ms (+11%) within noise
supabase stack start --runtime native --eager 6054-6464 ms 6289-7631 ms +618 ms (+10%) slower
supabase stack status 199-235 ms 198-266 ms +2 ms (+1%) within noise
supabase stack stop 3220-3734 ms 3209-3407 ms +1 ms (+0%) within noise
supabase db query --local select 1 216-336 ms 216-301 ms -27 ms (-10%) within noise
supabase db reset 8656-9944 ms 8542-9943 ms -541 ms (-6%) within noise

API calls per command

API calls were not counted.

Container operations per command

Command Runtime Operation Before After
supabase start docker cp 3 3
supabase start docker create 12 12
supabase start docker image inspect 17 17
supabase start docker network create 1 1
supabase start docker network inspect 2 2
supabase start docker run 3 3
supabase start docker start 12 12
supabase start docker volume create 3 3
supabase start docker volume inspect 1 1
supabase stop --no-backup docker container prune 1 1
supabase stop --no-backup docker network prune 1 1
supabase stop --no-backup docker ps 1 1
supabase stop --no-backup docker stop 12 12
supabase stop --no-backup docker volume ls 1 1
supabase stop --no-backup docker volume prune 1 1
supabase db start docker cp 1 1
supabase db start docker create 1 1
supabase db start docker image inspect 4 4
supabase db start docker network create 1 1
supabase db start docker network inspect 1 1
supabase db start docker run 3 3
supabase db start docker start 1 1
supabase db start docker volume create 1 1
supabase db start docker volume inspect 1 1
supabase status docker ps 1 1
supabase db reset docker container rm 1 1
supabase db reset docker cp 1 1
supabase db reset docker create 1 1
supabase db reset docker exec 1 1
supabase db reset docker image inspect 4 4
supabase db reset docker network inspect 1 1
supabase db reset docker restart 4 4
supabase db reset docker run 3 3
supabase db reset docker start 1 1
supabase db reset docker stop 4 4
supabase db reset docker volume create 1 1
supabase db reset docker volume rm 1 1
supabase test db docker image inspect 1 1
supabase test db docker run 1 1
supabase gen types --lang typescript --local docker container inspect 1 1
supabase gen types --lang typescript --local docker container inspect 1 1
supabase gen types --lang typescript --local docker container inspect 1 1
supabase db diff --use-pg-delta docker cp 3 3
supabase db diff --use-pg-delta docker create 1 1
supabase db diff --use-pg-delta docker image inspect 4 4
supabase db diff --use-pg-delta docker network inspect 1 1
supabase db diff --use-pg-delta docker rm 1 1
supabase db diff --use-pg-delta docker run 3 3
supabase db diff --use-pg-delta docker start 2 2
supabase db diff --use-pg-delta docker stop 1 1
supabase db diff --use-pg-delta docker cp 2 2
supabase db diff --use-pg-delta docker create 1 1
supabase db diff --use-pg-delta docker image inspect 1 1
supabase db diff --use-pg-delta docker network inspect 1 1
supabase db diff --use-pg-delta docker rm 1 1
supabase db diff --use-pg-delta docker start 1 1
supabase db dump --local docker image inspect 1 1
supabase db dump --local docker run 1 1

Slowest phases

Command Span Change Before -> after
supabase db dump --local db.dump +21 ms 202 -> 223 ms
supabase db dump --local cli.run +21 ms 219 -> 240 ms
supabase db dump --local command.db.dump +20 ms 205 -> 224 ms
supabase stack start --runtime native --eager experimental.stack.start +350 ms 6219 -> 6569 ms
supabase stack start --runtime native --eager cli.run +349 ms 6238 -> 6587 ms
supabase stack start --runtime native --eager command.stack.start +347 ms 6223 -> 6570 ms

Workflow run

Comment /bench (only includes lightweight benchmarks) or /bench full to run it again on the latest commit.

@Prashansa-K Prashansa-K mentioned this pull request Oct 30, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@supabase-oss supabase-oss Awaiting requested review from supabase-oss supabase-oss will be requested when the pull request is marked ready for review supabase-oss is a code owner

Assignees

No one assigned

Labels

do not merge Approve to apply; do not merge.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Prashansa-K