Skip to content

test: wait for pool connections to close before dropping test dbs - #67

Merged
kamilmysliwiec merged 2 commits into
masterfrom
test/postgres-flakes
Oct 29, 2026
Merged

kamilmysliwiec merged 2 commits into
masterfrom
test/postgres-flakes

Conversation

@kamilmysliwiec

Copy link
Copy Markdown
Member

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Other... Please describe: flaky e2e tests

What is the current behavior?

Issue Number: N/A

About one CI run in five fails (66 of the last 337). Most of those are an e2e job whose tests all passed, failed by Vitest caught N unhandled errors: error: terminating connection due to administrator command (57P01). Over the last month, tests/postgres/migrations.spec.ts produced 176 of these errors, on master, renovate and feature branches alike, and tests/postgres/contract-kysely.spec.ts produced 3.

The cause: pool.end() resolves once it has told each connection to close, not once they have. Right after, the test database's afterAll runs DROP DATABASE ... WITH (FORCE), which terminates the connections still closing. Their pool re-emits the 57P01 as error, nothing listens, and it becomes an uncaught exception. tests/support/postgres.ts already has endPool() for exactly this, but migrations.spec.ts ended its pools with pool.end(), and the Kysely client's db.destroy() ends its pool the same way. TypeORM and adapter-pg listen for their pools' errors, so a late 57P01 there is logged, not thrown.

Separately, contract-prisma.spec.ts failed once (2026-10-29): write() with a signal records it as signal() does, only when the write lands reported "the lease holder ends the child: expected true, got false". The case never checks what its claims return, so the miss surfaced two steps after it happened. It can only mean the claim at 5 didn't lease the child.

What is the new behavior?

  • migrations.spec.ts ends its pools with endPool(). Vitest runs afterAll hooks in reverse registration order (sequence.hooks: 'stack'), so that runs before the database is dropped.
  • The Kysely client ends its pool through endPool(pool, () => db.destroy()). endPool() takes an end for a pool a library holds, and still waits for its connections to close.
  • The contract case checks each step: the parent claimed, parked, then the child claimed under the lease that ends at 4 and again at 5. Any correct store already does this, as the claim cases before it test, so a run that misses one now names the step.

Evidence, against a postgres:18.6 container like CI's service:

  • A forced drop right after pool.end(): the pool emitted an error in 38 of 40 rounds (68 errors in 40 rounds with the whole postgres-store project running beside it). Right after endPool(): none, both times.
  • The Prisma case's sequence, run through fromPrisma 1,800 times (300 alone, 1,500 beside the whole project), never missed. Neither did 4 passes of the whole Prisma contract (200 cases), and no connection was left idle in a transaction after any of them. Prisma runs a raw query straight on the pool and awaits its COMMIT, so I found no lock that could outlive a statement. Its cause is unknown; the checks above will say which step misses if it happens again.

Checked locally:

  • Typecheck and lint pass.
  • The postgres-store project on the container: 11 files, 369 passed, no unhandled errors.
  • The contract on the memory and PGlite stores: 99 passed.

Not addressed here: MySQL's shutdown hand-back cases (interrupts, deployment, recovery) failed 5 times up to 2026-10-13 and not since, and tiny-queue.spec.ts failed once (2026-10-07).

Does this PR introduce a breaking change?

  • Yes
  • No

The contract (@nestjs/workflows/testing) checks more in one case. A store that passes the claim cases passes these checks.

Other information

migrations.spec.ts ended its pools with pool.end(), which resolves
once each connection was told to close, not once it has. The test
database's DROP DATABASE ... WITH (FORCE) right after terminated the
connections still closing, and their pools, which nothing listened
to, threw the 57P01 as uncaught errors, the most common reason an e2e
job went red. Kysely's db.destroy() ends the pool it was given the
same way. Both now wait for the connections to close with endPool(),
which takes the library's own end for Kysely's.
The case never looked at what its claims and the parent's write
returned, so a claim that missed the child surfaced two steps later
as "the lease holder ends the child". It now checks each, so a run
that misses one names that step.
@kamilmysliwiec
kamilmysliwiec merged commit 020c877 into master Oct 29, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant