Backport cluster shutdown, storage recovery and workflow suspension fixes to v3

[tim-smart]

Backport the cluster and workflow fixes to v3 so shutdown and shard reassignment settle outgoing requests, abandoned durable work can replay without persisting a false completion, and deferred completions wake suspended workflow parents. The bundle also repairs storage cleanup, lock recovery, stream completion and resource acquisition failure handling.

Closes EFF-1345
Closes EFF-1349

Nested-workflow test correction; awaiting independent review. Test tip 5448e8c688dfe6fa942d81b500245befb20f297b. packages/workflow/test/WorkflowEngine.test.ts now waits for the asynchronously resumed parent with the existing polling helper, capped at 100 yields, then retains the exact Complete(Exit.void) assertion. It does not advance the clock again, manually resume the workflow or increase the test timeout. Production and the single changeset are unchanged from 0666b53a3c.

The completed Node 4/4 failure at 0666b53a3c returned undefined from the immediate nested-workflow poll. This is separate from the earlier Ryuk acquisition failures. Under a controlled Scheduler.timer(10), the original test fails in 202 ms, the corrected test passes in 232 ms, and a parent deliberately stalled after child completion still fails the bounded assertion in 1252 ms, without a Vitest timeout. These temporary controls are archived in EFF-1345; only the focused test correction is committed.

Validation: the workflow suite passes 20/20 in 6.62 seconds, and the full cluster/workflow/RPC bundle passes 282/282 across 25 files in 201.04 seconds. Root lint-fix, type checking, build, docgen and diff checks pass. Production and changeset diffs are empty. New CI is not part of this local result; EFF-1345 retains reproducible controls and logs.

Attribution limits: the delayed scheduler reproduces the early poll, not the exact CI interleaving. Ordinary local repetitions passed. Original v3 1af4232fea7 remained suspended under the prior scheduler probe, so the precise CI race is not proven inherited. The latest #6972 correction does not change this memory-engine path; the broader backport changes workflow scheduling, and its effect on the race's frequency is unmeasured. Historical failure evidence and deployment limits below remain in force.

#6972 persisted reply delivery passed independent review. Implementation tip 0666b53a3c274fc17613d1851dab55c57938e801, with tests unchanged from 2634df4a1f9607b5c6d6b11775f796468a204ed0. The encoded adapter carries the reply it actually persisted back to MessageStorage.make. The wrapper classifies and delivers that reply, so a malformed chunk releases parked waiters and callers receive the stored terminal defect. Valid chunks retain their values and keep waiters registered until normal terminal completion. The encoded save completes before waiter release or response callbacks; a failed save cannot notify them.

V3 API change: MessageStorage.saveReply now returns ReplyWithContext<R> in its effect instead of void. Main changes only the input adapter contract; v3 also returns the reply from the service to preserve existing MessageStorage.make(existingService) wrapping. Custom service/constructor implementations must return the actual persisted reply, including a fallback. Use Effect.asVoid where a void result is required. Encoded driver signatures and error channels are unchanged. The single changeset documents the migration with unchanged release levels.

The unchanged MessageStorage and RunnerServer suites pass 20/20 in 5.71 seconds, including malformed success/error/chunk delivery and the valid-chunk control. The full cluster/workflow/RPC bundle passes 282/282 tests across 25 files in 178.58 seconds, including 59/59 SQL tests. Root lint-fix, type checking, build, docgen and diff checks pass. Only MessageStorage.ts and the single changeset change. EFF-1345 retains the implementation patch, commands and logs.

Historical discrimination at production bd24b741bb and test tip 2634df4a1f: malformed success/error replies persisted their defect and released their waiter, but the callback received the original invalid reply. The malformed chunk stored a WithExit defect while its waiter stayed pending and its callback received Chunk. The focused suite had 13 passes and these three failures in 7.14 seconds; the full bundle had 279 passes and the same three failures across 25 files in 176.86 seconds. The one-second live observation joined interruption cleanup before asserting, with no unrelated failures or Vitest timeouts. Original commands, patch and logs remain in EFF-1345.

These are storage-module callback checks, alongside existing RunnerServer terminal-stream controls. Pending-cache work and unrelated simplifications remain outside this correction. The multi-runner harness remains removed; historical integration evidence, limitations and the full-stop deployment requirement below are preserved.

Whole-PR simplification pass. Tip bd24b741bb on top of e05241c69d. Production changes are behavior-preserving: ResourceRef rebuild reuses a new aroundEntity helper on the teardown tracker instead of hand-rolled acquire/ensuring/release, MemoryDriver.clearAddress shares one address predicate, and DurableDeferred drops an else after return. Tests drop dimensions that could not change outcomes: the catchAllCause/exit recovery split (both compile to the same runtime opcode), the interruptible control of the masked-send case, the unpersisted AckChunk case, the two extra observe and typed-failure loops, and the inert ResourceRef rebuild control in Sharding.test.ts (its local ref had no teardown and could not affect classification). Three malformed-reply tests merge into one, the two PostgreSQL prefix-isolation tests merge into one inside the shared container block, the empty liveness probe folds into the bounded-recovery test, and the duplicate MemoryLive layer in MessageStorage.test.ts uses the shared fixture. The cluster durable-interrupt abandonment check now runs in-process inside ClusterWorkflowEngine.test.ts; fixtures/run-fixture.ts and fixtures/workflow-interrupt-abandonment.ts are removed, so no child process remains in the suite. Mailbox.fromStream was not reintroduced. Validation at bd24b741bb: non-SQL cluster/workflow/RPC bundle 220/220 across 23 files, SQL suites 59/59, lint-fix, type checking, cluster/workflow build and docgen, and git diff --check all passed.

Mailbox cancellation corrected; awaiting independent review. Implementation tip: e05241c69dac03dd83d1e799e6a2501631d39356, based on test tip 396bdd9ad73fed92ea8e0177c5521c2d5f37f273. The explicit bounded, interruptible pump is restored in Sharding.ts. Mailbox.fromStream inherits acquisition masking and shuts down its mailbox with a successful end, so it cannot preserve these request-cancellation semantics. The restored pump permits scope closure under masked acquisition and forwards interruption to readers outside the request scope. Owner signalling still precedes forwarded abandonment; one-element RPC buffering, requested/default consumer capacity, buffered terminal values and cancellation of blocked forwarding are retained.

All 54 focused mailbox/abandonment cases pass in 19.00 seconds. The full cluster/workflow/RPC bundle passes 309/309 tests across 25 files in 196.75 seconds, including Sharding 74/74, cluster workflows 69/69 and SQL 61/61. Root lint-fix, type checking, build, docgen and diff checks pass. Tests and the single changeset are unchanged from 396bdd9ad7; only the pump block and its explanatory comment changed. EFF-1345 retains the implementation patch, commands and logs. The other simplifications remain intact, and the multi-runner harness remains removed.

Historical failure evidence: at a8fe71464a, the two masked-acquisition cases left scope closure pending until cleanup released the remote stream. The two unmasked cases closed but returned Exit.Success([emptyChunk, true]) to the external reader. The test-only bundle had 305 passes and these four failures in 194.78 seconds, without unrelated failures or Vitest timeouts. Cleanup completed before the assertions. Restoring only the prior pump in an isolated checkout passed all 54 focused cases with the other simplifications intact. Adding Effect.interruptible around Mailbox.fromStream alone would leave its clean-shutdown behavior for external readers. Original logs and the isolated control patch remain in EFF-1345.

Simplification pass. Production tip a8fe71464a replaced the explicit pump with Mailbox.fromStream. Its original 305-test bundle passed, but independent review found the two lifecycle changes above; this restoration addresses them. The original behavior-equivalence claim remains withdrawn. The shared owner-signal helper, inlined Effect.onError, lazy pending-map allocation, shared polling and fixture/config helpers, shutdown-send helper and direct optional-capacity arguments are preserved. Historical integration evidence and the full-stop deployment limits below remain unchanged.

Prior explicit bounded pump (variant A). The behavior and historical validation in this section describe the prior implementation now restored above. Implementation tip: 3a8d77d0ce89fc09ad278914dec2d149a73cd049, with tests unchanged from 938e823786237026ed625b1c7897faa7d2511dd8. Raw entity clients using asMailbox: true now use a one-element RPC buffer and a consumer mailbox with the requested streamBufferSize (default 16). The request-scoped pump forwards one element at a time, retaining consumer capacity and takeN semantics. This fixes the unbounded producer lead introduced by the first scoped-pump implementation at 7ef053dbda.

The pump still interrupts the tracked active workflow/activity owner before forwarding abandonment. Abandonment can wait behind buffered values until consumption resumes; request-scope closure cancels a pump blocked on either a source read or consumer offer. Raw non-owner mailbox readers receive a recoverable cause rather than being re-interrupted, including when no owner is tracked. Unary and ordinary stream paths, the shared interruption helper and the public send-abandonment wait are unchanged.

For finite consumer capacity C, the pump adds one RPC-buffer element and at most one forwarding element; a producer chunk may also be in flight. Singleton producer lead is bounded by C + 3 in the regression fixture (19 with the default capacity), compared with the former wrapper's measured 17. Element-wise forwarding adds per-element effect/queue work and can change batching; throughput and sustained heap usage have not been benchmarked. These bounds count elements, not arbitrary upstream allocations or heap bytes. The single changeset records the buffering tradeoffs, non-owner behavior and cancellation guarantee, while retaining release levels, v3 DeferredState API notes and the full-stop requirement.

All 50 unchanged focused cases pass in 19.10 seconds. The full cluster/workflow/RPC bundle passes 305/305 tests across 25 files in 210.40 seconds, including Sharding 70/70, cluster workflows 69/69 and SQL 61/61, with no failures or timeouts. Root lint-fix, type checking, build, docgen and diff checks pass. Only Sharding.ts and the single changeset changed; all tests remain byte-for-byte unchanged from the test tip. EFF-1345 retains the implementation patch, commands and validation logs.

Historical failure evidence is unchanged. At production 7ef053dbda, the four producer-bound cases emitted all 128 values with no consumer, and both takeN cases exceeded the requested capacity. The test-only bundle at 938e823786 had 299 passes and these six expected failures; its final focused rerun had 44 passes and the same six failures. Architect's isolated variant A control passed 50/50. Before these regressions were added, the unbounded pump passed all 295 existing bundle tests and all 40 focused lifecycle/durability cases. EFF-1345 retains the original logs, commands and control patch. The multi-runner harness remains removed; historical integration evidence and limitations below still apply. Independent review approved that explicit bounded implementation; the later Mailbox.fromStream lifecycle regressions are tracked above.

Baseline discrimination is preserved: the wrapper passed all 295 bundle tests at b95e934cf2; an isolated passive scoped mailbox.await watcher passed 38 selected cases but failed both strengthened durability cases by persisting a false Complete. Restoring the old post-recovery yield made both cases pass, confirming it hid the race. That mutation was reverted and is not part of the PR. EFF-1345 retains the original reproduction commands, patches and logs.

Masked public Sharding.sendOutgoing abandonment fixed. Implementation tip: 8bc6e25d68c07da712c1cfc468ec25535c6ffc84. Persisted sendOutgoing(request, false) now follows the abandonment signal with a local Effect.interruptible(Effect.never). Storage still completes first; the interruptible wait then prevents a masked caller from continuing past the abandoned send. ClusterAbandon.interrupt itself and its cleanup re-signal masking behavior remain unchanged, as do volatile sends, acknowledgements and discard paths. The existing changeset documents this behavior; release levels, v3 DeferredState API notes and the full-stop requirement remain unchanged. Other audit follow-ups are outside this fix.

At masked-send implementation tip 8bc6e25d68, the two unchanged cases from test tip 2b14a0f5d27401f6c15d8270bde7bc6bd23245a9 passed: masked and interruptible callers stopped without continuation, exited with interruption, and left the request pending. The full cluster/workflow/RPC bundle passed 287/287 tests across 25 files in 195.89 seconds, and root lint-fix, type checking, build, docgen and diff checks passed. That implementation kept tests and the shared abandonment helper unchanged. Independent review confirmed baseline discrimination and 287/287 passes with no blockers. The bounded-pump implementation above follows that fix.

Baseline evidence at production ccd1e66a48 and test tip 2b14a0f5d27401f6c15d8270bde7bc6bd23245a9: the interruptible control passed, while the masked caller continued before interruption took effect at the mask boundary. Both fibers settled and left the request pending. The test-only full bundle repeat had 286 passes and that expected failure in 195.95 seconds. Its first bundle also had three unchanged five-second cluster workflow timeouts (should run a workflow, interrupt, Activity.raceAll durable) while root validation jobs ran concurrently; those did not recur after the jobs finished. No assertions or deadlines changed. EFF-1345 retains the original results and reproduction commands. The multi-runner harness remains removed.

Paired deferred-wake fixes (#8243 and #8244). Implementation tip: ccd1e66a48bca988f902eebf6f652211a2610505. A deferred completion can arrive while a discarded run's Suspended reply is still being persisted. The old handler read no reply and returned, leaving the workflow suspended. Deferred wakes now share the existing run-reply waiter with child resumes, subscribing before reading storage and releasing the resume gate before reset can start replay.

The current run retains its awaited-deferred set through reply persistence. After v3's deferredDone records the result and preempts an awaiting run, the wake checks that set: unrelated completions use the ordinary resume path instead of occupying a run-reply wait. This preserves mailbox capacity for the required completion. The handler uses v3's existing Rpc.wrap({ fork: true, uninterruptible: true }) mechanism to bypass RPC concurrency while waiting, rather than importing main's activation-scope machinery. Pending-result retention and public API shape remain unchanged.

Unchanged module regressions at test tip 1f1b0eb9222481065952eeba7f5bc06ec3f1831a cover the suspension-commit race, unrelated/required completions at capacities 2 and 3, and handover before deferred-reply persistence. All four focused cases now pass. At that implementation tip, the full cluster/workflow/RPC bundle passed 285/285 tests across 25 files in 202.60 seconds with tests unchanged from that tip. Root lint-fix, type checking, build, docgen and diff checks also passed. Before this implementation the full bundle had 284 passes and one #8243 failure. The isolated #8243-only mutation fixed that failure but stalled capacity 2; capacity 3 and handover passed. Reverting the retention hunk separately made handover fail. EFF-1345 retains the commands, mutation patches and logs. The multi-runner harness remains removed.

packages/workflow/test/WorkflowEngine.test.ts now uses the two in-process cases accepted in main #8229 (10761707b5), adapted to v3: success verifies cleanup-before-replay and the final result; failure verifies the typed boom error. The six-scenario subprocess harness and fixtures/deferred-completion.ts are removed. Dedicated coverage removed includes external completion remaining pending during cleanup, unrelated completion not preempting the active run, exactly-once terminal finalizers, late-completion stability, and wake cancellation during teardown (including the workflow-finalizer blocking control). The hard subprocess watchdog is also gone; the in-process timeout cannot independently terminate uninterruptible cleanup. Other module and shared contract tests remain; they are not claimed as substitutes for these deleted checks. The cluster abandonment check later moved in-process as well (see the simplification pass above).

The single changeset now documents v3's DeferredState.pendingResult and the exit argument to deferredDone, which differ from v4. It retains the same release levels and full-stop deployment requirement. That implementation kept tests unchanged from 1f1b0eb922; the public-send regression described above was added in the subsequent test-only run.

Based on current v3 at 1af4232fea7bc613e1dc68db9bec7b1f596d9e68. The scope recheck found no additional complete scoped backport already present since the original audit; the intervening v3 change was the TOML dependency update. This is the single integration PR for the 27 upstream PRs below (26 fixes, with #7350/#7351 treated together).

Deployment requires a full stop. For #6798, stop every runner using the old PostgreSQL advisory-lock protocol before starting any runner using the new protocol. A rolling deployment can allow old and new runners to own the same shard because the lock namespace now includes the storage prefix. Runner Notify messages also gain a required persisted boolean; update all runner peers together. Lock-isolation tests do not replace this operational requirement.

Coverage paths below are relative to packages/cluster/test/ unless they start with packages/. Tests are grouped by module; the engine-agnostic contract cases live in packages/workflow/test/WorkflowEngineContractTest.ts and run under both the memory and cluster engine suites. “Ported” includes the v3 adaptations described below.

Upstream PR	v3 result	Coverage
#7032	Ported single-runner shutdown settlement and finalizer delivery.	Sharding.test.ts, SocketRunner.test.ts: finalizing-entity calls, request/discard and preemptive settings, remote finalizer discard.
#7134	Ported abandoned-request/ack settlement and typed routing errors.	Sharding.test.ts, EntityProxy.test.ts: abandoned acknowledgements, RPC/HTTP error schemas and discard controls.
#7485	Ported transient persisted-message abandonment and workflow replay; adapted genuine interruption across v3 RPC/stream boundaries.	Sharding.test.ts (masked public-send control, mailbox forwarding/cleanup, bounded-pump regressions and masked-acquisition/external-reader cancellation coverage), ClusterWorkflowEngine.test.ts, MessageStorage.test.ts, ClusterAbandon.test.ts; standalone packages/rpc/test/RpcClient.test.ts.
#7489	Bound classification to active teardown, with counters owned by each Sharding.	Sharding.test.ts: colliding addresses across independent Shardings and nested teardown; Interruptors.test.ts.
#7018	Hold persisted messages while registration is building, with bounded missing-registration fallback.	Sharding.test.ts: memory and SQLite registration coverage.
#7195	Volatile discard completes after delivery without waiting for the handler.	SocketRunner.test.ts, Runners.test.ts, Sharding.test.ts: TCP delivery, transport retry and local controls.
#6972	Ported. Persist the defect fallback before releasing waiters and delivering the actual persisted reply.	MessageStorage.test.ts: malformed success/error/chunk fallback, waiter release and outgoing response delivery; valid-chunk retention and completion control. RunnerServer.test.ts: normal terminal-stream controls.
#7551	Empty shard-lock probes use the shared pool, bypassing a wedged reserved connection.	SqlRunnerStorage.test.ts: blocked reservation, pool probe and bounded recovery.
#7906	Use registration-time services while retaining runner-owned services.	Sharding.test.ts: registration overrides and caller-context controls.
#7837	Preserve the joined reply ID in SQL by-ID acknowledgement reads.	SqlMessageStorage.test.ts: PostgreSQL, MySQL and SQLite.
#7860	Return only requested PostgreSQL shard locks from acquisition.	SqlRunnerStorage.test.ts: held/requested shards and unrelated locks on the reserved connection.
#6798	Namespace advisory locks by storage prefix.	SqlRunnerStorage.test.ts: prefix isolation/exclusion, frozen UTF-8 hash vectors, foreign lock preservation.
#7039	Fail ResourceRef rebuild waiters and close failed acquisition scopes.	ResourceRef.test.ts: failure, cleanup, recovery and stale concurrent failure.
#7066	Close failed ResourceMap lookup scopes immediately.	ResourceMap.test.ts.
#7041	Emit terminal runner replies and close stream mailboxes.	RunnerServer.test.ts: persisted/volatile success and typed failure.
#7889	Forward disableFatalDefects to entity test servers.	Entity.test.ts: true/false/omitted settings, typed failure and entity isolation.
#7835	Remove queued control envelopes when clearing a memory address.	MessageStorage.test.ts: Interrupt/AckChunk deletion and other-address preservation.
#7038	Clear memory request deduplication state with the address.	MessageStorage.test.ts: primary-key reuse.
#7074	Preserve memory runner health at registration.	RunnerStorage.test.ts: unhealthy registration and health updates.
#8070	Port child-specific resume keys, run-reply waiting, resume gates and activity registration cleanup. The activity retry guard is defensive in v3.	packages/workflow/test/WorkflowEngineContractTest.ts (memory and cluster): fan-out, cleanup wakeups and retry-budget controls.
#7840	Honor numeric/bigint zero DurableClock thresholds.	packages/workflow/test/DurableClock.test.ts: numeric, bigint, string and omitted thresholds.
#7179	Track awaited deferreds and pending completions to wake active races.	packages/workflow/test/WorkflowEngineContractTest.ts (memory and cluster) and packages/workflow/test/WorkflowEngine.test.ts: mapped/into/multi-await races, success-biased controls, interruption persistence, self-completion success, typed failure and cleanup-before-replay ordering.
#7000	Reuse memory workflow scope across suspension.	packages/workflow/test/WorkflowEngine.test.ts: scope continuity and terminal cleanup.
#7350	Deposit interrupts outside transient instances, after body finalizers and before workflow finalizers.	packages/workflow/test/WorkflowEngine.test.ts: body ordering, terminal ordering and late-completion controls.
#7351	Documented no-op: v3 has no interruptUnsafe; #7350 makes the equivalent change for its existing interrupt entry point.	Same paired coverage; no claim to exercise an absent API.
#7428	Workflow RPC/HTTP discard endpoints declare and return the deterministic execution ID.	packages/workflow/test/WorkflowProxy.test.ts: schemas, RPC types and actual HTTP response.
#8132	Make activity-count registration interruption-safe with acquire/use/release.	packages/workflow/test/Workflow.test.ts: interrupted activity-count acquisition.

#7016 is excluded: the v4 fractional-clock schema defect does not apply to v3; SQL compatibility for that excluded item remains unverified.

Compatibility and v3 adaptations

One changeset gives @effect/cluster and @effect/workflow minor releases and @effect/rpc a patch release. Entity clients and non-discard EntityProxy request schemas include EntityNotAssignedToRunner. Persisted-request abandonment interrupts for replay; volatile requests and abandoned acknowledgements expose typed routing errors. Local discard settles during shutdown, while remote volatile discard still reaches a live peer. Runners.notify accepts volatile messages and exposes their delivery failures. Workflow proxy discard success changes from void to string. The workflow additions include makeDeferredState/DeferredState, WorkflowInstance.abandoned/awaitedDeferreds and an optional third WorkflowInstance.initial scope parameter. Custom instance construction must initialize both new fields; initial supplies them. Activity.execute keeps its existing environment type. Activities that suspend run again on replay, so side effects before suspension must be idempotent. Use Workflow.addFinalizer for terminal work that must observe a deposited workflow interrupt; body-level finalizers run before it is exposed.

The RPC cleanup, mailbox pump and attempt-owner mechanism are v3-specific adaptations. In v3, interruptWith alone is catchable, RPC writes run in a separate fiber, and stream channels relay exits across fibers. The port records genuine fiber interruption, re-signals abandonment in unary callers and stream consumers, and interrupts the active workflow/activity owner before the raw-mailbox pump exposes a marked failure. Raw non-owner mailbox readers can recover as described above. Re-signalling preserves masking; the generic RPC onError cleanup settles the source mailbox even when its write fiber is interrupted, preserving the writer's failed exit.

Owner escalation is deliberately stronger than upstream: a detached child encountering marked abandonment also interrupts its still-active owner. After execution ends, the owner is inert, but inherited contexts retain its RuntimeFiber reference until released. No claim is made that unowned stream runners acquire workflow-owner semantics. The generic RPC cleanup also retains ordinary failures and can produce a Debug-level unhandled-fiber log below the default logging threshold.

Deferred tracking goes beyond upstream in two ways: completion is recorded even when no run is tracked, and a pending map is retained for each execution that suspends without resuming on that runner. Both engines also alias the mutable awaited-deferred set into activity instances, extending preemption to deferreds awaited inside activities. The run-reply waiter starts before storage is read and remains interruptible in its owning scope; v3 uses Runtime.runFork under acquire/release for that ordering.

Validation and limits

Cleanup tip: 980119a03ce19c392c0771396deb5e8206fcd20b.

Validation after removal: 285/285 tests passed across 25 files in 198.85 seconds using nix develop -c pnpm test run packages/cluster packages/workflow packages/rpc --maxWorkers 3. Root lint-fix, type checking, build, docgen and diff checks passed. All 21 module coverage paths still exist, and no references to the removed harness remain in the repository.

The dedicated multi-runner integration suite, controller, worker and guide have been removed as requested. Production fixes, all other regression tests and the single changeset are unchanged. Current coverage paths are listed in the module table above; the integration results below are historical evidence, not a suite retained in this PR.

Validation at the prior documentation tip 3f8fa15171: cluster workflow 63/63, root lint-fix, type checking, build, docgen and diff check passed. The integration matrix was not rerun for prose edits; the results below remain tied to the stated test tip.

The historical independently reviewed integration test tip is b21c32c892657fcf50661d6008ba987c21333b3d, with production at 868849cb58596141a3655d7aa9d2420c3e4640b2 for those runs. Those integration runs have not been repeated for this paired fix; the harness remains removed. At that tip, the fixture resumed application values from lastSentChunkValue, following TestEntity.ts, while the runtime resumed reply sequencing. Exact 0..7 assertions, workloads, gates and deadlines were preserved. These fixture details describe the historical test tip; the dedicated integration files are no longer present at the current tip.

The required four-scenario matrix, repeated three times, passed 11/12 cases in 241.72 seconds. All 288 requests, 72 parents, 72 deferred races and 72 streams returned their expected values, with no premature parent completion. Full-stop recovery took 7.342–9.320 seconds; graceful shutdowns exited without forced cleanup. One full-stop case failed the handler audit. Three additional isolated full-stop controls passed, with recovery in 7.359–11.846 seconds. These controls do not erase the original failure.

Independent replication at b21c32c892 passed steady and full-stop twice each: 4/4 cases, 193 samples without ownership violations, final partitions with 12 locks, and no unprocessed message groups. Full-stop recovery took 7.6–8.2 seconds. Independent checks also passed the 285-test cluster/workflow/RPC bundle (opt-in integration skipped), type checking and cluster/workflow lint. Review found no blocking change in the reviewed range.

Separate instrumented diagnostics passed 30/34 cases. Four cases under six competing CPU loads missed the unchanged 45-second partition-convergence deadline, including one in the smaller-pool batch; at least one failed before workload submission. None of 4,875 samples showed an ownership violation. These observations do not establish timely convergence under CPU contention or exclude overlap between samples.

The diagnostics and an independent reproduction confirmed an inherited sql-pg cancellation race. PgClient.ts queues pg_cancel_backend through the shared pool, bounded to five seconds, while the target connection can already have returned to the pool. A delayed cancel can then kill an unrelated query on that reused connection with SQLSTATE 57014. The deterministic reproduction triggered the race 3/3; an independent repeat confirmed it. An instrumented eight-connection run caught an audit finalizer being cancelled and leaving its row open. The backport leaves sql-pg unchanged, but stronger teardown interruption may increase exposure. The partial baseline comparison cannot quantify that increase. A separate v3 sql-pg fix remains a follow-up.

The original unclosed request-22 row is well supported as stale audit cleanup, rather than directly proven: the handover timeline, sibling cleanup and reproduced cancellation mechanism support that attribution, but the original run lacked per-handler exit and cancellation instrumentation. The audit equates an open row on a live process with an active handler, so failed cleanup can look like overlap. It filters out exited processes, hiding their failed cleanup, and 100 ms sampling can miss brief ownership overlap. The original audit assertion and failed result remain unchanged.

One original steady snapshot retained four losing-race activity rows. These activities remain uninterruptible server-side and can finish after the workflow result. All 34 diagnostic cases found zero residual rows after results and after drainage checks, with observed drain times of 0–2 ms. This supports transient drainage in those runs; successful results and single snapshots do not prove cleanup for every workload.

Earlier failures remain in the EFF-1345 reports. At aefc51e4be, only 2/12 cases passed: 66/72 parents completed and 24/72 stream checks passed. Production 868849cb58 recovered every parent and fixed decoding, but the old fixture repeated stream prefixes, leaving 6/12 cases and 57/72 exact streams passing. The corrected fixture passes every stream value check without consumer deduplication or relaxed assertions.

Production fixes in 868849cb58: SQL chunk rows use kind = null, which Number(null) misclassified as WithExit; the decoder now tests null directly. This was inherited from v3. Separately, the backport's concurrent deferred/replay path could start a suspended execution on a new owner before the completion reply reached SQL. The buffer now records completion before that owner's first local run. Cleanup ordering and terminal removal remain unchanged. Pending completion maps can remain until a terminal local run or engine disposal; owner churn can extend retention when no terminal run occurs locally.

Baseline v3 1af4232fea7bc613e1dc68db9bec7b1f596d9e68 serialized deferred completion through reply persistence. Comparisons identified inherited SQL behavior, but baseline startup/reassignment failures prevented a complete matrix comparison. They do not establish that every scenario is regression-free.

Historical reproduction remains available in Git history: use an isolated checkout of test tip b21c32c892657fcf50661d6008ba987c21333b3d and the archived guide with validation limits. The removed files were packages/cluster/test/ClusterWorkflowEngine.integration.test.ts and packages/cluster/test/fixtures/{multi-runner.ts,multi-runner-worker.ts,multi-runner.md}. EFF-1345 retains the commands, topology, workloads, timings, SQL snapshots, logs, original failures, diagnostic instrumentation and independent review. These bounded TCP/PostgreSQL observations support recovery in the tested conditions; they are not an unconditional production-safety claim. The full-stop deployment requirement above remains mandatory.

Earlier local validation passed root lint-fix, type checking, build, docgen and diff checks. The first full bundle had 284 passes and an existing Sharding forced-release timeout; isolation then passed Sharding 52/52 and the bundle 285/285 across 25 files in 193.84 seconds. No code or timeout changed between runs. Independent validation above is separate from those results.

Prior documentation cleanup: 39b1be986438c0dc60c60712dab415919cd46e3d. The documentation cleanup consolidates the three changesets into one and shortens code comments. Syntax comparisons with comments removed confirm no executable changes.

Documentation cleanup validation: 98/98 workflow and ClusterWorkflowEngine tests passed. Root lint-fix, type checking, build, docgen and diff check passed; syntax comparisons covered all 17 edited TypeScript files. The single changeset preserves the same release levels and deployment requirement.

Production implementation: 670fcbefc827ce4bfe0960227767f0031b22fa58.

Memory-engine self-completion records the result and schedules an interruptible wake in the engine scope. The completion finalizer can return while the wake waits for the old run's cleanup before replay. Explicit interruptibility prevents the wake from inheriting the completion finalizer's uninterruptible flag, allowing engine teardown to cancel the wait. External completion retains its synchronous wait. The shared deferred-state helper and cluster engine are unchanged.

The memory self-completion coverage has been replaced by the two in-process cases described above. The following baseline and validation results are historical results from the retired subprocess harness.

Baseline discrimination:

• At c517e07943 (also the unchanged production at test tip d93c4ee2ca), the finalizer-origin wake prevents engine closure until workflow cleanup is released. Completion from an interruptible workflow region passes with the same blocked cleanup, and external completion remains pending as required. The affected suites had 97 passes and this one regression failure.

• Both self-completion cases failed before the deadlock fix, at production 218ef218ba. Against archived v3 workflow sources (1af4232fea), self-success, self-failure and unrelated completion pass; the external replay control fails because the baseline lacks the deferred race wake-up fix.

• Affected workflow and ClusterWorkflowEngine suites: 98/98 passed, including the formerly failing teardown regression and both external-completion and cleanup-ordering controls. Command: nix develop -c pnpm test run packages/workflow packages/cluster/test/ClusterWorkflowEngine.test.ts --maxWorkers 3.

• Full cluster/workflow/RPC bundle: 285/285 passed across 25 files, including 61/61 SQL tests, in 197 seconds. Command: nix develop -c pnpm test run packages/cluster packages/workflow packages/rpc --maxWorkers 3.

• Root nix develop -c pnpm lint-fix, nix develop -c pnpm check, nix develop -c pnpm build, nix develop -c pnpm docgen, and git diff --check: passed.

• That historical validation used test logic and assertions from d93c4ee2ca. These are local results from before the in-process replacement.

Both subprocess fixtures have since been removed; the cluster abandonment check runs in-process.

Prior staged baseline checks establish discrimination, rather than just passing controls: the corrected workflow tests produced 22 assertion failures and eight passes on pre-implementation af931e70; the standalone RPC mailbox/stream regressions both failed before cleanup at 369fb7a431; and the Sharding isolation regression failed only during overlapping teardown with production at 7990c71550. The control with completed teardown passed on both revisions. Original cluster testing also demonstrated failures against v3; direct tests of newly introduced marker/resource APIs could not run unchanged on that base. Nested teardown, some registration protection and ordinary shutdown cases are compatibility controls.

The Activity.retryOnInterrupt portion of #8070 has no demonstrated v3 regression: retry-budget controls pass before and after the port. It is retained as a defensive upstream guard, not claimed as a behavioral fix in v3.

Accepted coverage limits remain:

• Historical integration runs covered bounded multi-process TCP/PostgreSQL transfer and restart; the dedicated suite has since been removed. The inherited sql-pg cancellation risk, possible increased exposure during teardown, loaded convergence failures, audit blind spots and partial baseline comparison are detailed above. Network partitions, database failure, client restart, rolling mixed-version upgrades, sustained production load and every workflow graph remain outside this coverage.
• No complete upstream race matrix or adopted-registration interruption coverage is claimed. Broader masked stream/mailbox consumption, clear and direct mailbox evaluation, terminal/abandonment combinations beyond the finite mailbox cases above, durable-activity stream abandonment with owner rebinding, detached-child escalation and stale inherited owner contexts remain follow-ups.
• Earlier broad runs occasionally hit unchanged nested-workflow/failover timing cases. The final review isolated a separate 100 ms empty-probe round-trip flake under SQL load. This PR bounds that probe to ten retries spaced 20 ms apart, preserving its tight per-attempt deadline and the permanently wedged reserved-connection assertion.

[changeset-bot]

🦋 Changeset detected

Latest commit: 5448e8c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages

Name	Type
@effect/cluster	Minor
@effect/workflow	Minor
@effect/rpc	Patch
@effect/platform-bun	Patch
@effect/platform-node-shared	Patch
@effect/platform-node	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR