fix(orchestrator): Fail the run when a provider-turn.start effect dead-letters

[mwolson]

Summary

• When a provider-turn.start or provider-turn.restart outbox effect
  exhausts its retry budget, transition the owning run to failed (attempt,
  root node, and a visible Provider error turn item included) instead of
  leaving the run in starting with a pending attempt forever.
• Adds a compensateDeadLetter hook to the orchestration effect executor. The
  worker invokes it on the dead-letter branch before the terminal
  outbox.fail settles; the executor routes the two turn-start effect types
  to a new ProviderTurnStartServiceV2.failFromDeadLetter.

Problem and Fix

Problem and Why it Happened	Fix
When a turn-start effect failed all 5 attempts, EffectWorker.runOnce marked the outbox row failed and stopped. Nothing owned the run transition, so the run stayed starting with a pending attempt indefinitely, the thread kept showing work in progress, and the only trace was the outbox row and a log warning. Observed live when a provider session.fork returned 400 on every attempt: the thread hung with no error. The gap is generic to any turn-start failure mode that survives the retry budget (adapter bug, provider binary drift, spawn failure).	EffectWorker now runs the executor's compensateDeadLetter before committing the terminal outbox.fail. For provider-turn.start and provider-turn.restart (which chains into the same turn start and resets its run to starting first), ProviderTurnStartServiceV2.failFromDeadLetter emits the compensating events: attempt failed, root node failed, a Provider error turn item carrying the redacted, bounded cause, and run failed. The write is guarded by EventSink.writeIfRunCurrent (expectedStatus: "starting", same activeAttemptId), so a run that advanced or terminalized through any other path, or a replayed compensation, is a no-op. The event shape follows the existing prepared-run.fail path and makeProviderFailureTurnItem.
Compensating after the terminal fail would not be durable: a crash or write failure between the two leaves a permanently failed effect with a still-hung run, because failed outbox rows are never claimed again.	Compensation runs first. At every crash point, either the run is already compensated or the effect is still unsettled, so startup reconciliation settles the run. Compensation failures are logged and never affect outbox settlement.
A dead-lettered restart can inherit open run-owned work from the interrupted attempt (subagents, child nodes, provider turns, streaming messages, in-flight turn items), including rows routed onto linked child threads, which usually carry runId: null and are never touched by startup reconcile (it only iterates per-run rows). Failing only the run would strand those entities open, and the child-thread ones permanently.	failFromDeadLetter sweeps the parent projection the way ProviderRuntimeRecoveryService.reconcileProjection does (open subagents, non-root nodes, and turn items go failed, superseded provider turns go cancelled, streaming messages stop streaming, an active provider thread returns to idle) and then follows lifetime subagent linkage, terminal links included, recursing through nested subagents, to fail each linked child thread's open rows in the same guarded commit. A child row that names a nonterminal run in its own thread belongs to an independently live app-owned delegation and is left alone.

Other effect types were reviewed case by case and deliberately left on the
existing warning-only dead-letter behavior: interrupt/steer target an
already-running turn whose normal ingestion owns settlement,
detach/respond/rollback/cleanup hold no run in a pre-provider state, and a
dead-lettered checkpoint.capture is self-healed by startup reconcile.

Validation

• vp check: pass
• vp run typecheck: pass (only pre-existing suggestions)
• Unit (EffectWorker.test.ts): the worker compensates before the terminal
  fail settles (ordering asserted), still settles cleanly when the lease was
  lost mid-dead-letter, routes start/restart (and only those) to the run
  failure path, and swallows a failing compensation.
• Integration (FoundationPersistence.test.ts): the real
  ProviderTurnStartServiceV2 and effect worker run against seeded projection
  state in SQLite; a dead-lettered turn start yields a failed run, attempt,
  and node, a Provider error item, cascaded open subagent/node/turn-item
  rows, a stopped streaming message, and no non-terminal runs, and a replayed
  compensation is a guarded no-op. Linked-child coverage: a provider-native
  child and a nested grandchild whose rows carry runId: null terminalize,
  while an app-owned child with its own live run survives untouched.
• Full orchestration-v2 suite: 54 files, 554 tests, green.
• Live (headless, isolated state): pointed the Grok binaryPath at a shim
  that breaks only agent stdio, so ACP initialize failed inside session open
  on every attempt. The outbox row ended status=failed, attempt_count=5 with
  the ProviderTurnStartError cause chain (the exact incident shape), and the
  run terminalized as failed with a visible error item within seconds; the
  thread settled. A healthy thread on the same server completed normally as
  the control.

Note

Fail the run when a provider-turn.start effect dead-letters in the orchestrator

• Adds compensateDeadLetter(effect, error) to OrchestrationEffectExecutorV2Shape in EffectWorker.ts; for provider-turn.start and provider-turn.restart effects, it calls ProviderTurnStartServiceV2.failFromDeadLetter before the outbox marks the effect failed.
• Implements failFromDeadLetter in ProviderTurnStartService.ts, which transitions a starting run to failed and cascades terminalization to the active attempt, root node, subagents, non-root nodes, provider turns, messages, and turn items.
• Compensation errors are caught and logged rather than propagated, so a failing compensation never blocks outbox settlement.
• The commit is guarded by writeIfRunCurrent with expectedStatus: 'starting', making repeated compensation calls idempotent.

📊 Macroscope summarized 7c7d299. 2 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 81faf1f9-6c3e-4805-8463-fdce039bf4ae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

🟠 High orchestration-v2/EffectWorker.ts:560

compensateDeadLetter runs before outbox.fail without first verifying that this worker still owns the effect lease. If execution exceeds leaseDurationMs, another worker can reclaim the row and successfully start the replacement turn, but this stale worker still calls failFromDeadLetter, which terminalizes a run that now belongs to the new worker's in-flight turn. The subsequent outbox.fail returns false too late to undo the corruption. Consider verifying lease ownership before calling compensateDeadLetter, or making compensation and outbox settlement atomic and lease-guarded.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/EffectWorker.ts around line 560:

`compensateDeadLetter` runs before `outbox.fail` without first verifying that this worker still owns the effect lease. If execution exceeds `leaseDurationMs`, another worker can reclaim the row and successfully start the replacement turn, but this stale worker still calls `failFromDeadLetter`, which terminalizes a run that now belongs to the new worker's in-flight turn. The subsequent `outbox.fail` returns `false` too late to undo the corruption. Consider verifying lease ownership before calling `compensateDeadLetter`, or making compensation and outbox settlement atomic and lease-guarded.

[mwolson]

The reclaim step in this scenario does not exist in this outbox. claimNext only claims status = 'pending' rows and additionally skips any thread that already has a running effect (claimableCandidatePredicate), and lease_expires_at is write-only across the repo (it never appears in a claim predicate), so a running row cannot be reclaimed while its worker is alive, regardless of lease expiry. FoundationPersistence.test.ts pins this ("does not reclaim a running effect after its process-local lease expires"). After a process crash, reconcileAfterProcessLoss cancels process-bound effect types, including provider-turn.start/provider-turn.restart, rather than requeueing them, and the crashed worker no longer exists.

The one operation that frees the per-thread claim gate under a live worker is cancelUnsettled, and both of its callers (the pre-start interrupt path and recovery reconcile) terminalize the owning run in the same commit, so the writeIfRunCurrent CAS in failFromDeadLetter observes a non-starting run and the compensation is a no-op. A lease re-check before compensation would also reintroduce the failure this PR removes, by skipping compensation when a row is cancelled without the run being terminalized. This invariant is now documented in the compensateDeadLetter doc comment.

[macroscopeapp]

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

[macroscopeapp]

🟡 Medium orchestration-v2/ProviderTurnStartService.ts:675

A ProjectionStoreV2Error while reading a linked child thread projection is silently swallowed, so the dead-letter compensation can still successfully settle the parent run while leaving the child's open nodes, turn items, streaming messages, and subagents stranded permanently — there is no retry of this sweep. The code catches all read failures and treats them as a deleted child (// A deleted or unreadable child thread has nothing to settle.), but ProjectionStoreV2Error also covers transient or unreadable store failures, not just missing children. Only a confirmed missing/deleted child should be skipped; other read failures must propagate so the overall compensation fails and can be retried.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/ProviderTurnStartService.ts around line 675:

A `ProjectionStoreV2Error` while reading a linked child thread projection is silently swallowed, so the dead-letter compensation can still successfully settle the parent run while leaving the child's open nodes, turn items, streaming messages, and subagents stranded permanently — there is no retry of this sweep. The code catches all read failures and treats them as a deleted child (`// A deleted or unreadable child thread has nothing to settle.`), but `ProjectionStoreV2Error` also covers transient or unreadable store failures, not just missing children. Only a confirmed missing/deleted child should be skipped; other read failures must propagate so the overall compensation fails and can be retried.