diff --git a/docs/architecture/queue-scheduler.md b/docs/architecture/queue-scheduler.md index 63a5b4f39..da9e07441 100644 --- a/docs/architecture/queue-scheduler.md +++ b/docs/architecture/queue-scheduler.md @@ -197,6 +197,10 @@ When a worker dequeues a message whose `run.status === "processing"`, it consult > **Why re-defer instead of delete?** Recovery of a `processing` run whose worker dies hard (OOM/SIGKILL — no graceful failure write) relies on the queue redelivering its message so a healthy worker eventually sees the stale heartbeat. The *only* trigger is that the message still exists. If a fresh-heartbeat duplicate were **deleted**, the recovery token would be destroyed: worker A loses its popReceipt → worker B dequeues the dupe, sees A's fresh beat, deletes the message → A then dies hard → the run is `processing` with **no queue message in existence**, and the scheduler only dispatches `pending` runs. Re-deferring keeps the token alive: each time the message resurfaces, a fresh beat re-defers (cheap), a stale beat marks failed. The terminal-`done` guard still deletes the message when a later worker dequeues it after the original completed successfully — so "run completed, but the queue message still exists" is a normal, self-healing outcome. +If a re-defer fails, the worker keeps the run untouched and emits a warning with +`event: "defer_failed"`, `messageId`, and `queueName`. Count this event in worker +logs to detect repeated queue visibility or pop-receipt failures. + The default threshold is `2 × HEARTBEAT_VISIBILITY_SECONDS` (= 120s). Override with the `SCOPE_RUN_HEARTBEAT_STALE_MS` env var (milliseconds). The Redis TTL defaults to `5 × HEARTBEAT_VISIBILITY_SECONDS` (= 300s); override with `SCOPE_RUN_HEARTBEAT_REDIS_TTL_MS`. ### Stuck-Run Reaper (scheduler backstop) diff --git a/packages/shared/src/queue/base-queue-processor.test.ts b/packages/shared/src/queue/base-queue-processor.test.ts index 00d485be4..ed30951c4 100644 --- a/packages/shared/src/queue/base-queue-processor.test.ts +++ b/packages/shared/src/queue/base-queue-processor.test.ts @@ -300,9 +300,15 @@ describe("BaseQueueProcessor.safeDeferMessage", () => { it("swallows updateMessage failures (never throws)", async () => { const updateMessage = vi.fn().mockRejectedValue(new Error("stale receipt")); const proc = makeProc(updateMessage); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); await expect( (proc as any).safeDeferMessage("msg-1", "pop-1", 120), ).resolves.toBeUndefined(); + expect(warn).toHaveBeenCalledWith( + expect.stringContaining("stale receipt"), + { event: "defer_failed", messageId: "msg-1", queueName: testConfig.queueName }, + ); + warn.mockRestore(); }); }); diff --git a/packages/shared/src/queue/base-queue-processor.ts b/packages/shared/src/queue/base-queue-processor.ts index 08bc35d51..22181603e 100644 --- a/packages/shared/src/queue/base-queue-processor.ts +++ b/packages/shared/src/queue/base-queue-processor.ts @@ -480,7 +480,11 @@ export abstract class BaseQueueProcessor