fix(pegboard-envoy): retry failed actor event dispatch instead of dropping events

[eersnington]

Retry a failed actor event dispatch instead of dropping every later event for that actor.

 channel_handler
   recv batch
-  dispatch_events() fails → log, break     // later events dropped, including "stopped"
+  dispatch_events() fails → log, backoff, retry same batch
+  cancelled on shutdown → return           // envoy resends unacked events on reconnect

• Problem: In actor_event_demuxer.rs, one dispatch_events error (for example UDB "max number of transaction retries reached") ended the handler loop. The actor never got its "stopped" event, so every later wake waited forever.
• Fix: Retry the same batch, in order, with Backoff::new(5, None, 250, 250) until it lands. This is safe because the actor2 workflow ignores events it already received (dedupe on generation and index).
• shutdown() cancels the token after it drops the senders. Handlers still dispatch buffered events, and a handler that is retrying exits.
• dispatch_events takes &[EventWrapper] so the batch can be retried.
• The test runs pegboard-envoy against Postgres through Toxiproxy, cuts storage for 2 seconds while events are sent, and checks that all events reach the actor workflow.

[railway-app]

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

[claude]

Review: retry failed actor event dispatch

The approach is sound. Retrying the same batch in order fixes the dropped "stopped" bug, and graceful_not_found already covers the "workflow gone" case. A few things to consider:

Potential issues

1. Ordering across handler replacement. The idle-channel cleanup (retain on last_seen) drops the sender. A handler stuck in its retry loop keeps running after that. If the same actor sends another event, ingest creates a new channel and handler, and two handlers can dispatch for one actor concurrently. Before this PR a failed handler exited, so this did not arise. During a long outage this could deliver events out of order. Consider not evicting channels whose handler is mid-retry.
2. Unbounded retry on permanent errors. Every Err is retried forever at 250ms. That is right for transient UDB failures, but a deterministic error would wedge that actor channel and grow its buffer. Consider classifying errors or escalating the log level after N attempts. The warn! repeats every 250ms for the whole outage, so rate-limiting it would help.
3. Backpressure. Check whether the channel behind ingest is bounded. If it is unbounded, memory grows during an outage.
4. Confirm Backoff::new(5, None, 250, 250) gives a flat 250ms and that tick() does not return immediately once its retry count is exhausted.

Tests

• The Toxiproxy/Postgres test is thorough, but it includes ../src/actor_event_demuxer.rs through #[path] and re-creates a metrics shim. That is fragile if the module gains more crate:: dependencies. Exposing the demuxer from the lib would be more robust.
• The 2s sleep is what guarantees event 1 dispatch failed. Asserting on an actual failed attempt would be more deterministic.
• unsafe { set_var } in a multi-thread tokio test is racy if more tests are added to the binary.

Nits

• events.to_vec() clones the batch on every attempt, including the first. This is acceptable.
• In shutdown(), the order of dropping senders and then cancel() matters for the drain semantics, so a short comment saying so would help.

No security concerns. Overall this looks good, and (1) is the one I would address before merging.

[claude]

Review

The fix is correct and small. Retrying the same batch in order stops one transient UDB error from wedging an actor's event stream, and cancelling the token after the senders drop keeps buffered events draining.

1. Eviction can race with a retrying handler. The idle sweep drops the sender but not the task. A handler stuck retrying through an outage longer than max_last_seen can be evicted, and the next ingest creates a new handler that may dispatch later events (including "stopped") before the old one lands earlier ones. This breaks the in-order guarantee. Consider not evicting channels that are mid-retry.

2. Unbounded retry on non-transient errors. Backoff::new(5, None, ...) retries forever for any signal-send error. A deterministic failure would block that actor's channel permanently with a warn every ~8s. Consider retrying only storage errors, or escalating to error! after N attempts.

3. Minor

• The first tick() does not sleep, so the first retry is immediate.
• events.to_vec() clones the batch each attempt, including on success.
• pub mod actor_event_demuxer widens the public surface for the test. #[doc(hidden)] would signal intent.

4. Test. Good end-to-end repro with real Postgres and Toxiproxy. Not covered: the eviction case in point 1, and the shutdown-while-retrying path. set_var is only safe while no other test in the binary runs concurrently, so a comment would help.

Overall this looks good. Point 1 is the one I would most like addressed or acknowledged.