Skip to content

Answer a rate-limited EVENT with OK false and retry it once the gate clears - #6589

Open
tlongwell-block wants to merge 2 commits into
mainfrom
meli/first-send-rate-limit
Open

Answer a rate-limited EVENT with OK false and retry it once the gate clears#6589
tlongwell-block wants to merge 2 commits into
mainfrom
meli/first-send-rate-limit

Conversation

@tlongwell-block

@tlongwell-block tlongwell-block commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

The first message after switching communities in the desktop app (and for some users, the first message after a just production cold launch) spins, fails with "Timed out while sending the message." after 25s, then works on the second try.

Reported by Tyler in #buzz-bugs; root cause and live reproduction in RESEARCH/FIRST_SEND_AFTER_COMMUNITY_SWITCH.md (Meli's workspace).

Root cause

  1. A community switch remounts <AppReady key={communityKey}>, so every relay hook re-subscribes on the fresh socket. useLiveChannelUpdates alone sends 2 REQs per sidebar channel (useLiveChannelUpdates.ts:393,497) plus ~10 singleton REQs. The initial subscribe() path is unpaced.
  2. Relay WS admission bills REQ and EVENT frames against the same per-pubkey window: human_ws_events_per_sec (10) × WS_BURST_WINDOW_SECS (5) = 50 frames / 5s (crates/buzz-relay/src/admission.rs). ≳20 channels blows the budget on every switch.
  3. The rejection was asymmetric (connection.rs send_admission_result): an over-quota REQ got CLOSED <subId> rate-limited:… (correlated → client retries it). An over-quota EVENT got a bare NOTICE rate-limited:… — uncorrelated. The desktop client armed its gate but pendingEvents[id] never received an OK, so it sat until PUBLISH_TIMEOUT_MS = 25s.
  4. The retry lands after the window rolls → success.

Live proof against the prod relay with my agent key (.scratch/firstsend-probe.mjs, node + nostr-tools):

10 REQ + 1 EVENT → OK false "restricted: not a channel member"     ← admission passed
60 REQ + 1 EVENT → NOTICE "rate-limited: quota exceeded; retry in 3s"; no OK for the EVENT, ever

Fix (both halves, one PR — per Tyler)

Relay — crates/buzz-relay/src/connection.rs

request_rejection_message(Option<&str>, …)request_rejection_message(RejectionTarget, …). An EVENT refused by admission (both Exceeded and Unavailable) or by the handler semaphore is now answered with ["OK", <event_id>, false, "rate-limited: …"]. REQ keeps CLOSED; COUNT keeps NOTICE. The rate-limited: prefix and retry in Ns hint are unchanged, so every client's existing parser still applies. With only this half, the desktop fails in ~RTT with a real reason instead of spinning 25s.

Desktop — desktop/src/shared/api/relayPublishRecovery.ts (handlePublishOk, called from relayClientSession.ts handleOk)

On OK false rate-limited: arm the gate and re-send the EVENT once the gate clears instead of rejecting the publish — the relay dedups by id (insert_event is ON CONFLICT DO NOTHING, duplicate → OK true "duplicate:"), so a resend can never double-post. Exactly one retry per publish (PendingEvent.retriedAfterRateLimit); a second refusal, or a gate ≥ PUBLISH_TIMEOUT_MS, fails fast with the relay's reason. The retry is guarded by identity on the pendingEvents entry, so a timeout/reset/community switch during the wait cancels it. The uncorrelated NOTICE rate-limited: still arms the gate but never retries: a NOTICE may have been caused by a REQ/COUNT, and an ephemeral event (e.g. presence kind 20001, which goes through publishEvent) that the relay already fanned out has no id dedup, so a blanket resend could double-deliver (Max's review finding). With this half the first send succeeds (≈ gate duration later) rather than failing.

buzz-acp — crates/buzz-acp/src/relay.rs

Previously an OK false rate-limited: fell through to acknowledge_observer_frame, silently dropping the refused observer frame from the in-flight window (no requeue, no gate). Now: arm the gate and put that one frame back at the head of the paced drain (requeue_observer_frame, correlated by id — narrower than the NOTICE path's requeue-everything).

Mobile — mobile/lib/shared/relay/relay_session.dart

_handleOk arms the gate on OK false rate-limited: before failing the publish, so concurrent work backs off. (Mobile's 8s publish timeout and simpler send path didn't exhibit the 25s spin; this is consistency.)

Unchanged

CLI publishes via REST (429 path) — not affected.

Tests

Where What Fail-first?
connection.rs rejections_are_correlated_with_the_rejected_frame EVENT→OK false / REQ→CLOSED / COUNT→NOTICE shapes replaces req_rejections_are_subscription_scoped
relayClientSession.publishRateLimit.test.mjs (5 tests, fake timers, driven through handleWsMessage) retry once gate clears; second refusal fails fast + bounded; non-rate-limit OK=false rejects immediately; uncorrelated NOTICE arms the gate but never resends; reset during the wait cancels the retry ✅ 3/5 failed before the change (non-rate-limit and NOTICE paths are unchanged behavior)
relay_rate_limit_gate_test.dart activateIfRateLimited arms only on a back-pressure message the shared classify→arm helper that replaced three copies in relay_session.dart new
relay.rs rate_limited_ok_requeues_only_the_refused_frame only the refused observer frame moves to the drain head; unrefused stays in flight new
relay_session_test.dart rate-limited OK=false fails the publish and arms the gate gate armed with the 4s hint ✅ verified failing with lib change stashed

Suites at 43c5dc8d2 (review follow-up — desktop retry extracted to relayPublishRecovery.ts, NOTICE retry dropped, mobile classify→arm deduped; both oversized session files now net-shrink under the file-size ratchet: relayClientSession.ts 1084→1078, relay_session.dart 1000→995): desktop pnpm test 5402/5402, tsc --noEmit clean, biome clean; just file-size-check green (desktop/web/mobile); mobile flutter test 1663/1663, flutter analyze clean, dart format clean. Rust crates untouched since e3a3247b1.

Suites at e3a3247b1: cargo test -p buzz-acp 802/802; cargo test -p buzz-relay 905 pass + 2 fail — both fail identically on clean origin/main e23632941 (api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo known flaky; telemetry::tests::trace_context_lookup_does_not_enable_callsites order-dependent, passes in isolation). cargo clippy -p buzz-relay -p buzz-acp --all-targets -D warnings clean; cargo fmt --check clean. Desktop pnpm test 5402/5402, tsc --noEmit clean, biome clean. Mobile flutter test test/shared/relay/ 44/44, dart analyze clean.

Notes for reviewers

  • Overlaps open fix(relay,acp,cli): rate-limit overhaul — ephemeral kinds off the Messages quota, limit_type discriminator, client retry fixes #4912 (Eva/Sami, eva/rate-limit-fixes), which also edits send_admission_result (adds a limit_type arg). Whichever lands second has a small textual conflict in that function; the RejectionTarget parameter composes with their change.
  • Behavior I'm not fixing here (follow-up): the REQ fan-out itself. Coalescing per-channel live+mention REQs into multi-filter frames would keep startup under 50 frames and make the rate limiter a non-event. Separate PR.
  • Not in the PR: the desktop e2e relay-reconnect*.spec.ts harness has no rate-limit fixture; the unit tests above drive the real handleWsMessagehandleOk → gate → sendRaw path with fake timers instead.

…clears

The first message after a community switch (or a cold launch) spun for
25s, failed with "Timed out while sending the message", then worked on
the second try. The relay's WS admission window (50 frames / 5s per
pubkey) counts REQ and EVENT together, and the post-(re)connect REQ
fan-out alone can exhaust it. An over-quota REQ already got a correlated
`CLOSED <subId> rate-limited:`, but an over-quota EVENT got a bare
`NOTICE rate-limited:` — nothing ever settled the publisher's pending
OK, so it sat until PUBLISH_TIMEOUT_MS.

Relay (`connection.rs`): replace the `Option<sub_id>` passed into
`request_rejection_message` with a `RejectionTarget` so an EVENT refused
by admission or the handler semaphore is answered with
`["OK", <id>, false, "rate-limited: …"]`. REQ keeps CLOSED; COUNT keeps
NOTICE. The `rate-limited:` prefix and `retry in Ns` hint are unchanged
so every client's existing parser still applies.

Desktop (`relayClientSession.ts`): on `OK false rate-limited:` arm the
gate and re-send the EVENT once it clears instead of rejecting the
publish — the relay dedups by id, so the resend cannot double-post.
Exactly one retry per publish; a second refusal, or a gate longer than
the publish timeout, fails fast with the relay's reason. The NOTICE
branch gives in-flight publishes the same single retry so the fix also
holds against relays that predate this change.

buzz-acp (`relay.rs`): an `OK false rate-limited:` used to fall through
to `acknowledge_observer_frame`, silently dropping the refused observer
frame from the in-flight window. Arm the gate and put that one frame
back at the head of the paced drain instead.

Mobile (`relay_session.dart`): an `OK false rate-limited:` now arms the
gate before failing the publish, so concurrent work backs off.

Tests: relay frame-shape unit test; desktop fail-first unit tests for
the retry, its bound, non-rate-limit rejection, legacy NOTICE, and a
reset during the wait; acp requeue-only-the-refused-frame; mobile
gate-armed-on-OK-false (verified failing before the lib change).

Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 23, 2026 00:52
Review follow-up for the first-send-after-community-switch fix.

Desktop: move the OK-frame settle/retry logic from relayClientSession.ts
(already over the file-size ratchet; grew 1084 -> 1129) into a pure
`handlePublishOk` in relayPublishRecovery.ts, taking the pending map and
a send callback. The session file now shrinks to 1078.

Drop the uncorrelated NOTICE retry-all. A `NOTICE rate-limited:` may have
been triggered by a REQ or COUNT, and an ephemeral event (e.g. presence,
kind 20001) the relay already fanned out has no id dedup, so resending
every in-flight publish could double-deliver. Only the correlated
`OK false rate-limited:` is retried; the NOTICE still arms the gate.
The test now asserts NOTICE never resends, and the reset test stubs the
Tauri websocket disconnect so it runs without a stderr TypeError.

Mobile: relay_session.dart tripped the same ratchet (1000 -> 1005). The
classify-then-arm sequence appeared three times, so it becomes
`RelayRateLimitGate.activateIfRateLimited(message)`; the session file
drops to 995. Gate unit test added.

Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant