Answer a rate-limited EVENT with OK false and retry it once the gate clears - #6589
Open
tlongwell-block wants to merge 2 commits into
Open
Answer a rate-limited EVENT with OK false and retry it once the gate clears#6589tlongwell-block wants to merge 2 commits into
tlongwell-block wants to merge 2 commits into
Conversation
…clears The first message after a community switch (or a cold launch) spun for 25s, failed with "Timed out while sending the message", then worked on the second try. The relay's WS admission window (50 frames / 5s per pubkey) counts REQ and EVENT together, and the post-(re)connect REQ fan-out alone can exhaust it. An over-quota REQ already got a correlated `CLOSED <subId> rate-limited:`, but an over-quota EVENT got a bare `NOTICE rate-limited:` — nothing ever settled the publisher's pending OK, so it sat until PUBLISH_TIMEOUT_MS. Relay (`connection.rs`): replace the `Option<sub_id>` passed into `request_rejection_message` with a `RejectionTarget` so an EVENT refused by admission or the handler semaphore is answered with `["OK", <id>, false, "rate-limited: …"]`. REQ keeps CLOSED; COUNT keeps NOTICE. The `rate-limited:` prefix and `retry in Ns` hint are unchanged so every client's existing parser still applies. Desktop (`relayClientSession.ts`): on `OK false rate-limited:` arm the gate and re-send the EVENT once it clears instead of rejecting the publish — the relay dedups by id, so the resend cannot double-post. Exactly one retry per publish; a second refusal, or a gate longer than the publish timeout, fails fast with the relay's reason. The NOTICE branch gives in-flight publishes the same single retry so the fix also holds against relays that predate this change. buzz-acp (`relay.rs`): an `OK false rate-limited:` used to fall through to `acknowledge_observer_frame`, silently dropping the refused observer frame from the in-flight window. Arm the gate and put that one frame back at the head of the paced drain instead. Mobile (`relay_session.dart`): an `OK false rate-limited:` now arms the gate before failing the publish, so concurrent work backs off. Tests: relay frame-shape unit test; desktop fail-first unit tests for the retry, its bound, non-rate-limit rejection, legacy NOTICE, and a reset during the wait; acp requeue-only-the-refused-frame; mobile gate-armed-on-OK-false (verified failing before the lib change). Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Review follow-up for the first-send-after-community-switch fix. Desktop: move the OK-frame settle/retry logic from relayClientSession.ts (already over the file-size ratchet; grew 1084 -> 1129) into a pure `handlePublishOk` in relayPublishRecovery.ts, taking the pending map and a send callback. The session file now shrinks to 1078. Drop the uncorrelated NOTICE retry-all. A `NOTICE rate-limited:` may have been triggered by a REQ or COUNT, and an ephemeral event (e.g. presence, kind 20001) the relay already fanned out has no id dedup, so resending every in-flight publish could double-deliver. Only the correlated `OK false rate-limited:` is retried; the NOTICE still arms the gate. The test now asserts NOTICE never resends, and the reset test stubs the Tauri websocket disconnect so it runs without a stderr TypeError. Mobile: relay_session.dart tripped the same ratchet (1000 -> 1005). The classify-then-arm sequence appeared three times, so it becomes `RelayRateLimitGate.activateIfRateLimited(message)`; the session file drops to 995. Gate unit test added. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first message after switching communities in the desktop app (and for some users, the first message after a
just productioncold launch) spins, fails with "Timed out while sending the message." after 25s, then works on the second try.Reported by Tyler in #buzz-bugs; root cause and live reproduction in
RESEARCH/FIRST_SEND_AFTER_COMMUNITY_SWITCH.md(Meli's workspace).Root cause
<AppReady key={communityKey}>, so every relay hook re-subscribes on the fresh socket.useLiveChannelUpdatesalone sends 2 REQs per sidebar channel (useLiveChannelUpdates.ts:393,497) plus ~10 singleton REQs. The initialsubscribe()path is unpaced.human_ws_events_per_sec(10) ×WS_BURST_WINDOW_SECS(5) = 50 frames / 5s (crates/buzz-relay/src/admission.rs). ≳20 channels blows the budget on every switch.connection.rssend_admission_result): an over-quota REQ gotCLOSED <subId> rate-limited:…(correlated → client retries it). An over-quota EVENT got a bareNOTICE rate-limited:…— uncorrelated. The desktop client armed its gate butpendingEvents[id]never received anOK, so it sat untilPUBLISH_TIMEOUT_MS= 25s.Live proof against the prod relay with my agent key (
.scratch/firstsend-probe.mjs, node + nostr-tools):Fix (both halves, one PR — per Tyler)
Relay —
crates/buzz-relay/src/connection.rsrequest_rejection_message(Option<&str>, …)→request_rejection_message(RejectionTarget, …). An EVENT refused by admission (bothExceededandUnavailable) or by the handler semaphore is now answered with["OK", <event_id>, false, "rate-limited: …"]. REQ keepsCLOSED; COUNT keepsNOTICE. Therate-limited:prefix andretry in Nshint are unchanged, so every client's existing parser still applies. With only this half, the desktop fails in ~RTT with a real reason instead of spinning 25s.Desktop —
desktop/src/shared/api/relayPublishRecovery.ts(handlePublishOk, called fromrelayClientSession.tshandleOk)On
OK false rate-limited:arm the gate and re-send the EVENT once the gate clears instead of rejecting the publish — the relay dedups by id (insert_eventisON CONFLICT DO NOTHING, duplicate →OK true "duplicate:"), so a resend can never double-post. Exactly one retry per publish (PendingEvent.retriedAfterRateLimit); a second refusal, or a gate ≥PUBLISH_TIMEOUT_MS, fails fast with the relay's reason. The retry is guarded by identity on thependingEventsentry, so a timeout/reset/community switch during the wait cancels it. The uncorrelatedNOTICE rate-limited:still arms the gate but never retries: a NOTICE may have been caused by a REQ/COUNT, and an ephemeral event (e.g. presence kind 20001, which goes throughpublishEvent) that the relay already fanned out has no id dedup, so a blanket resend could double-deliver (Max's review finding). With this half the first send succeeds (≈ gate duration later) rather than failing.buzz-acp —
crates/buzz-acp/src/relay.rsPreviously an
OK false rate-limited:fell through toacknowledge_observer_frame, silently dropping the refused observer frame from the in-flight window (no requeue, no gate). Now: arm the gate and put that one frame back at the head of the paced drain (requeue_observer_frame, correlated by id — narrower than the NOTICE path's requeue-everything).Mobile —
mobile/lib/shared/relay/relay_session.dart_handleOkarms the gate onOK false rate-limited:before failing the publish, so concurrent work backs off. (Mobile's 8s publish timeout and simpler send path didn't exhibit the 25s spin; this is consistency.)Unchanged
CLI publishes via REST (429 path) — not affected.
Tests
connection.rsrejections_are_correlated_with_the_rejected_framereq_rejections_are_subscription_scopedrelayClientSession.publishRateLimit.test.mjs(5 tests, fake timers, driven throughhandleWsMessage)relay_rate_limit_gate_test.dartactivateIfRateLimited arms only on a back-pressure messagerelay_session.dartrelay.rsrate_limited_ok_requeues_only_the_refused_framerelay_session_test.dartrate-limited OK=false fails the publish and arms the gateSuites at
43c5dc8d2(review follow-up — desktop retry extracted torelayPublishRecovery.ts, NOTICE retry dropped, mobile classify→arm deduped; both oversized session files now net-shrink under the file-size ratchet:relayClientSession.ts1084→1078,relay_session.dart1000→995): desktoppnpm test5402/5402,tsc --noEmitclean, biome clean;just file-size-checkgreen (desktop/web/mobile); mobileflutter test1663/1663,flutter analyzeclean,dart formatclean. Rust crates untouched sincee3a3247b1.Suites at
e3a3247b1:cargo test -p buzz-acp802/802;cargo test -p buzz-relay905 pass + 2 fail — both fail identically on cleanorigin/main e23632941(api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echoknown flaky;telemetry::tests::trace_context_lookup_does_not_enable_callsitesorder-dependent, passes in isolation).cargo clippy -p buzz-relay -p buzz-acp --all-targets -D warningsclean;cargo fmt --checkclean. Desktoppnpm test5402/5402,tsc --noEmitclean, biome clean. Mobileflutter test test/shared/relay/44/44,dart analyzeclean.Notes for reviewers
eva/rate-limit-fixes), which also editssend_admission_result(adds alimit_typearg). Whichever lands second has a small textual conflict in that function; theRejectionTargetparameter composes with their change.relay-reconnect*.spec.tsharness has no rate-limit fixture; the unit tests above drive the realhandleWsMessage→handleOk→ gate →sendRawpath with fake timers instead.