fix(realtime): widen sessions-plane coalescing to cap signal storms - #743
Merged
Conversation
A streaming session (own pushes or a teammate's) bumps the org sessions signal about once per second, and every broadcast re-listed the org and pulled collab state through the 750ms coalescer -- the cmd+5 panel showed cloud_list_org_sessions at 38/min during any active synced session. The sessions plane now coalesces at 15s: an isolated change still refreshes instantly on the leading edge, a sustained storm costs 4 bounded delta pulls per minute instead of ~40. No new recurring timers; idle stays at zero calls.
The comments broadcast handler bumped the org listing directly, bypassing the plane coalescer -- a comment/task storm could still re-list the org per broadcast. The listing bump now rides the same 15s sessions plane via a shared full-refresh callback (the coalescer keeps the first-armed refresh per window, so every entry point must request the same work). The open thread's own signal stays immediate; the rare control-plane sessions event keeps its immediate bump.
Five-domain audit against the PR #743 call-discipline standard (no polling, storm-coalesced, idle-silent, echo-aware, focus-bounded, bounded retry). 6 violations (3 hand-verified), 12 watch items, and keep-with-reason records for the deliberate exceptions. Audit only -- no source changes in this commit.
Audit P0 batch (docs/call-hotspot-audit-2026-08-07):
- Peer ORG_CONTROL_CHANGED{sessions} broadcasts (sent after every
successful peer push, 250ms sender collapse only) now ride the 15s
sessions plane instead of bumping the listing immediately -- one
streaming teammate no longer re-lists the org 20-40x/min on every
receiver.
- member_runtime signals are recognized and routed to their own plane:
a teammate's telemetry heartbeat no longer triggers the coarse
full-plane fallback, and the Team Runtime roster now refreshes live
off the new per-org version atom instead of waiting for a remount
(visible-edge refetch also gains a 30s cooldown).
- Ordinary realtime invalidations no longer clear the org entitlement
backoff -- only full recovery (resumeOrg / edge recovery / online)
does, restoring the documented 5/30-minute cool-down contract. The
test pinning the old behavior now pins the new contract.
Audit P1 batch (docs/call-hotspot-audit-2026-08-07): - The 15s storm window (renamed STORM_SIGNAL_COALESCE_MS) now covers every plane whose refresh is a full listing or multi-RPC pass: comments, inbound (projects/workItems), channels, and the coarse legacy/safety-net bundle. channelMessages keeps 750ms deliberately (bounded live-chat delta); roster/policy stay short (admin-paced, entitlement already TTL'd). - Team Inbox refresh gains a 15s floor with a trailing catch-up: the permanently-mounted dual-source listing no longer re-lists per invalidation burst, and the newest request version is never dropped. - Focus/visible-edge refetches get a shared 30s cooldown (FOCUS_REFRESH_COOLDOWN_MS): remote-sessions full recovery, channels force-fresh re-list, roster convergence, and (previous commit) the team runtime roster. The 5-min convergence timer still owns guaranteed catch-up; channels' force-fresh in-flight eviction stays (correctness: a stale listing must not reach tab reconciliation).
Audit P2 batch (docs/call-hotspot-audit-2026-08-07): - runCoarseSignalRefresh checks hidden BEFORE marking planes handled (same reorder the edge-recovery path got in fbc6cd8). - Comments error retry and the builder-profile extraction drain stay network/IPC-silent while the document is hidden. - Member-runtime passes share the burst sample, daily rollup, and profile read across due orgs (agents probe already was shared). - The activity/data-changed trailing debounces gain max-wait deadlines (15s native / 60s imported / 15s projects): continuous sub-window activity batches pushes instead of starving them. - Shareable-scope transport failures get a 30s negative cache so render-path callers stop re-firing the git-remotes IPC per re-render while the git backend is down. - Stale doc comments referencing the removed 60s pass and the absent comment-task runner are corrected. Deferred (needs a Rust event-payload change, separate PR): the org-less orgii-data-changed signal still fans listOrgCollabState across all orgs per projects pass.
set_window_focus exonerated (single-mount, deduped, flag-flip; trace prominence = window-switch fan-out behind it). Two IPC violations found (MiniCpm popover 5s poll without hidden gate; launchpad triple focus FS-scan without coalescing). External-history plane clean by design (timer-polled deliberately, #608 re-hash guard verified current); also corrects the original storm attribution -- live CLI sessions push once per scanner tick, so the 38/min source was native in-app session traffic, not the imported CLI session.
Instance 2 (vinceorz418) streamed a real agent round in a shared org while Instance 1 ran the patched build. The peer-broadcast path is self:false, so this is the only way to exercise P0 finding #1. Receiver listing rate held at 1.5/min (3 calls/2min) while the peer's append rate varied 6.5-9.0/min -- pre-fix this scenario produced one immediate listing per peer push (20-40/min). Destructive-effect audit over the storm window: 3 epoch rewrites, all explained as first- selection re-anchors in the newly switched org (one chainMismatch); zero retract/delete/vanish, zero ERROR, zero watchdog fires. Cloud ledger diff and the two-boot/fault-injection cells recorded as UNCOVERED.
- MiniCpmCompactCard's raw 5s setInterval becomes a visibility-aware poller: a hidden window no longer fires the compaction-status IPC indefinitely while the popover stays mounted. - The three launchpad discovery hooks share a flap-cooled focus rescan (30s per hook) instead of three unthrottled parallel FS-scan IPCs on every window focus. - useWorkspaceGitStatus's editor-websocket wait backs off 1s -> 30s cap instead of retrying every second forever in workspaces where no editor ever opens.
Collaborator
Author
|
Wave-2 findings also fixed in 68ecf58: MiniCpm popover 5s poll is now visibility-aware, the launchpad's three focus FS-scans share a 30s flap cooldown, and the workspace git-status websocket wait backs off 1s→30s. Full audit trail: docs/call-hotspot-audit-2026-08-07/ (5-domain sweep + wave-2 IPC/scanner reports + the dual-instance storm cell that verified the receiver cap at 1.5/min under a live teammate stream). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Any actively-streaming synced session (own pushes or a teammate's) bumps the org
sessionschange signal about once per second (server-side per-kind debounce is 1s). Every broadcast ran the sessions-plane refresh through the 750ms coalescer, re-listing the org and pulling collab state on nearly every signal — cmd+5 showed 38/min sustained. A follow-up call-discipline audit of the entire session-sharing + runtime surface (reports indocs/call-hotspot-audit-2026-08-07/) found the remaining storm paths; this PR closes all of them.Commits
c1c60d7 / ab7109b — the original fix: per-call coalescer window; sessions db-change plane and comment-broadcast listing bumps coalesce at 15s (leading edge keeps isolated changes instant; storms cap at 4 pulls/min).
5e15987 — audit reports: five-domain audit against the call-discipline standard (no polling / storm-coalesced / idle-silent / echo-aware / focus-bounded / bounded retry). 6 violations, 12 watch items, keep-with-reason records.
7bd7196 — P0:
ORG_CONTROL_CHANGED {sessions}broadcasts (sent after every successful peer push; only 250ms sender collapse;self:falseso single-machine testing never showed it) now ride the 15s sessions plane — one streaming teammate no longer re-lists the org 20-40×/min on every receiver. This was the surviving half of the original 38/min bug.member_runtimesignals are recognized (they previously fell through to the FULL coarse multi-plane fallback on every teammate heartbeat) and routed to their own plane; the Team Runtime roster consumes the new per-org version atom and refreshes live.f3af84a — P1:
STORM_SIGNAL_COALESCE_MS) extended to every full-listing/multi-RPC plane: comments, inbound (projects/workItems), channels, coarse (legacy + safety net).channelMessagesdeliberately keeps 750ms (bounded live-chat delta); roster/policy stay short (admin-paced; entitlement already 10s-TTL'd).FOCUS_REFRESH_COOLDOWN_MS): remote-sessions full recovery, channels re-list, roster convergence, team-runtime roster.0829a9b — P2: coarse-refresh hidden check before markHandled; comments retry + builder-profile drain hidden-silent; member-runtime collectors (burst sample / 35-day rollup / profile) shared across due orgs per pass; max-wait deadlines (15s/60s/15s) so trailing debounces batch instead of starve; 30s negative cache for shareable-scope transport failures; stale doc comments corrected.
Deferred (cross-layer, separate PR): the org-less
orgii-data-changedTauri event still fanslistOrgCollabStateacross all orgs per projects pass — needs a Rust payload change.Verification
Worst-case sustained load under a full multi-plane storm is now bounded at ~4 refreshes/min per plane, idle remains zero calls and zero timers, and the only recurring timer in the whole plane stays the deliberate 5-min visible-only roster convergence net.