Summary
On Maincloud, when one transaction's update fans out to 50–230 WebSocket subscribers at once (~10–90 MB in total), some connections stop receiving bytes for ~7.5 / 15 / 31 / 63 s after the reducer call has returned, then receive the rest in one burst. The gaps match TCP retransmission backoff, which suggests repeated packet loss somewhere on the Maincloud side (front end or host egress).
Why we think it is platform-side
- It happened at the same moments, with the same worst cases, to clients on three independent vantages: two Linux hosts on one ISP, and Cloudflare Durable Objects using raw WebSockets (no SDK). Example: 45.1 / 45.0 / 45.0 s on the same transition.
- A frozen connection receives 12–160 KB of the update, then zero bytes for 7.5, 15 or 30 s, then the remainder. The client event loop lagged ≤ 21 ms and the client kernel dropped nothing; bytes stop arriving before the client reads them.
- Same behaviour with confirmed reads on and off.
- Light traffic is clean: 28,440 small deliveries peaked at 438 ms. Once, 2 KB updates froze ~28 s about 2 minutes after a heavy run, so the effect seems to linger.
Minimal repro (SpacetimeDB 2.10.1, TypeScript module)
- One public table
beat { id: u32 (pk), n: u64, pad: string, at: timestamp }.
- Reducer
set_beat(n: u64, padLen: u32) overwrites row 1 with a pad of padLen bytes.
- 230 subscribers to
SELECT * FROM beat.
- Call
set_beat(n, 200000) every 8 s.
- Result: last-subscriber lag up to ~61 s, and ~25 % of deliveries arrived more than 2.5 s late.
The repro database is coinsteam-staging-stall-probe on Maincloud; we can keep it up for you. Happy to share the probe scripts and raw timings.
Questions
- Is there burst shaping or policing on Maincloud's WebSocket front end or host egress?
- Is this a known issue, and is there a recommended limit on per-transaction fan-out bytes?
Why it matters to us: we rely on subscription updates to remove revoked data from users' screens within a bounded time. A frozen connection also delays those deletions.
Summary
On Maincloud, when one transaction's update fans out to 50–230 WebSocket subscribers at once (~10–90 MB in total), some connections stop receiving bytes for ~7.5 / 15 / 31 / 63 s after the reducer call has returned, then receive the rest in one burst. The gaps match TCP retransmission backoff, which suggests repeated packet loss somewhere on the Maincloud side (front end or host egress).
Why we think it is platform-side
Minimal repro (SpacetimeDB 2.10.1, TypeScript module)
beat { id: u32 (pk), n: u64, pad: string, at: timestamp }.set_beat(n: u64, padLen: u32)overwrites row 1 with apadofpadLenbytes.SELECT * FROM beat.set_beat(n, 200000)every 8 s.The repro database is
coinsteam-staging-stall-probeon Maincloud; we can keep it up for you. Happy to share the probe scripts and raw timings.Questions
Why it matters to us: we rely on subscription updates to remove revoked data from users' screens within a bounded time. A frozen connection also delays those deletions.