Skip to content

fix: let reset_peer be interrupted while waiting for its timeout - #787

Open
soujasK wants to merge 1 commit into
Shopify:mainfrom
soujasK:fix/reset-peer-interrupt
Open

soujasK wants to merge 1 commit into
Shopify:mainfrom
soujasK:fix/reset-peer-interrupt

Conversation

@soujasK

@soujasK soujasK commented Oct 25, 2026

Copy link
Copy Markdown

Problem

Removing a reset_peer toxic after it has received data blocks for the rest of its timeout, and so does adding another toxic after it in the same direction. The collection lock is held the whole time, so with a timeout over 25s the API request returns 503 and other toxic requests for that proxy block until the timer fires.

Root cause

After reading the first chunk, ResetToxic.Pipe waits with a bare <-time.After(timeout). InterruptToxic blocks until the running toxic receives from Interrupt, which reset_peer doesn't select on while waiting, so RemoveToxic / AddToxic / UpdateToxic wait for the timeout.

Solution

Select on Interrupt while waiting, and return without resetting when interrupted, the same way slow_close, timeout and latency handle interrupts during their waits.

Verification

  • New TestRemoveResetPeerToxicDoesNotWaitForTimeout: a reset_peer with a 10s timeout receives a chunk, then is removed. On main the removal is still blocked after 2s; with the change it returns right away.
  • Linux: go test -race for the root package and ./toxics -run Reset (3 runs) pass; golangci-lint v2.14.0: 0 issues.

Compatibility and risks

Removing a reset_peer while its reset is pending now cancels the reset, instead of resetting the connection after the API call has been stuck for the timeout.

If another toxic is added or updated while a reset is pending, reset_peer is restarted and waits for the next chunk before starting its timeout again, so the reset is postponed until more data arrives. The other timer-based toxics behave the same way after an interrupt. Keeping the original deadline across restarts would need per-link state; I left that out to keep this small.

May address some of the reports in #500.

@soujasK
soujasK force-pushed the fix/reset-peer-interrupt branch from 8aecdd6 to 2f3e64f Compare October 26, 2026 10:41
After the first chunk, reset_peer waited on time.After without selecting on
Interrupt, so removing it (or adding a toxic after it) blocked for the whole
timeout while holding the toxic collection's lock. With a timeout over the
API's 25s limit the request failed with 503.
@soujasK
soujasK force-pushed the fix/reset-peer-interrupt branch from 2f3e64f to eb1137a Compare October 26, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant