Skip to content

fix: close writeDone before RemoveLink to avoid a ToxicDelete deadlock - #785

Open
soujasK wants to merge 1 commit into
Shopify:mainfrom
soujasK:fix/write-done-deadlock
Open

soujasK wants to merge 1 commit into
Shopify:mainfrom
soujasK:fix/write-done-deadlock

Conversation

@soujasK

@soujasK soujasK commented Oct 25, 2026

Copy link
Copy Markdown

Problem

If a client stops reading while a toxic is being removed, and then disconnects, the proxy's toxic API stays stuck for good: the DELETE returns 503 after the 25s TimeoutHandler, and every later request for that proxy (GET /proxies/{proxy}, adding/removing toxics) keeps returning 503 until the server is restarted. This matches what's described in #500.

Reproduced end to end against main with the server binary: a proxy with a latency toxic, an upstream that keeps sending, and a client that connects but never reads:

step before after
DELETE /proxies/p/toxics/lat while the client is stalled 503 (25s) 503 (25s)
GET /proxies/p 5s / 10s / 15s / 20s after the client disconnected 503 (25s) each 200

Root cause

ToxicCollection.RemoveToxic holds the collection's mutex while it interrupts the toxic. The toxic is blocked in WriteOutput(c, 0), which only returns once the destination reads or link.writeDone closes (#760).

When the destination dies, ToxicLink.write's io.Copy returns, and write calls link.toxics.RemoveLink(name), which takes the same mutex, before its deferred close(link.writeDone) runs. So writeDone never closes, the toxic never stops, and the two goroutines wait on each other forever.

Solution

Close writeDone as soon as io.Copy returns, before RemoveLink. writeDone means "write stopped draining link.output", which is already true at that point, and nothing after it reads link.output.

Verification

  • New TestRemoveToxicDoesNotDeadlockWhenStalledDestinationDies: a latency toxic, a destination (io.Pipe) that nobody reads, RemoveToxic started and holding the lock, then the destination is closed. It fails on main (removal never returns) and passes with the change.
  • Root package: go test -race ./... passes on Linux; the removal tests also passed 20 times in a row with -race.
  • golangci-lint v2.14.0 (the CI version): 0 issues.
  • On my machine the toxics latency timing tests (TestFullstreamLatencyBias*, TestZeroLatency) fail under -race in WSL on main as well, so I don't think they're related.

Compatibility and risks

Only the point where writeDone closes moves. A stub still sending after that gets the same "output already closed" error #760 already handles.

Not changed here: while the client is connected but not reading, removing a toxic still waits until it reads or disconnects, since #760 deliberately waits for a live reader rather than dropping data.

Related to #500.

@soujasK
soujasK force-pushed the fix/write-done-deadlock branch from 23dccb6 to c919c0d Compare October 26, 2026 10:41
A RemoveToxic waiting for a stalled destination holds the toxic
collection's lock. When that destination died, write called RemoveLink,
which needs the same lock, before its deferred close of writeDone ran, so
the removal never saw writeDone close and both goroutines waited on each
other forever. The proxy's toxic API kept returning 503 from then on.

Follow-up to Shopify#760, which added writeDone.
@soujasK
soujasK force-pushed the fix/write-done-deadlock branch from c919c0d to 569ab48 Compare October 26, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant