Repository navigation
Conversation
soujasK
force-pushed
the
fix/write-done-deadlock
branch
from
October 26, 2026 10:41
23dccb6 to
c919c0d
Compare
A RemoveToxic waiting for a stalled destination holds the toxic collection's lock. When that destination died, write called RemoveLink, which needs the same lock, before its deferred close of writeDone ran, so the removal never saw writeDone close and both goroutines waited on each other forever. The proxy's toxic API kept returning 503 from then on. Follow-up to Shopify#760, which added writeDone.
soujasK
force-pushed
the
fix/write-done-deadlock
branch
from
October 26, 2026 15:05
c919c0d to
569ab48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
If a client stops reading while a toxic is being removed, and then disconnects, the proxy's toxic API stays stuck for good: the
DELETEreturns 503 after the 25sTimeoutHandler, and every later request for that proxy (GET /proxies/{proxy}, adding/removing toxics) keeps returning 503 until the server is restarted. This matches what's described in #500.Reproduced end to end against
mainwith the server binary: a proxy with alatencytoxic, an upstream that keeps sending, and a client that connects but never reads:DELETE /proxies/p/toxics/latwhile the client is stalledGET /proxies/p5s / 10s / 15s / 20s after the client disconnectedRoot cause
ToxicCollection.RemoveToxicholds the collection's mutex while it interrupts the toxic. The toxic is blocked inWriteOutput(c, 0), which only returns once the destination reads orlink.writeDonecloses (#760).When the destination dies,
ToxicLink.write'sio.Copyreturns, andwritecallslink.toxics.RemoveLink(name), which takes the same mutex, before its deferredclose(link.writeDone)runs. SowriteDonenever closes, the toxic never stops, and the two goroutines wait on each other forever.Solution
Close
writeDoneas soon asio.Copyreturns, beforeRemoveLink.writeDonemeans "write stopped draininglink.output", which is already true at that point, and nothing after it readslink.output.Verification
TestRemoveToxicDoesNotDeadlockWhenStalledDestinationDies: a latency toxic, a destination (io.Pipe) that nobody reads,RemoveToxicstarted and holding the lock, then the destination is closed. It fails onmain(removal never returns) and passes with the change.go test -race ./...passes on Linux; the removal tests also passed 20 times in a row with-race.toxicslatency timing tests (TestFullstreamLatencyBias*,TestZeroLatency) fail under-racein WSL onmainas well, so I don't think they're related.Compatibility and risks
Only the point where
writeDonecloses moves. A stub still sending after that gets the same "output already closed" error #760 already handles.Not changed here: while the client is connected but not reading, removing a toxic still waits until it reads or disconnects, since #760 deliberately waits for a live reader rather than dropping data.
Related to #500.