Repository navigation
Conversation
radix v4 does not return when a command's context is done: the caller keeps waiting until the connection reads the reply or fails, and it does so while holding the pool's read lock. So although the gRPC deadline set by Envoy (20ms by default in Envoy Gateway) is passed all the way down to PipeDo, a call to an unresponsive Redis blocks far beyond it, piling up goroutines and in-flight requests. Run each Redis call on its own goroutine and return as soon as the call completes or the context is done, whichever comes first. The call finishes in the background, bounded by the I/O timeout. Since it may outlive PipeDo, replies are decoded into private receivers and copied into the caller's only when PipeDo succeeds, so a call the caller gave up on never writes into its memory. PipeDo documents that receivers are only written on success. Calls whose context is already done are not sent to Redis: radix treats that case as a connection error and discards a healthy connection. Add REDIS_MAX_INFLIGHT / REDIS_PERSECOND_MAX_INFLIGHT (default 0, unbounded) to cap the commands in flight per client, including those still running after their caller gave up, so a slow Redis cannot pile up unbounded work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
radix v4 does not return when a command's context is done: the caller keeps waiting until the connection reads the reply or fails, and it does so while holding the pool's read lock. So although the gRPC deadline set by Envoy (20ms by default in Envoy Gateway) is passed all the way down to PipeDo, a call to an unresponsive Redis blocks far beyond it, piling up goroutines and in-flight requests.
Run each Redis call on its own goroutine and return as soon as the call completes or the context is done, whichever comes first. The call finishes in the background, bounded by the I/O timeout. Since it may outlive PipeDo, replies are decoded into private receivers and copied into the caller's only when PipeDo succeeds, so a call the caller gave up on never writes into its memory. PipeDo documents that receivers are only written on success.
Calls whose context is already done are not sent to Redis: radix treats that case as a connection error and discards a healthy connection.
Add REDIS_MAX_INFLIGHT / REDIS_PERSECOND_MAX_INFLIGHT (default 0, unbounded) to cap the commands in flight per client, including those still running after their caller gave up, so a slow Redis cannot pile up unbounded work.