Skip to content

redis: bound Do() calls by REDIS_TIMEOUT, not just the initial dial - #1283

Closed
harmseru wants to merge 1 commit into
envoyproxy:mainfrom
harmseru:fix/redis-io-deadline-enforcement
Closed

harmseru wants to merge 1 commit into
envoyproxy:mainfrom
harmseru:fix/redis-io-deadline-enforcement

Conversation

@harmseru

Copy link
Copy Markdown

Fixes #1282.

REDIS_TIMEOUT/REDIS_PERSECOND_TIMEOUT are documented as covering "Redis connection and I/O operations", but createDialer() only ever threads the configured value into net.Dialer.Timeout, which bounds Dial() alone. No deadline is attached to any Do()/pipeline call anywhere in the service, so the documented I/O timeout has never actually applied to I/O — see #1282 for the full writeup and the production impact (a single silently-dead Redis connection, caused in our case by a CNI dropping the underlying NAT/conntrack entry with no FIN/RST, blocked its goroutine forever and leaked every caller queued behind it for a pool slot, since the radix v4 pool is fixed-size and blocks when empty).

What this changes

Adds deadlineEnforcingClient in src/redis/driver_impl.go, which wraps the radix client returned from newClientImpl so every Do() call is bounded by the existing REDIS_TIMEOUT/REDIS_PERSECOND_TIMEOUT value:

  • only applies the timeout when the caller's context has no deadline, or a looser one — never widens a caller's own tighter deadline
  • skipped entirely when the configured timeout is <= 0, preserving today's unbounded behavior as an explicit opt-out
  • applies to both the main and per-second Redis clients, since both go through newClientImpl

No new config surface. This makes the existing, already-documented setting do what it says.

I verified against radix/v4 v4.1.4 that this is sufficient on its own: conn.EncodeDecode and Pool.Do already select on ctx.Done() — EncodeDecode force-unblocks a pending read via SetReadDeadline and marks the connection unusable so the pool discards/redials it, and Pool.Do aborts while still queued for a connection if the context is done first. So no change is needed in radix itself.

Compatibility note

This is technically a behavior change: a Do() call that previously could block forever on a dead connection will now return an error after REDIS_TIMEOUT (default 10s). Anyone who wants to keep the old unbounded behavior can set REDIS_TIMEOUT=0 / REDIS_PERSECOND_TIMEOUT=0.

Test plan

  • New tests in src/redis/driver_impl_test.go: a stuck call gets bounded by ioTimeout instead of hanging, an existing tighter caller deadline is respected rather than widened, the <=0 opt-out returns the original client unchanged, and a normal fast call is unaffected.
  • go test ./src/redis/..., go vet ./src/redis/..., gofmt -l, and pre-commit run -a all pass.

/cc @mattklein123 @ysawa0

REDIS_TIMEOUT/REDIS_PERSECOND_TIMEOUT are documented ("sets the timeout
for Redis connection and I/O operations") and the README repeats the
same claim, but createDialer only ever threads the configured value
into net.Dialer.Timeout, which bounds Dial() alone. Once a connection
is established, no deadline is attached to any Do() call anywhere in
the service, so the documented I/O timeout has never actually applied
to I/O.

In practice this means a connection that goes silently dead - for
example its underlying TCP state is dropped by a NAT/conntrack layer
with no FIN/RST exchanged, which we observed happening intermittently
between our ratelimit sidecars and Redis - blocks the goroutine that
owns it forever. Because the radix v4 pool is fixed-size and blocks
when empty (as the comment above the retry loop already notes), every
other caller queued behind that connection for a pool slot leaks too.
Under load this turned a single dead connection into several hundred
leaked goroutines and a CPU spike, recoverable only by restarting the
container.

radix v4's conn.EncodeDecode and Pool.Do both already select on
ctx.Done(): EncodeDecode force-unblocks a pending read via
SetReadDeadline and reports the connection unusable so the pool
discards and redials it, and Pool.Do aborts if the context is done
before a connection is even acquired. So wrapping every Do() call with
a deadline derived from the existing timeout setting is sufficient for
the pool to detect and replace a dead connection on its own, with no
change needed in radix itself.

newDeadlineEnforcingClient wraps the underlying radix client and:
  - applies ioTimeout only when the caller's context has no deadline,
    or a looser one than ioTimeout, so it never widens a caller's own
    tighter deadline
  - is skipped entirely when ioTimeout <= 0, preserving the current
    unbounded behavior as an explicit opt-out

This applies to both the main and per-second Redis clients, since both
go through newClientImpl.

Signed-off-by: Rudolf Harmse <rharmse@visa.com>
@harmseru

Copy link
Copy Markdown
Author

Closing in favor of #1271, which was already open before we filed this and addresses the same root cause more completely.

Our fix here bounded Do() with a context deadline at the call site, which stops the caller from hanging forever but doesn't actually fix the pool: conn.go's doRead() translates a context-deadline error into resp.ErrConnUsable, so radix puts the dead connection back in the pool instead of closing it — the next caller just times out again, with no self-healing. It also doesn't bound write-side stalls.

#1271 avoids both problems by wrapping the underlying net.Conn and distinguishing its own liveness-deadline errors from genuine caller-context cancellations, so radix correctly closes and replaces a dead connection rather than recycling it, and both reads and writes are bounded. Left a comment there with production metrics corroborating the failure mode. Would recommend prioritizing that one instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REDIS_TIMEOUT doesn't bound Redis I/O, only the initial connection — dead connections leak goroutines forever

1 participant