Repository navigation
Conversation
REDIS_TIMEOUT/REDIS_PERSECOND_TIMEOUT are documented ("sets the timeout
for Redis connection and I/O operations") and the README repeats the
same claim, but createDialer only ever threads the configured value
into net.Dialer.Timeout, which bounds Dial() alone. Once a connection
is established, no deadline is attached to any Do() call anywhere in
the service, so the documented I/O timeout has never actually applied
to I/O.
In practice this means a connection that goes silently dead - for
example its underlying TCP state is dropped by a NAT/conntrack layer
with no FIN/RST exchanged, which we observed happening intermittently
between our ratelimit sidecars and Redis - blocks the goroutine that
owns it forever. Because the radix v4 pool is fixed-size and blocks
when empty (as the comment above the retry loop already notes), every
other caller queued behind that connection for a pool slot leaks too.
Under load this turned a single dead connection into several hundred
leaked goroutines and a CPU spike, recoverable only by restarting the
container.
radix v4's conn.EncodeDecode and Pool.Do both already select on
ctx.Done(): EncodeDecode force-unblocks a pending read via
SetReadDeadline and reports the connection unusable so the pool
discards and redials it, and Pool.Do aborts if the context is done
before a connection is even acquired. So wrapping every Do() call with
a deadline derived from the existing timeout setting is sufficient for
the pool to detect and replace a dead connection on its own, with no
change needed in radix itself.
newDeadlineEnforcingClient wraps the underlying radix client and:
- applies ioTimeout only when the caller's context has no deadline,
or a looser one than ioTimeout, so it never widens a caller's own
tighter deadline
- is skipped entirely when ioTimeout <= 0, preserving the current
unbounded behavior as an explicit opt-out
This applies to both the main and per-second Redis clients, since both
go through newClientImpl.
Signed-off-by: Rudolf Harmse <rharmse@visa.com>
|
Closing in favor of #1271, which was already open before we filed this and addresses the same root cause more completely. Our fix here bounded #1271 avoids both problems by wrapping the underlying |
Fixes #1282.
REDIS_TIMEOUT/REDIS_PERSECOND_TIMEOUTare documented as covering "Redis connection and I/O operations", butcreateDialer()only ever threads the configured value intonet.Dialer.Timeout, which boundsDial()alone. No deadline is attached to anyDo()/pipeline call anywhere in the service, so the documented I/O timeout has never actually applied to I/O — see #1282 for the full writeup and the production impact (a single silently-dead Redis connection, caused in our case by a CNI dropping the underlying NAT/conntrack entry with no FIN/RST, blocked its goroutine forever and leaked every caller queued behind it for a pool slot, since the radix v4 pool is fixed-size and blocks when empty).What this changes
Adds
deadlineEnforcingClientinsrc/redis/driver_impl.go, which wraps the radix client returned fromnewClientImplso everyDo()call is bounded by the existingREDIS_TIMEOUT/REDIS_PERSECOND_TIMEOUTvalue:<= 0, preserving today's unbounded behavior as an explicit opt-outnewClientImplNo new config surface. This makes the existing, already-documented setting do what it says.
I verified against
radix/v4 v4.1.4that this is sufficient on its own:conn.EncodeDecodeandPool.Doalreadyselectonctx.Done()—EncodeDecodeforce-unblocks a pending read viaSetReadDeadlineand marks the connection unusable so the pool discards/redials it, andPool.Doaborts while still queued for a connection if the context is done first. So no change is needed in radix itself.Compatibility note
This is technically a behavior change: a
Do()call that previously could block forever on a dead connection will now return an error afterREDIS_TIMEOUT(default10s). Anyone who wants to keep the old unbounded behavior can setREDIS_TIMEOUT=0/REDIS_PERSECOND_TIMEOUT=0.Test plan
src/redis/driver_impl_test.go: a stuck call gets bounded byioTimeoutinstead of hanging, an existing tighter caller deadline is respected rather than widened, the<=0opt-out returns the original client unchanged, and a normal fast call is unaffected.go test ./src/redis/...,go vet ./src/redis/...,gofmt -l, andpre-commit run -aall pass./cc @mattklein123 @ysawa0