[nacx]

Since the move to radix v4, REDIS_TIMEOUT only bounds dialing. radix v4 has no read or write timeout: once a command is written it waits for the reply for as long as the socket stays open. When a Redis server goes away without resetting its connections (e.g. a managed Redis failover that keeps the IP), every pooled connection blocks until the kernel gives up retransmitting, about 15 minutes with default Linux settings. Meanwhile every rate limit call hangs, nothing is logged and no error is counted.

Dial Redis connections through our own NetDialer that wraps each connection so that every read radix waits on, and every write, is bounded by an I/O timeout. radix clears the read deadline only right before it reads a reply, so idle connections are never timed out. When the timeout fires the read fails with an error radix treats as an I/O failure, so the connection is closed, the commands in flight on it fail and the pool dials a replacement. The dialer also does the TLS handshake itself so TLS connections are bounded the same way.

The I/O timeout defaults to REDIS_TIMEOUT, which restores its documented meaning ("connection and I/O operations"), and can be set separately with REDIS_IO_TIMEOUT / REDIS_PERSECOND_IO_TIMEOUT (0 disables it). Sentinel connections are not bounded, since radix keeps an idle pubsub connection to the sentinels.

[harmseru]

We hit exactly this in production and independently traced it down to the same root cause (filed as #1282 / a competing patch as #1283, which we're closing in favor of this one — see below for why this PR's approach is more complete than ours).

In our case the dead connections were caused by a CNI (kube-router/iptables conntrack) silently dropping the NAT entry for the ratelimit→Redis TCP connection under load — no FIN/RST exchanged, so the connection looks ESTABLISHED forever on both ends. That's functionally the same failure mode described here (peer disappears without resetting the connection); conntrack eviction is just another way to get there besides a failover that keeps the IP.

Correlating the service's own metrics across two incidents confirmed the mechanism precisely:

metric	baseline	during incident
ratelimit.go.numGoroutine	86	4728
ratelimit.redis_pool.cx_active	10 (= pool size)	still 10 — never drops
ratelimit.redis_pool.cx_total / cx_local_close	~0	spike only after the goroutine pile-up, right as it recovers

cx_active never dipping is the tell: it's a raw open-socket gauge, so a connection that's dead but still technically open is indistinguishable from a healthy one. Recovery only happened once something (eventually) closed and redialed the stuck connections — nothing bounded how long that could take, so under load one dead connection turned into ~4700 leaked goroutines before the container had to be restarted.

On why we're closing our own patch (#1283) in favor of this one: we initially bounded Do() with a context.WithTimeout at the call site rather than at the socket level. That stops the caller from hanging forever, but it doesn't fix the actual problem — conn.go's doRead() translates a context-deadline error into resp.ErrConnUsable, so radix puts the connection back in the pool instead of closing it. The dead connection just gets handed to the next caller, which also times out, forever — no self-healing. This PR avoids that entirely by using a distinct ErrIOTimeout sentinel (not os.ErrDeadlineExceeded) for the liveness-deadline case, so radix correctly closes and replaces the connection instead of recycling it. It also covers write-side stalls, which a context-only fix doesn't reliably bound. Would support getting this merged.