Stop concurrent coroutines sharing a phpredis socket - #5
Merged
Conversation
Swoole binds a hooked phpredis socket to the coroutine that first reads
from it. When a second coroutine issues a command on that connection the
scheduler raises
Swoole\Error: Socket#N has already been bound to another coroutine#M
which is not thrown through the calling frame, so userland try/catch
cannot contain it and the worker dies. Workers dying under load caused
request pile-up and, in turn, memory_limit fatals.
Every occurrence observed in production came in through
RateLimiter->tooManyAttempts -> Repository->get -> RedisStore->get.
Illuminate registers RateLimiter as a boot-time singleton wrapping a
cache Repository, so its RedisStore keeps pointing at the worker-level
Redis manager. RequestScope already isolated `redis` and `cache`, but an
object built at boot never resolves them again.
Two independent defences:
- RequestScope now scopes Illuminate\Cache\RateLimiter, cloning the base
limiter so named RateLimiter::for() registrations survive and swapping
only the cache repository for the coroutine-local one.
- CoroutineRedisManager makes the worker-level manager itself
coroutine-aware, so a singleton captured at boot -- including ones this
package never sees -- still gets a per-coroutine connection, because
RedisStore re-resolves its connection per operation. It strips
persistent/persistent_id, since persistent sockets are shared
process-wide by id and would reintroduce the sharing. It is installed
with extend() because Illuminate's RedisServiceProvider is deferred and
would otherwise overwrite a plain singleton() on first resolve.
Connections are released when the request coroutine finishes.
Sentry's transport sends events with curl_exec, which some Swoole builds
make coroutine-only. Laravel's shutdown handler reports fatals from
outside any coroutine, so reporting an error could raise a second fatal
that replaced the original in the logs. Captures now run with the curl
hooks temporarily cleared when outside a coroutine, and any transport
failure is swallowed: losing one event beats losing the worker.
The end-to-end probe runs in a subprocess, since surviving this fatal is
only observable as a clean exit. The unfixed combination is asserted to
fail, so the passing cases cannot silently become vacuous.
Verified on the production runtime image (phpswoole/swoole:php8.3 --
PHP 8.3.32, Swoole 6.2.1, phpredis 6.3.0).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
Swoole binds a hooked phpredis socket to the coroutine that first reads from it. When a second coroutine issues a command on that same connection, the scheduler raises:
This is raised by the scheduler, not thrown through the calling frame — a
try/catch (\Throwable)wrapped directly around the failing call does not catch it, and the worker dies. Worker deaths under load caused request pile-up, and in turnmemory_limitfatals.Root cause
Every occurrence observed in production came in through the same path:
Illuminate\Cache\RateLimiteris registered byCacheServiceProvideras a boot-time singleton wrapping a cacheRepository, so itsRedisStorekeeps pointing at the worker-level Redis manager.RequestScopealready isolatedredisandcacheper coroutine, but an object constructed at boot never resolves them again — so every concurrent request throttled through one shared socket.The fix — two independent defences
1.
RequestScopescopesIlluminate\Cache\RateLimiter. The base limiter is cloned rather than rebuilt, so namedRateLimiter::for()registrations made during boot survive; only the cache repository is swapped for the coroutine-local one.2.
CoroutineRedisManagermakes the worker-level manager itself coroutine-aware. This covers boot-time singletons that this package never sees, becauseRedisStorere-resolves its connection on every operation. Notes:persistent/persistent_id— persistent phpredis sockets are shared process-wide by id and would reintroduce the sharing.extend()rather thansingleton(), because Illuminate'sRedisServiceProvideris deferred and would otherwise overwrite the binding on first resolve.$this->config, since resolving performs I/O and a coroutine can park mid-connect.3. Sentry transport hardening. Sentry sends events with
curl_exec, which some Swoole builds make coroutine-only. Laravel's shutdown handler reports fatals from outside any coroutine, so reporting an error could raise a second fatal (API must be called in the coroutine) that replaced the original in the logs. Captures now run with curl hooks temporarily cleared when outside a coroutine, and any transport failure is swallowed — losing one event beats losing the worker.Verification
The end-to-end probe runs in a subprocess, because surviving this fatal is only observable as a clean exit. Run against the production runtime image (
phpswoole/swoole:php8.3— PHP 8.3.32, Swoole 6.2.1, phpredis 6.3.0):The unfixed combination is asserted as a failing test, so if the harness ever stops reproducing the bug the passing cases cannot silently become vacuous.
136/136 tests pass (was 106 — 30 new), both locally and inside the production image.
Trade-off
Each coroutine now opens its own Redis connection per request rather than reusing a persistent one. This already applied to anything resolved during a request; this change extends it to the RateLimiter path, so the net increase is modest. If Redis
maxclientsor connect latency becomes a concern, a pooled free-list (mirroring the existingDatabasePool) is the follow-up — deliberately left out here, since returning a connection interrupted mid-reply to a pool carries real corruption risk and stability was the priority.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.