Fix RedisWorker fetch_task() head-of-line blocking - #7901
Open
dkliban wants to merge 1 commit into
Open
Conversation
dkliban
force-pushed
the
fetch-task-blocked-resource-skip
branch
from
July 24, 2026 23:16
7081c45 to
5b330ea
Compare
dkliban
force-pushed
the
fetch-task-blocked-resource-skip
branch
2 times, most recently
from
July 25, 2026 00:25
5af3ed3 to
c99b617
Compare
dkliban
force-pushed
the
fetch-task-blocked-resource-skip
branch
from
July 25, 2026 21:16
99eb67e to
423a674
Compare
When the task queue head is dominated by tasks needing the same exclusive resource (e.g., 17,000 tasks for one repository), all workers scan the same blocked tasks and never reach tasks with free resources further down. Replace the exponential-doubling strategy with blocked-resource DB-level exclusion: after acquire_locks reports a resource is blocked, exclude tasks needing that resource from subsequent queries using reserved_resources_record array overlap. The loop re-queries from position 0 with growing exclusions until no new blocked resources are discovered. In the incident scenario (2026-07-24), this reduces task selection from ~10 exponentially growing queries to 2 queries: one to discover the blocked resource, one filtered query that skips all 17,000 blocked tasks and returns tasks with free resources. Fixes: pulp#7900 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dkliban
force-pushed
the
fetch-task-blocked-resource-skip
branch
from
July 26, 2026 00:28
423a674 to
a266ef2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RedisWorker.fetch_task()when the queue head is dominated by tasks needing the same exclusive resourceacquire_locksreports a resource is blocked, exclude tasks needing that resource from subsequent DB queries usingreserved_resources_record__overlapshared:prefixed variants to the exclusion set so tasks needing shared access to an exclusively-locked resource are also skippedTest plan
fetch_task()against real PostgreSQL with mocked Redis (25 blocked tasks + 1 free, all blocked, multiple blocked resources, shared exclusion, FIFO order)Fixes #7900
🤖 Generated with Claude Code