feat: reranking concurrency gate, total deadline, and 512-passage nemotron batches - #2566
Closed
erichare wants to merge 3 commits into
Closed
feat: reranking concurrency gate, total deadline, and 512-passage nemotron batches#2566erichare wants to merge 3 commits into
erichare wants to merge 3 commits into
Conversation
…l deadline A 1000-request findAndRerank burst currently fans out ~10 concurrent rerank calls per request with no limit anywhere, overrunning the reranking NIM's queue and mass-failing on the read timeout. This adds a per provider+model concurrency gate so bursts degrade into orderly queueing instead of a thundering herd: - RerankingConcurrencyGate: non-blocking bulkhead capping in-flight rerank calls per pod (max-concurrent-calls, default 32) with a bounded FIFO wait queue (max-queued-calls, default 1000). Overflow fails fast with the new RERANKING_PROVIDER_OVERLOADED error without calling the provider, and is deliberately never retried. Cancellation while parked frees the queue slot without consuming a permit. - Per-request fan-out cap via Uni.join().usingConcurrencyOf() (max-concurrent-batches, default 8). - Total per-request deadline (total-timeout-millis, default 30000) covering queue wait, all batches, and retries; expiry surfaces RERANKING_PROVIDER_TIMEOUT and abandons parked work before it reaches the provider. - Providers are constructed per API request, so gates live in the new @ApplicationScoped RerankingConcurrencyGateRegistry and are passed through the RerankingProvider base constructor; the embedding gateway path (RerankingEGWClient) inherits the gate and deadline, which also closes the previously-unbounded gRPC path. - New properties flow through RequestProperties, the yaml schema, and embedding_gateway.proto (optional fields 7-10) with unset/out-of-range values falling back to the Data API defaults, never zero. - Micrometer meters per gate: rerank.all.queue.depth, rerank.all.inflight.calls, rerank.all.queue.wait.duration, rerank.all.queue.rejected.count.
nvidia/llama-3.2-nemoretriever-500m-rerank-v2 (NIM 2.x) accepts 512 passages per call, so a typical findAndRerank becomes a single reranking call instead of a fan-out of ten batch-10 calls - the main throughput lever for absorbing large request bursts. Non-default; the legacy rerankqa-1b-v2 model and the default /nvidia/v1/ranking route are unchanged. The model URL uses the per-model ingress route pattern; the corresponding GPU-plane deployment and route are tracked in docs/reranking-nim-capacity-recommendations.md. On embedding-gateway deployments the served model list comes from the gateway, which needs the matching entry (docs/reranking-egw-coordination.md).
…otes - CONFIGURATION.md: new Reranking configuration section documenting the per-model properties including the concurrency gate knobs, the queue sizing rule, and the gate metrics. - docs/reranking-nim-capacity-recommendations.md: gpu-helm-charts side of the 1000-burst target - nemotron NIM deployment, ingress route, NIM 2.x queue and shedding env vars, and the replica sizing formula. - docs/reranking-egw-coordination.md: embedding gateway proto fields to mirror (7-10), fallback semantics, and the requirement to serve the nemotron model with max_batch_size 512 for Astra.
Contributor
📈 Unit Test Coverage Delta vs Main Branch
|
Contributor
Unit Test Coverage Report
|
Contributor
Author
|
Closing — this bundles too much for one review. Splitting into smaller PRs: (1) the nemotron 512-passage model entry, (2) a total reranking deadline, (3) the concurrency gate. Branch feat/reranker-concurrency-gate is retained as the reference implementation. |
4 tasks
Contributor
📉 Integration Test Coverage Delta vs Main Branch (dse69-it)
|
Contributor
Integration Test Coverage Report (dse69-it)
|
Contributor
📉 Integration Test Coverage Delta vs Main Branch (hcd-it)
|
Contributor
Integration Test Coverage Report (hcd-it)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
PepsiCo needs 1000 simultaneous
findAndRerankrequests to succeed without errors (latency may grow). Today each request fans outceil(passages/10)concurrent rerank POSTs with no concurrency limit anywhere, so a 1000-request burst is ~10,000 concurrent calls; the reranking NIM's internal queue blows past the read timeout and requests mass-fail withRERANKING_PROVIDER_TIMEOUT.Queuing alone converts errors into latency but does not create throughput, so this PR combines three things:
1. Reranking concurrency gate (per provider+model, per pod)
max-concurrent-calls(default 32) rerank calls in flight across all requests; excess calls park in a bounded FIFO queue (max-queued-calls, default 1000). New non-blockingRerankingConcurrencyGate+@ApplicationScopedRerankingConcurrencyGateRegistry(providers are constructed per request, so shared state must live in a singleton).RERANKING_PROVIDER_OVERLOADEDerror, without ever calling the provider — and it is deliberately never retried (retrying a saturation rejection deepens the overload).max-concurrent-batches(default 8) viaUni.join().usingConcurrencyOf().total-timeout-millis(default 30000) covers queue wait + all batches + retries. Expiry surfacesRERANKING_PROVIDER_TIMEOUT; parked work is abandoned before it reaches the provider. The embedding-gateway path inherits the gate and deadline through theRerankingProviderbase class — previously the gRPC path had no client-side time bound at all.rerank.all.queue.depth,rerank.all.inflight.calls(gauges),rerank.all.queue.wait.duration(timer, parked grants),rerank.all.queue.rejected.count(counter).2. nemotron rerank model with 512-passage batches
nvidia/llama-3.2-nemoretriever-500m-rerank-v2(max-batch-size: 512, non-default) collapses a typical findAndRerank from 10 rerank calls to 1 — the lever that makes the 1000-burst arithmetically feasible (PepsiCo's own 2-pod nemotron benchmark absorbed an 1800-request burst at 0% errors).3. Config plumbing + handoff docs
RequestProperties, the yaml schema, andembedding_gateway.proto(optionalfields 7–10). Unset or out-of-range gateway values fall back to the Data API defaults, never zero, so the Data API can deploy before EGW.CONFIGURATION.mdgains a Reranking configuration section (there was none).docs/reranking-nim-capacity-recommendations.md: gpu-helm-charts changes (nemotron NIM deployment, ingress route,NIM_SERVER_MAX_QUEUE_SIZE/NIM_SERVER_REQUEST_TIMEOUT_S, replica sizing formula).docs/reranking-egw-coordination.md: EGW proto mirror + serving nemotron with batch 512 for Astra.Behavior changes to note
RERANKING_PROVIDER_OVERLOADED(clients should treat as retry-later).Test plan
RerankingConcurrencyGateTest(9): cap enforcement incl. multi-threaded contention, FIFO order, no barging, overflow fail-fast (work never invoked), cancel-while-parked frees the slot, release on failure/cancellation, meter trackingRerankingProviderTest(+6): fan-out cap with correct aggregation, shared gate across provider instances, deadline-while-parked (provider never called, queue drained), deadline mid-call releases permit, overflow surfaced to caller, OVERLOADED never retried / TIMEOUT still retriedRerankingProvidersConfigTest/...OverrideTest/RerankingProviderConfigProducerTest: yaml defaults, per-deployment overrides, EGW proto pass-through + fallback semantics (incl. explicitmax_queued_calls=0honored)AllErrorCodesLoadTestcovers the new error templateFindRerankingProvidersIntegrationTestexpectations updated for the new model entryOVERLOADEDonly beyond the configured envelope