perf: raise rerank max-batch-size to the NIM's 512-passage request limit - #2567
Draft
erichare wants to merge 5 commits into
Draft
perf: raise rerank max-batch-size to the NIM's 512-passage request limit#2567erichare wants to merge 5 commits into
erichare wants to merge 5 commits into
Conversation
nvidia/llama-3.2-nemoretriever-500m-rerank-v2 (NIM 2.x) accepts 512 passages per call, so a typical findAndRerank becomes a single reranking call instead of a fan-out of ten batch-10 calls, greatly reducing burst load on the reranking service. Non-default; the legacy rerankqa-1b-v2 model and the default /nvidia/v1/ranking route are unchanged. The model URL follows the GPU plane's per-model ingress route pattern; the route and NIM deployment land in the gpu-helm-charts repo and should be live before this model is put into use.
Contributor
➡️ Unit Test Coverage Delta vs Main Branch
|
Contributor
Unit Test Coverage Report
|
Contributor
➡️ Integration Test Coverage Delta vs Main Branch (dse69-it)
|
Contributor
Integration Test Coverage Report (dse69-it)
|
Contributor
📈 Integration Test Coverage Delta vs Main Branch (hcd-it)
|
Contributor
Integration Test Coverage Report (hcd-it)
|
The reranking NIM accepts up to 512 passages per request; it rejects 513+ with a 400 validation error. With max-batch-size 10, a findAndRerank with 100 candidates fans out into 10 concurrent reranking calls, multiplying request load on the reranking service by ~10x for no benefit. At 512 a typical findAndRerank is a single call. Verified against the dev reranking endpoint (rerankqa-1b-v2, NIM 1.3.1.0): 512-passage requests return 200 with all 512 rankings in ~0.36s warm; 600 passages returns 400 "List should have at most 512 items". Deployments that serve reranking config via the embedding gateway or a config override need the same value there for this to take effect.
erichare
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Raises
max-batch-sizefornvidia/llama-3.2-nv-rerankqa-1b-v2from 10 to 512, the reranking NIM's actual per-request limit.With
max-batch-size: 10, afindAndRerankwith 100 candidates fans out into 10 concurrent reranking calls; a burst of N API requests can hit the reranking service with M×N concurrent POSTs. At 512, a typicalfindAndRerankis one call.Which issue(s) this PR fixes:
Fixes #2569
Checklist