add Flashinfer sampling backend#1328
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request replaces the sglang_kernel sampling backend with flashinfer across the CLI arguments, start arguments, and post-processing logic. It also updates the flashinfer package versions in requirements.txt and introduces configurable sampling parameters to the QPS benchmark script. The feedback points out that the flashinfer backend does not support per-request random seeds, which would break deterministic generation, and suggests falling back to the triton backend when a custom seed is requested.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if get_env_start_args().sampling_backend == "triton": | ||
| sampling_backend = get_env_start_args().sampling_backend | ||
|
|
||
| if sampling_backend == "triton": |
There was a problem hiding this comment.
The flashinfer sampling backend does not support per-request random seeds (exist_req_use_random_seed), which will break deterministic/seeded generation when a custom seed is provided. To ensure correctness, we should fall back to the triton sampling implementation when exist_req_use_random_seed is True.
| if sampling_backend == "triton": | |
| if sampling_backend == "triton" or exist_req_use_random_seed: |
1 participant
No description provided.