[https://nvbugs/6650817][fix] Removed only the two GB300 waiver lines for this test (id 6661948) after… - #18565
Conversation
… pp4 on GB300 The reported failure does not reproduce. The abort behind it, Assertion failed: Deepseek should be supported by fmha in context part. (cpp/tensorrt_llm/common/attentionOp.cpp:3204) comes from a build whose CUDA architecture list omitted SM103. GB300 is SM103, and DeepSeek context MLA needs the HQk192HV128 separate-QKV shape, which exists as kSM_100/kSM_103/kSM_107 but has no kSM_100f sibling. Since isSMCompatible() accepts only kSM_100f or kSM_103 on an SM103 device, the lookup misses once the SM103 cubins are compiled out, FmhaDispatcher::isSupported() returns false and the check fires. The CI symptom, MPI_ERR_OTHER out of _wait_for_executor_workers_ready, is the orchestrator observing the worker die during setup_engine. Rebuilding the same worktree for 103-real makes the cubins present: FmhaSm103a goes from 0 to 3228 symbols, of which 48 cover HQk192HV128, including the exact shape the failing run could not resolve (QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausal...Context). Verified on GB300 (SM103) with a 103-real build. Both previously waived parametrizations run, and the trtllm-gen kernel-lookup misses are gone: headDimQk=192 misses, unfused-MHA fallbacks and the context-FMHA assertion all drop from 32/32/1 occurrences to zero. GSM8K accuracy is 63.419 against a reference of 64.740 and a threshold of 61.537. Only the two lines for this test are removed; the neighbouring waivers, including the one for the NVFP4 sibling, are left untouched. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review. WalkthroughRemoved two GB300 DeepSeek-V3-Lite FP8 block-scale test skip entries associated with NVBUG 6661948. Changes
Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: ⚪ Minimal · up to This change removes two test waivers so the covered parametrizations are expected to run normally; no actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Description checkExplanation The description explains the root cause, fix, affected tests, verification results, and bug link. It does not use the template's exact Description, Test Coverage, or checklist sections, but it provides the required information in equivalent sections. Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
2 participants
Summary
pytest "tests/integration/defs/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=False-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False]" "tests/integration/defs/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False]" -vTest plan
Links
Reproduction comparison
Signature: signal 6 (Aborted).
Dev Engineer Review
QA Engineer Review
tests/integration/test_lists/waives.txt.103-realbuild.