Repository navigation
[Common] CGEMM stream ID logging - #3727
jberchtold-nvidia wants to merge 6 commits into
Conversation
Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com>
Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com>
f17b8d7 to
4383a83
Compare
|
/te-ci arm64 L1 |
[Low impact] The change affects logging during collective GEMM context creation. Add the missing cuBLASMp skip before merging so optional builds do not fail the distributed tests. Findings
SummaryAdds opt-in logging of collective GEMM stream IDs and ranks.
Reviews (4) · Last reviewed commit: "Merge branch 'main' into jberchtold/cgem..." · Reviewed by Greptile |
|
/te-ci arm64 L1 |
Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com>
Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com>
|
/te-ci L1 arm64 |
Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com>
| TEST(CommGemmStreamLogging, LogsStreamIdToStdout) { | ||
| int nranks{}; | ||
| int rank{}; | ||
| CHECK_MPI(MPI_Comm_size(MPI_COMM_WORLD, &nranks)); | ||
| CHECK_MPI(MPI_Comm_rank(MPI_COMM_WORLD, &rank)); | ||
| ASSERT_EQ(cudaSetDevice(rank), cudaSuccess); |
There was a problem hiding this comment.
The new test calls nvte_comm_gemm_ctx_create without checking nvte_built_with_cublasmp(). With NVTE_WITH_CUBLASMP disabled—the default—the call throws instead of skipping. The QA script launches this test whenever four GPUs are available, so builds without cuBLASMp now fail this suite.
Add the same early skip used by test_comm_gemm.cu, before creating the NCCL communicator.
Description
Default is no logging. If the user sets
NVTE_LOG_COLLECTIVE_GEMM_STREAM_IDS=1, then stream IDs will be loggedType of change
Changes
Checklist: