Skip to content

Keep the sparse Jensen-Shannon post-processing in the input precision - #2535

Open
maxwbuckley wants to merge 1 commit into
NVIDIA:mainfrom
maxwbuckley:sm120/sparse-jensen-shannon-fp32-sqrt
Open

Keep the sparse Jensen-Shannon post-processing in the input precision#2535
maxwbuckley wants to merge 1 commit into
NVIDIA:mainfrom
maxwbuckley:sm120/sparse-jensen-shannon-fp32-sqrt

Conversation

@maxwbuckley

Copy link
Copy Markdown
Contributor

Problem

The final step of the sparse Jensen-Shannon distance is a raft::linalg::map over the whole a_nrows x b_nrows output:

[=] __device__(value_t input) { return raft::sqrt(0.5 * input); }

0.5 is a double, so for value_t == float the multiply promotes the expression and raft::sqrt resolves to the fp64 overload.

Unlike a stray double in an add or a multiply, an fp64 sqrt is not one instruction — it is a Newton-Raphson refinement sequence. The SASS emitted for the float instantiation of the map kernel was:

DMUL        R14, R4, 0.5      <-- the literal
MUFU.RSQ64H R13, R15
DMUL        R16, R12, R12
DFMA        R16, R14, -R16, 1
DFMA        R18, R16, R18, 0.5
DMUL        R16, R12, R16
DFMA        R16, R18, R16, R12
DMUL        R18, R14, R16
DFMA        R22, R18, -R18, R14
DFMA        R4,  R22, R20, R18

Ten fp64 instructions per output element, each running at 1/64 the fp32 rate on consumer parts.

Fix

Write the constant as value_t(0.5) so overload resolution picks the float path. The same region becomes:

FMUL     R0, R4, 0.5
MUFU.RSQ R7, R0
FMUL.FTZ R5, R0, R7
FMUL.FTZ R7, R7, 0.5
FFMA     R0, -R5, R5, R0
FFMA     R5, R0, R7, R5

Numerics

0.5 is exactly representable in binary, so the double instantiation is bit-for-bit unchanged. The float instantiation now rounds once instead of twice, so individual results move by at most an ulp.

Measurements

RTX 5090 (sm_120a), CUDA 13.2, random CSR inputs, median of 5 runs, libcuvs.so swapped between the two builds and interleaved:

rows x cols, nnz/row map kernel speedup full pairwise_distance speedup
4096 x 4096, 32 263 -> 25 us 10.4x 6.40 -> 6.16 ms 1.04x
8192 x 4096, 32 1046 -> 319 us 3.28x 25.25 -> 24.54 ms 1.03x
16384 x 4096, 8 4168 -> 1396 us 2.99x 29.16 -> 26.38 ms 1.11x
16384 x 16384, 32 4168 -> 1394 us 2.99x 130.08 -> 127.32 ms 1.02x

Output checksums (mean over the full distance matrix) are identical to six decimal places in all four shapes.

At 16384 rows the map kernel now moves 2.1 GB in 1.39 ms, i.e. it sits at the memory roofline rather than being fp64-throughput-bound — which is why it stops at 3x. The 4096 case reaches 10.4x because its output fits in L2.

End-to-end gains are smaller because the balanced COO SpMV that produces the map's input is 85-95% of the call.

Same class of issue as #2531.

The final step of the sparse Jensen-Shannon distance is

    [=] __device__(value_t input) { return raft::sqrt(0.5 * input); }

`0.5` is a `double`, so for `value_t == float` the multiply promotes the
expression and `raft::sqrt` resolves to the fp64 overload. Unlike a stray
`double` in an add or multiply, an fp64 sqrt is not one instruction: it is a
Newton-Raphson refinement sequence. The generated SASS for the float
instantiation of the map kernel was

    DMUL        R14, R4, 0.5
    MUFU.RSQ64H R13, R15
    DMUL        R16, R12, R12
    DFMA        R16, R14, -R16, 1
    ... 6 more DMUL/DFMA

ten fp64 instructions per element, which run at 1/64 the fp32 rate on
consumer parts. Writing the constant as `value_t(0.5)` picks the float
overload and reduces this to `FMUL` + `MUFU.RSQ` + four float ops.

`0.5` is exactly representable in binary, so the `double` instantiation is
bit-for-bit unchanged. The `float` instantiation now rounds once instead of
twice, so results move by at most an ulp.

Measured on an RTX 5090 (sm_120a, CUDA 13.2), random CSR inputs, median of 5:

  rows x cols, nnz/row   map kernel       full pairwise_distance
  4096  x 4096,  32      263 -> 25 us      6.44 -> 6.20 ms   1.04x
  8192  x 4096,  32     1046 -> 319 us    25.31 -> 24.57 ms  1.03x
  16384 x 4096,   8     4167 -> 1395 us   29.21 -> 26.42 ms  1.11x
  16384 x 16384, 32     4169 -> 1394 us  130.30 -> 127.53 ms 1.02x

The map kernel itself is 3.0-10.4x faster. At 16384 rows it now moves
2.1 GB in 1.39 ms, i.e. it has gone from fp64-throughput-bound to sitting at
the memory roofline; the 4096 case is faster still because its output fits in
L2. End-to-end gains are smaller because the balanced COO SpMV that produces
the input dominates the call.

Output checksums are unchanged to six decimal places across all four shapes.
@maxwbuckley
maxwbuckley requested a review from a team as a code owner August 31, 2026 19:53
@copy-pr-bot

copy-pr-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@maxwbuckley

Copy link
Copy Markdown
Contributor Author

@lowener thank you :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant