Bound CAGRA hash table sizing loops so they cannot spin forever

[shaunakkapur]

Closes #2523.

Problem

search_plan_impl::calc_hashmap_params() sizes its hash tables with loops that grow a bit length until the requested table fits, and check the supported maximum only after the loop:

hash_bitlen = min_bitlen;
while (max_traversed_nodes > hashmap::get_size(hash_bitlen) * max_fill_rate) {
  hash_bitlen += 1;
}
RAFT_EXPECTS(hash_bitlen <= 25, "hash_bitlen cannot be largen than 25 (32M)");

hashmap::get_size() is 1U << bitlen, which is undefined once bitlen reaches 32. In practice the shift count is masked to 5 bits, so get_size(32) wraps to 1, get_size(33) to 2, and so on. hash_bitlen is int64_t, so for any request needing a table larger than 2^31 entries the loop condition stays permanently true, the bit length grows without bound, and the RAFT_EXPECTS is never reached. The search hangs on the host rather than returning an error.

check_params() only caps itopk_size at 1024 for SINGLE_CTA, so MULTI_CTA and MULTI_KERNEL are the two algorithms that can reach the sizing loops with an oversized request. A normal single-query AUTO search resolves to MULTI_CTA.

Fix

Bound each of the four sizing loops by the same maximum its post-loop check already uses, so an oversized request exits the loop and raises the existing error instead of spinning:

while (hash_bitlen <= max_hash_bitlen_mc &&
       max_traversed_nodes > hashmap::get_size(hash_bitlen) * max_fill_rate) {
  hash_bitlen += 1;
}
RAFT_EXPECTS(hash_bitlen <= max_hash_bitlen_mc, "hash_bitlen cannot be largen than 25 (32M)");

Each limit is hoisted into a local constant so the loop bound and the check cannot drift apart, which is the failure mode this bug came from.

All four loops are covered (lines 270, 285, 303, 343). Only 285, 303 and 343 are reachable with user controlled sizes; 270 is bounded by graph_degree in practice and is bounded here for consistency.

Behaviour

No change for any request that was previously accepted. A request that previously hung now raises the error it should always have raised. A request that previously errored still errors.

Testing

New regression test cpp/tests/neighbors/ann_cagra/bug_issue_2523_hashmap_bitlen.cu, registered under NEIGHBORS_ANN_CAGRA_TEST_BUGS, covering MULTI_CTA oversized, MULTI_KERNEL oversized, and valid itopk_size on all three algorithms.

Note that a regression here resurfaces as a test timeout rather than a failed assertion, since the failure mode is a hang. This is called out in the test file header.

I also checked that the test actually catches this. With the fix reverted and nothing else changed, MultiCtaOversizedItopkThrows hangs and gets killed at the 300s timeout (exit 124) instead of failing an assertion. With the fix back in, it passes in about 1.6s.

Verified on an RTX PRO 6000 Blackwell:

• itopk_size = 1,100,000,000 previously hung indefinitely (confirmed past a 60s timeout, with the thread stuck in native code). It now raises the hash_bitlen error in a few seconds.
• Full NEIGHBORS_ANN_CAGRA_TEST_BUGS suite: 10/10 passed, including the five pre-existing reproducers.
• A standalone harness replicating the old and new loop logic across all four loops and 40 powers of two: identical results on every in-range input (136 cases), equivalent error signalling where the old loop already errored (104 cases), and clean errors in place of 88 cases where the old loop did not terminate. Zero mismatches.
• pre-commit hooks pass on the changed files.

Notes for reviewers

• I did not change hashmap::get_size() itself. Making it safe for bitlen >= 32 (returning uint64_t, or asserting on its input) would remove the undefined behaviour at the source, but it is used in device code and by callers that assign to uint32_t, so it is a wider change. Bounding the loops fixes the reachable defect locally. Happy to follow up on get_size() separately if you would prefer that hardening too.
• The copyright header line in search_plan.cuh was rewritten by the verify-copyright pre-commit hook to the canonical form. That is unrelated to the fix but the hook applies it to any touched file.