From 6dc5ef0679494f7632bce1b0f2f8b3b1cd5cc164 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Mon, 17 Aug 2026 14:16:14 +0300 Subject: [PATCH 01/19] fix(uint8): make the integer accumulators exact, with a dim fallback The uint8 kernels accumulate products or squared differences of bytes, so the total reaches 255 * 255 * dim = 65025 * dim. Four paths could not hold that, all of them on the plain int8/uint8 index paths that ship today. * IP.cpp / L2.cpp: ret_t for a 1-byte element type was int, so the scalar UINT8_InnerProduct and UINT8_L2Sqr executed signed-overflow UB from dimension 33,026, while the comment claimed support to 2^16. The conditional was also dead: only int8_t and uint8_t instantiate these, both 1 byte, so it always selected int. ret_t is now 64-bit for every element type, which also covers int8 at dimension 131,072. Kept signed so the "1 - ip" in the wrappers stays signed arithmetic and cannot underflow. The L2 comment still carried the old byte-counting rationale and is fixed. * UINT8_InnerProductImp returned float on NEON and SVE, accumulating exactly in integer lanes and then discarding it, exact only to dimension 258 since 2^24 / 65025 = 258. * AVX-512 reduced 16 int32 lanes with _mm512_reduce_add_epi32 into a signed int, wrapping from dimension 33,026. * L2_AVX512F_BW_VL_VNNI_UINT8 and L2_NEON_UINT8 read an unsigned horizontal reduce back into a signed int, so the distance went negative from the same dimension. L2_NEON_DOTPROD_UINT8 and L2_SVE_UINT8 were already unsigned. The accumulation was never the problem. The SIMD adds wrap modulo 2^32 and are bit-exact, so the bit pattern was already correct; the top bit was being read as a sign. An unsigned 32-bit reduce therefore costs nothing over the original and is exact through dimension 66,051 = floor(UINT32_MAX / 65,025), twice the old signed limit of 33,025. Verified: the AVX-512 object is byte-for-byte the same size as before at 514,792, with the same 33 and 40 instructions for residual 0 and 32. Above 66,051 the choosers hand back the scalar kernel, which after the ret_t change is exact to roughly dimension 2.8e14. One comparison at index creation, reusing the "if (dim < 32) return ret_dist_func" idiom the choosers already had, and it leaves every kernel untouched. Three alternatives were tried and rejected, each on evidence: * Widening the horizontal reduce unconditionally. Measured on an Ice Lake-SP Xeon it costs 4 extra uops in the epilogue: +20% at dimension 32, +8-11% across 55-200, +4-5% at 900-1024, on byte-identical loop code. * A runtime branch selecting the width per call. Measured at +0.4 to 0.6 ns per call, +15% at dimension 32, and it cost 31 of the 65 Cosine wrappers their inlining, growing .text by 18.4%. * Compile-time selection between two named wrappers, extending SIMD to a second bound of 4 * 66,051. This one is free on the narrow path, verified: the narrow wrappers stayed byte-identical and the out-of-line count unchanged. It was rejected for correctness, not cost. That bound assumes products spread evenly across the four uint32 lanes after NEON's 32-bit vaddq_u32 merge, and the even case already lands within 1,020 of UINT32_MAX, while the masked residual load can add up to 16 products, or 1,040,400, into specific lanes. So lanes wrap before the widened reduce sees them, and a correct bound would have to be derived per kernel from its accumulator count and residual distribution. The narrow reduce needs none of that: its bound is on the horizontal total, which does not depend on how products land in lanes. Recorded for whoever revisits this: on ARM the widening reduce is free instruction-for-instruction. Cross-compiling with clang++ --target=aarch64-linux-gnu -O2 emits addv/fmov w/ucvtf against uaddlv/fmov x/ucvtf, three instructions either way. So the obstacle to a wider band is the lane bound, not the reduce. Nothing comparable supports that range regardless. Lucene caps its scalar-quantized format at 1,024 dimensions and Elasticsearch caps dense vectors at 4,096, both keeping a 32-bit accumulator safe by contract. Faiss's QT_8bit_direct accumulates full-range bytes into 32-bit lanes with no widening and carries the same theoretical limit. Qdrant quantizes to 0..127, lowering the per-element cap to 16,129, and its raw uint8 metric still sums into i32. The SQ8_SQ8 kernels reuse this helper, on main as much as here, so they now take its result as uint32_t. Previously the AVX-512 one assigned it to int, which wrapped past 33,025 once the helper stopped returning int, and the three ARM ones to float, which lost exactness past 258. Note the SQ8_SQ8 choosers have no dimension guard, unlike the uint8 ones, so the fence belongs with SQ8 index creation in #1007; on main nothing constructs an SQ8 index. Split out of #1011 because none of this depends on the SQ8 metadata contract that PR is changing, while all of it affects code reachable today. #1011 depends on this, through the helper above. The regressions use all-255 bytes, the worst case, which makes the expected value an exact integer, at dimensions 33,026 and 40,000 for the SIMD path and 66,052 for the fallback. The fallback test asserts the returned function pointer, not just the distance: on a host with no uint8 SIMD tier the value comparison would pass either way, but the pointer identity would not. The existing UINT8 suites stop at dimension 128, which is why all of this went unseen; being SIMD-versus-scalar comparisons they would also have agreed with each other wherever both wrapped. Also fixes the uint8 spaces benchmark fixture, which paired new[] with delete and stored the trailing norms through unaligned float casts, so measurements taken from it can be trusted. Co-Authored-By: Claude Opus 5 (1M context) --- src/VecSim/spaces/IP/IP.cpp | 14 ++-- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h | 7 +- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 16 +++-- .../spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h | 7 +- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 13 ++-- src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h | 7 +- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 14 ++-- src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h | 7 +- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 17 ++--- src/VecSim/spaces/IP_space.cpp | 14 ++++ src/VecSim/spaces/L2/L2.cpp | 10 +-- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 8 ++- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 5 +- src/VecSim/spaces/L2_space.cpp | 7 ++ src/VecSim/spaces/spaces.h | 35 +++++++++ .../spaces_benchmarks/bm_spaces_uint8.cpp | 11 +-- tests/unit/test_spaces.cpp | 72 +++++++++++++++++++ 17 files changed, 219 insertions(+), 45 deletions(-) diff --git a/src/VecSim/spaces/IP/IP.cpp b/src/VecSim/spaces/IP/IP.cpp index 2140c2345..cefafb773 100644 --- a/src/VecSim/spaces/IP/IP.cpp +++ b/src/VecSim/spaces/IP/IP.cpp @@ -238,11 +238,15 @@ float FP16_InnerProduct(const void *pVect1, const void *pVect2, size_t dimension } // Return type for the inner product functions. -// The type should be able to hold `dimension * MAX_VAL(int_elem_t) * MAX_VAL(int_elem_t)`. -// To support dimension up to 2^16, we need the difference between the type and int_elem_t to be at -// least 2 bytes. We assert that in the implementation. +// The type must hold `dimension * MAX_VAL(int_elem_t) * MAX_VAL(int_elem_t)`. For uint8 that is +// 65025 * dimension, which overflows a 32-bit int from dimension 33,026 -- the alias was previously +// `int` for any 1-byte element, so UINT8_InnerProduct executed signed-overflow UB there. +// +// Signedness follows the element type, which matters for the wrappers below: UINT8_InnerProduct +// converts to float before subtracting from 1, so an unsigned accumulator is fine there, while +// INT8_InnerProduct computes `1 - ip` in integer arithmetic and needs a signed one. template -using ret_t = std::conditional_t; +using ret_t = std::conditional_t, uint64_t, int64_t>; template static inline ret_t @@ -273,7 +277,7 @@ float INT8_Cosine(const void *pVect1v, const void *pVect2v, size_t dimension) { float UINT8_InnerProduct(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto *pVect1 = static_cast(pVect1v); const auto *pVect2 = static_cast(pVect2v); - return 1 - INTEGER_InnerProductImp(pVect1, pVect2, dimension); + return 1.0f - static_cast(INTEGER_InnerProductImp(pVect1, pVect2, dimension)); } float UINT8_Cosine(const void *pVect1v, const void *pVect2v, size_t dimension) { diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h index ae6f96ea2..499eef8a4 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h @@ -40,7 +40,12 @@ template // 0..63 float SQ8_SQ8_InnerProductImp(const void *pVec1v, const void *pVec2v, size_t dimension) { // Compute raw dot product using efficient UINT8 AVX512 VNNI implementation // UINT8_InnerProductImp uses _mm512_dpwssd_epi32 for native integer dot product - int dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); + // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike + // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is + // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the + // fence belongs with SQ8 index creation (#1007) rather than here. + const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors // Layout: [data (dim)] [min (float)] [delta (float)] [sum (float)] diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index bd43bc901..1134fa033 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -31,8 +31,8 @@ static inline void InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i } template // 0..63 -static inline int UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, - size_t dimension) { +static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, + size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -87,19 +87,25 @@ static inline int UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v } while (pVect1 < pEnd1); } - return _mm512_reduce_add_epi32(sum); + // Unsigned reduce. The lanes are in range individually, but their total reaches 255*255*dim, + // which passes INT_MAX from dimension 33,027, so reading the result as a signed int wrapped it. + // The intrinsic's adds are vector operations, so the bit pattern is already correct modulo + // 2^32 and this cast simply reads it as unsigned. Exact through + // spaces::MAX_EXACT_UINT8_SIMD_DIM; above that the chooser selects the scalar kernel instead of + // this one. + return static_cast(_mm512_reduce_add_epi32(sum)); } template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + return 1.0f - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); } template // 0..63 float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + float ip = static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h index d7f5b444e..0a2d14cac 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h @@ -41,7 +41,12 @@ float SQ8_SQ8_InnerProductSIMD64_NEON_DOTPROD_IMP(const void *pVec1v, const void size_t dimension) { // Compute raw dot product using efficient UINT8 DOTPROD implementation // UINT8_InnerProductImp uses vdotq_u32 for native uint8 dot product - float dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); + // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike + // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is + // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the + // fence belongs with SQ8 index creation (#1007) rather than here. + const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors // Layout: [data (dim)] [min (float)] [delta (float)] [sum (float)] diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index 3abbd9bba..f5057417b 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -27,7 +27,7 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { } template // 0..63 -float UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { +uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -97,20 +97,21 @@ float UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dim uint32x4_t total_sum = vaddq_u32(sum0, sum1); - int32_t result = vaddvq_u32(total_sum); - - return static_cast(result); + // ADDV, unsigned. The total reaches 255*255*dim, so the previous int32_t receiving this + // wrapped negative from dimension 33,027. Exact through spaces::MAX_EXACT_UINT8_SIMD_DIM; + // above that the chooser selects the scalar kernel instead of this one. + return vaddvq_u32(total_sum); } template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + return 1.0f - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); } template // 0..63 float UINT8_CosineSIMD_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + float ip = static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h index 3e931ee0e..4628d3d5d 100644 --- a/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h @@ -41,7 +41,12 @@ float SQ8_SQ8_InnerProductSIMD64_NEON_IMP(const void *pVec1v, const void *pVec2v size_t dimension) { // Compute raw dot product using efficient UINT8 implementation // UINT8_InnerProductImp processes 16 elements at a time using native uint8 instructions - float dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); + // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike + // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is + // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the + // fence belongs with SQ8 index creation (#1007) rather than here. + const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of pVec1 // Layout: [data (dim)] [min (float)] [delta (float)] [sum (float)] diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index 2d2b3f555..c486d3598 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -35,7 +35,7 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { } template // 0..63 -float UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { +uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -105,20 +105,20 @@ float UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dim uint32x4_t total_sum = vaddq_u32(sum0, sum1); - // Horizontal sum of the 4 elements in the combined sum register - int32_t result = vaddvq_u32(total_sum); - - return static_cast(result); + // ADDV, unsigned. The total reaches 255*255*dim, so the previous int32_t receiving this + // wrapped negative from dimension 33,027. Exact through spaces::MAX_EXACT_UINT8_SIMD_DIM; + // above that the chooser selects the scalar kernel instead of this one. + return vaddvq_u32(total_sum); } template // 0..15 float UINT8_InnerProductSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + return 1.0f - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); } template // 0..63 float UINT8_CosineSIMD_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + float ip = static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h index 93ddb76cb..8751aac9f 100644 --- a/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h @@ -40,7 +40,12 @@ template float SQ8_SQ8_InnerProductSIMD_SVE_IMP(const void *pVec1v, const void *pVec2v, size_t dimension) { // Compute raw dot product using efficient UINT8 SVE implementation // UINT8_InnerProductImp uses svdot_u32 for native uint8 dot product - float dot_product = + // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike + // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is + // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the + // fence belongs with SQ8 index creation (#1007) rather than here. + const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index f6c6af3b8..b827ddb20 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -24,7 +24,7 @@ inline void InnerProductStep(const uint8_t *&pVect1, const uint8_t *&pVect2, siz } template -float UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { +uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { const uint8_t *pVect1 = reinterpret_cast(pVect1v); const uint8_t *pVect2 = reinterpret_cast(pVect2v); @@ -82,21 +82,22 @@ float UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dim sum0 = svadd_u32_x(svptrue_b32(), sum0, sum1); sum2 = svadd_u32_x(svptrue_b32(), sum2, sum3); - // Perform vector addition in parallel and Horizontal sum - int32_t sum_all = svaddv_u32(svptrue_b32(), svadd_u32_x(svptrue_b32(), sum0, sum2)); - - return sum_all; + // svaddv_u32 reduces into a 64-bit scalar; the previous int32_t truncated it, which wrapped + // negative from dimension 33,027. Narrowed to uint32_t, which is exact through + // spaces::MAX_EXACT_UINT8_SIMD_DIM; above that the chooser selects the scalar kernel. + return static_cast(svaddv_u32(svptrue_b32(), svadd_u32_x(svptrue_b32(), sum0, sum2))); } template float UINT8_InnerProductSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - - UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + return 1.0f - static_cast(UINT8_InnerProductImp( + pVect1v, pVect2v, dimension)); } template float UINT8_CosineSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = UINT8_InnerProductImp(pVect1v, pVect2v, dimension); + float ip = static_cast( + UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/IP_space.cpp b/src/VecSim/spaces/IP_space.cpp index 6706d8f31..44c495787 100644 --- a/src/VecSim/spaces/IP_space.cpp +++ b/src/VecSim/spaces/IP_space.cpp @@ -770,6 +770,13 @@ dist_func_t IP_UINT8_GetDistFunc(size_t dim, unsigned char *alignment, dist_func_t ret_dist_func = UINT8_InnerProduct; + // Above this dimension the 32-bit horizontal reduce in every uint8 SIMD kernel wraps, so fall + // back to the scalar kernel, which accumulates into a 64-bit ret_t and stays exact. See + // spaces::MAX_EXACT_UINT8_SIMD_DIM for why this is a fallback rather than a wider reduce. + if (dim > MAX_EXACT_UINT8_SIMD_DIM) { + return ret_dist_func; + } + [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); #ifdef CPU_FEATURES_ARCH_AARCH64 @@ -818,6 +825,13 @@ dist_func_t Cosine_UINT8_GetDistFunc(size_t dim, unsigned char *alignment dist_func_t ret_dist_func = UINT8_Cosine; + // Above this dimension the 32-bit horizontal reduce in every uint8 SIMD kernel wraps, so fall + // back to the scalar kernel, which accumulates into a 64-bit ret_t and stays exact. See + // spaces::MAX_EXACT_UINT8_SIMD_DIM for why this is a fallback rather than a wider reduce. + if (dim > MAX_EXACT_UINT8_SIMD_DIM) { + return ret_dist_func; + } + [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); #ifdef CPU_FEATURES_ARCH_AARCH64 diff --git a/src/VecSim/spaces/L2/L2.cpp b/src/VecSim/spaces/L2/L2.cpp index 015f2200d..9a2589789 100644 --- a/src/VecSim/spaces/L2/L2.cpp +++ b/src/VecSim/spaces/L2/L2.cpp @@ -133,11 +133,13 @@ float FP16_L2Sqr(const void *pVect1, const void *pVect2, size_t dimension) { } // Return type for the L2 functions. -// The type should be able to hold `dimension * MAX_VAL(int_elem_t) * MAX_VAL(int_elem_t)`. -// To support dimension up to 2^16, we need the difference between the type and int_elem_t to be at -// least 2 bytes. We assert that in the implementation. +// The type must hold `dimension * MAX_VAL(int_elem_t) * MAX_VAL(int_elem_t)`. For uint8 that is +// 65025 * dimension, which overflows a 32-bit int from dimension 33,026 -- the alias was previously +// `int` for any 1-byte element, so UINT8_L2Sqr executed signed-overflow UB there. Signedness +// follows the element type, matching the inner product; both wrappers convert to float before +// returning, so either would be safe here. diff_t stays signed, which the assert below enforces. template -using ret_t = std::conditional_t; +using ret_t = std::conditional_t, uint64_t, int64_t>; // Difference type for the L2 functions. // The type should be able to hold `MIN_VAL(int_elem_t)-MAX_VAL(int_elem_t)`, and should be signed diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 350b759ea..2b99c5b40 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -92,5 +92,11 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVec } while (pVect1 < pEnd1); } - return _mm512_reduce_add_epi32(sum); + // The lanes hold sums of squared byte differences, so the horizontal total is unsigned and + // reaches 255*255*dim. Reading it as a signed int wrapped it negative from dimension 33,026. + // Still a 32-bit reduce, so this stays exact only to dimension 66,051 (65025 * 66052 exceeds + // UINT32_MAX). Unlike the inner product, which returns uint64, widening here would mean + // splitting the reduce; left as is because no dimension near that is realistic, but the bound + // is real and undocumented bounds are how the signed version survived this long. + return static_cast(static_cast(_mm512_reduce_add_epi32(sum))); } diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index aa3769867..feb485308 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -126,7 +126,10 @@ float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t di total_sum = vaddq_u32(total_sum, sum3); // Horizontal sum of the 4 elements in the combined sum register - int32_t result = vaddvq_u32(total_sum); + // Unsigned: the total is a sum of squared byte differences, reaching 255*255*dim. As a signed + // int32 this wrapped negative from dimension 33,026. Still 32-bit, so exact only to dimension + // 66,051; see the AVX512 variant for why that is left rather than widened. + uint32_t result = vaddvq_u32(total_sum); // Return the L2 squared distance as a float return static_cast(result); diff --git a/src/VecSim/spaces/L2_space.cpp b/src/VecSim/spaces/L2_space.cpp index 07e638cba..41c394331 100644 --- a/src/VecSim/spaces/L2_space.cpp +++ b/src/VecSim/spaces/L2_space.cpp @@ -467,6 +467,13 @@ dist_func_t L2_UINT8_GetDistFunc(size_t dim, unsigned char *alignment, } dist_func_t ret_dist_func = UINT8_L2Sqr; + + // Above this dimension the 32-bit horizontal reduce in every uint8 SIMD kernel wraps, so fall + // back to the scalar kernel, which accumulates into a 64-bit ret_t and stays exact. See + // spaces::MAX_EXACT_UINT8_SIMD_DIM for why this is a fallback rather than a wider reduce. + if (dim > MAX_EXACT_UINT8_SIMD_DIM) { + return ret_dist_func; + } // Optimizations assume at least 32 uint8. If we have less, we use the naive implementation. [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); diff --git a/src/VecSim/spaces/spaces.h b/src/VecSim/spaces/spaces.h index 11b0f9801..482cb3693 100644 --- a/src/VecSim/spaces/spaces.h +++ b/src/VecSim/spaces/spaces.h @@ -52,6 +52,41 @@ static int inline is_little_endian() { return *(char *)&x; } +// Largest dimension for which a uint8 SIMD kernel may be selected. +// +// The uint8 kernels accumulate products or squared differences of bytes into 32-bit SIMD lanes and +// finish with a 32-bit horizontal reduce. Per element that caps at 255 * 255 = 65025, so the total +// is 65025 * dim, exactly representable in uint32 through dim 66,051 and wrapping at 66,052. Note +// this is twice the old signed limit of 33,025: nothing about the accumulation changed there, the +// top bit was simply being read as a sign, which is why the unsigned reduce costs nothing. +// +// Above this dimension the choosers hand back the scalar kernel, which accumulates into a 64-bit +// ret_t and is exact to roughly dim 2.8e14. One comparison at index creation, no branch in any +// kernel, and no second set of instantiations. Two alternatives were considered and rejected: +// +// * Widening the horizontal reduce. Measured on an Ice Lake-SP Xeon it costs 4 extra uops in the +// epilogue: +20% at dim 32, +8-11% across dim 55-200, +4-5% at dim 900-1024, on byte-identical +// loop code. That is a dependency-chain cost rather than a throughput one, which is why an +// instruction count understates it. It also only moves the limit rather than removing it, and +// to a different place per ISA: roughly dim 1,056,816 on AVX-512, but only 264,204 on NEON, +// where four accumulators are combined with vaddq_u32 in 32 bits before the widening reduce +// ever sees them. +// +// * Chunking the accumulation and flushing into a 64-bit total. Exact at any dimension and cheap +// if the chunk loop lives in the wrapper rather than the kernel (+2 instructions on the fast +// path, versus +12 to +21 when placed inside the kernel). Deferred rather than dismissed: it +// is only worth the restructuring if dimensions above 66,051 become a real workload. +// +// Nothing comparable supports that range today. Lucene caps its scalar-quantized format at 1,024 +// dimensions and Elasticsearch caps dense vectors at 4,096, both of which keep a 32-bit +// accumulator safe by contract. Faiss's QT_8bit_direct path accumulates full-range bytes into +// int/32-bit lanes with no widening, so it carries the same theoretical limit. Qdrant quantizes to +// 0..127 instead of 0..255, which lowers the per-element cap to 16,129 and pushes the signed limit +// out to dim 133,144, and its raw uint8 metric still sums into i32. So the scalar fallback here is +// already stricter than the alternatives, and optimizing the SIMD path beyond 66,051 would be +// optimizing a range none of them accept. +static constexpr size_t MAX_EXACT_UINT8_SIMD_DIM = 66051; + static inline auto getCpuOptimizationFeatures(const void *arch_opt = nullptr) { #if defined(CPU_FEATURES_ARCH_AARCH64) diff --git a/tests/benchmark/spaces_benchmarks/bm_spaces_uint8.cpp b/tests/benchmark/spaces_benchmarks/bm_spaces_uint8.cpp index 602fff719..33f819936 100644 --- a/tests/benchmark/spaces_benchmarks/bm_spaces_uint8.cpp +++ b/tests/benchmark/spaces_benchmarks/bm_spaces_uint8.cpp @@ -31,12 +31,15 @@ class BM_VecSimSpaces_Integers_UINT8 : public benchmark::Fixture { test_utils::populate_uint8_vec(v2, dim, 1234); // Store the norm in the extra space for cosine calculations - *(float *)(v1 + dim) = test_utils::integral_compute_norm(v1, dim); - *(float *)(v2 + dim) = test_utils::integral_compute_norm(v2, dim); + // memcpy because v1 + dim is not guaranteed to be 4-byte aligned for arbitrary dim. + const float norm1 = test_utils::integral_compute_norm(v1, dim); + const float norm2 = test_utils::integral_compute_norm(v2, dim); + memcpy(v1 + dim, &norm1, sizeof(norm1)); + memcpy(v2 + dim, &norm2, sizeof(norm2)); } void TearDown(const ::benchmark::State &state) { - delete v1; - delete v2; + delete[] v1; + delete[] v2; } }; diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 8e53c83d1..83da413ea 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -2186,6 +2186,78 @@ TEST_P(UINT8SpacesOptimizationTest, UINT8_full_range_test) { INSTANTIATE_TEST_SUITE_P(UINT8OptFuncs, UINT8SpacesOptimizationTest, testing::Range(32UL, 64 * 2UL + 1)); +// The accumulated total is 255 * 255 * dim, which passes INT_MAX from dimension 33,026: the scalar +// path was signed-overflow UB there, and the AVX512 and NEON L2 reduces read their unsigned total +// back as a signed int and went negative. All-255 bytes are the worst case and make the expected +// value an exact integer. The existing UINT8 suites stop at dim 128, which is why this went unseen. +TEST_F(SpacesTest, UINT8_L2Sqr_and_InnerProduct_are_exact_past_int32) { + for (const size_t dim : {33026UL, 40000UL}) { + std::vector v1(dim + sizeof(float), 255); + std::vector v2(dim + sizeof(float), 0); + + // L2 between all-255 and all-0 is 255^2 * dim. + const double expected_l2 = 255.0 * 255.0 * static_cast(dim); + const float l2 = UINT8_L2Sqr(v1.data(), v2.data(), dim); + EXPECT_GT(l2, 0.0f) << "dim " << dim << ": squared distance went negative"; + EXPECT_LT(std::abs(static_cast(l2) - expected_l2) / expected_l2, 1e-6) + << "scalar L2, dim " << dim; + + unsigned char alignment = 0; + auto dispatched_l2 = L2_UINT8_GetDistFunc(dim, &alignment, nullptr); + const float l2_simd = dispatched_l2(v1.data(), v2.data(), dim); + EXPECT_GT(l2_simd, 0.0f) << "dim " << dim << ": SIMD squared distance went negative"; + EXPECT_LT(std::abs(static_cast(l2_simd) - expected_l2) / expected_l2, 1e-6) + << "dispatched L2, dim " << dim; + + // IP between two all-255 vectors is 255^2 * dim, and the kernel returns 1 - IP. + const double expected_ip = 1.0 - 255.0 * 255.0 * static_cast(dim); + const double ip = static_cast(UINT8_InnerProduct(v1.data(), v1.data(), dim)); + EXPECT_LT(std::abs(ip - expected_ip) / std::abs(expected_ip), 1e-6) + << "scalar IP, dim " << dim; + + auto dispatched_ip = IP_UINT8_GetDistFunc(dim, &alignment, nullptr); + const double ip_simd = static_cast(dispatched_ip(v1.data(), v1.data(), dim)); + EXPECT_LT(std::abs(ip_simd - expected_ip) / std::abs(expected_ip), 1e-6) + << "dispatched IP, dim " << dim; + } +} + +// Past spaces::MAX_EXACT_UINT8_SIMD_DIM the 32-bit horizontal reduce in every uint8 SIMD kernel +// wraps, so the choosers must hand back the scalar kernel, which accumulates into a 64-bit ret_t. +// Asserting the returned pointer is the point of this test: on a host with no uint8 SIMD support +// the value comparisons below would pass either way, but the pointer identity would not. +TEST_F(SpacesTest, UINT8_dispatchers_fall_back_to_scalar_past_the_exact_dim) { + constexpr size_t dim = spaces::MAX_EXACT_UINT8_SIMD_DIM + 1; + unsigned char alignment = 0; + + EXPECT_EQ(L2_UINT8_GetDistFunc(dim, &alignment, nullptr), UINT8_L2Sqr); + EXPECT_EQ(IP_UINT8_GetDistFunc(dim, &alignment, nullptr), UINT8_InnerProduct); + EXPECT_EQ(Cosine_UINT8_GetDistFunc(dim, &alignment, nullptr), UINT8_Cosine); + + // And one dimension below the threshold the SIMD kernel is still eligible, so the guard is a + // boundary rather than a blanket disable. Only assert this where a uint8 SIMD tier exists. + const auto features = getCpuOptimizationFeatures(); + const bool has_uint8_simd = +#ifdef CPU_FEATURES_ARCH_X86_64 + features.avx512f && features.avx512bw && features.avx512vl && features.avx512vnni; +#else + features.sve2 || features.sve || features.asimddp || features.asimd; +#endif + if (has_uint8_simd) { + EXPECT_NE(L2_UINT8_GetDistFunc(spaces::MAX_EXACT_UINT8_SIMD_DIM, &alignment, nullptr), + UINT8_L2Sqr); + } + + // The scalar path must actually be exact here, which is the reason the fallback is safe. + // All-255 against all-0 gives 255^2 * dim, past UINT32_MAX at this dimension. + std::vector v1(dim + sizeof(float), 255); + std::vector v2(dim + sizeof(float), 0); + const double expected_l2 = 255.0 * 255.0 * static_cast(dim); + EXPECT_GT(expected_l2, 4294967295.0) << "test would not exercise the 32-bit wrap"; + const double l2 = static_cast(UINT8_L2Sqr(v1.data(), v2.data(), dim)); + EXPECT_LT(std::abs(l2 - expected_l2) / expected_l2, 1e-6); +} + class SQ8_FP32_SpacesOptimizationTest : public testing::TestWithParam {}; TEST_P(SQ8_FP32_SpacesOptimizationTest, SQ8_FP32_L2SqrTest) { From 2c061fee399e048e40733969bc3645f34a7d69d8 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Mon, 17 Aug 2026 15:59:48 +0300 Subject: [PATCH 02/19] fix(uint8): accumulate in chunks instead of capping the SIMD dimension The previous commit kept the 32-bit SIMD accumulators and had the choosers hand back the scalar kernel past the dimension where those accumulators stay exact. That works but gives up SIMD entirely for large-dimension indexes, and the bound it relied on was only sound for the even lane distribution. Instead, split each uint8 kernel into an Imp that returns its raw integer total and two wrappers over it: - the plain wrapper, unchanged in behaviour, for dimensions up to UINT8_CHUNK_ELEMENTS (65,536) - a chunked wrapper that calls Imp once per chunk and folds the per-chunk totals in 64 bits The choosers pick between them once per index, so the plain kernel carries no branch. 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, so each chunk's 32-bit total is exact, and because every contribution is non-negative no individual accumulator lane can exceed the chunk total either. That is the entire correctness argument: no reasoning about how work spreads across lanes. The first chunk absorbs the residual, which leaves every later chunk a whole multiple of the kernel's step and so matches the residual-0 precondition. On SVE the vector length is a runtime value, so that split is computed in the wrapper rather than at compile time. Covers all five kernel families (AVX512 VNNI, NEON, NEON DOTPROD, SVE, SVE2) for L2, inner product and cosine. SQ8_SQ8 calls the helper directly rather than through a uint8 chooser, so it does not gain chunking; it is capped well below the chunk size by its uint32 metadata slot, and that fence belongs with SQ8 index creation. Also marks each Imp static and always_inline. always_inline keeps the plain wrappers byte-identical now that Imp has more call sites: without it GCC outlines Imp and the plain wrapper loses its inlining too. static removes a latent ODR problem, since the NEON and NEON DOTPROD headers define the same Imp name with different bodies and both are compiled into an ARM build. Replaces the fallback test with one that checks the dispatched kernel agrees exactly with the 64-bit scalar kernel across the chunk boundary, and one that checks the chooser actually switches families using two dimensions with the same residual. Co-Authored-By: Claude Opus 5 --- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h | 9 +- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 60 ++++++++++- .../spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h | 9 +- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 53 +++++++++- src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h | 9 +- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 65 +++++++++++- src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h | 9 +- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 66 ++++++++++++- src/VecSim/spaces/IP_space.cpp | 14 --- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 57 +++++++++-- src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h | 50 +++++++++- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 51 ++++++++-- src/VecSim/spaces/L2/L2_SVE_UINT8.h | 56 ++++++++++- src/VecSim/spaces/L2_space.cpp | 6 -- .../spaces/functions/AVX512F_BW_VL_VNNI.cpp | 28 +++++- src/VecSim/spaces/functions/NEON.cpp | 24 ++++- src/VecSim/spaces/functions/NEON_DOTPROD.cpp | 25 ++++- src/VecSim/spaces/functions/SVE.cpp | 24 ++++- src/VecSim/spaces/functions/SVE2.cpp | 24 ++++- src/VecSim/spaces/spaces.h | 48 +++------ tests/unit/test_spaces.cpp | 99 ++++++++++++++----- 21 files changed, 646 insertions(+), 140 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h index 499eef8a4..61806504d 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h @@ -42,9 +42,12 @@ float SQ8_SQ8_InnerProductImp(const void *pVec1v, const void *pVec2v, size_t dim // UINT8_InnerProductImp uses _mm512_dpwssd_epi32 for native integer dot product // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int - // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is - // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the - // fence belongs with SQ8 index creation (#1007) rather than here. + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. + // + // Note this calls the helper directly rather than through a uint8 chooser, so it does not get + // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total + // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 + // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index 1134fa033..3c08844c1 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -8,6 +8,7 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS static inline void InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { __m512i va = _mm512_loadu_epi8(pVect1); // AVX512BW @@ -30,9 +31,12 @@ static inline void InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i // with the corresponding 32-bit integer in src, and store the packed 32-bit results in dst. } +// always_inline, not merely inline: the chunked wrapper below calls this twice, and without the +// attribute GCC outlines it once it has several callers, which also costs the plain wrapper its +// inlining. Measured: the plain residual-0 wrapper went from 33 instructions to 9 plus a call. template // 0..63 -static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, - size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -90,9 +94,8 @@ static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pV // Unsigned reduce. The lanes are in range individually, but their total reaches 255*255*dim, // which passes INT_MAX from dimension 33,027, so reading the result as a signed int wrapped it. // The intrinsic's adds are vector operations, so the bit pattern is already correct modulo - // 2^32 and this cast simply reads it as unsigned. Exact through - // spaces::MAX_EXACT_UINT8_SIMD_DIM; above that the chooser selects the scalar kernel instead of - // this one. + // 2^32 and this cast simply reads it as unsigned. Exact for up to + // spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. return static_cast(_mm512_reduce_add_epi32(sum)); } @@ -110,3 +113,50 @@ float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVe const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } + +// Chunked variants, selected by the chooser for dimensions past spaces::UINT8_CHUNK_ELEMENTS. Each +// chunk's 32-bit total is exact, and the chunks are folded in 64 bits, so these are exact at any +// dimension. The plain wrappers above are left untouched so their inlining is unaffected, and the +// choice is made once per index rather than per call. +// +// The first chunk absorbs the residual, which leaves the remaining length a whole multiple of 64, +// so every later chunk satisfies the residual-0 kernel's precondition. +template // 0..63 +static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + const size_t first = residual + (chunk - residual) / 64 * 64; + uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < chunk ? remaining : chunk; + total += UINT8_InnerProductImp<0>(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return total; +} + +template // 0..63 +float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + return 1.0f - + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); +} + +template // 0..63 +float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const float ip = + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); + const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); + return 1.0f - ip / (norm_v1 * norm_v2); +} diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h index 0a2d14cac..a05f932ee 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h @@ -43,9 +43,12 @@ float SQ8_SQ8_InnerProductSIMD64_NEON_DOTPROD_IMP(const void *pVec1v, const void // UINT8_InnerProductImp uses vdotq_u32 for native uint8 dot product // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int - // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is - // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the - // fence belongs with SQ8 index creation (#1007) rather than here. + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. + // + // Note this calls the helper directly rather than through a uint8 chooser, so it does not get + // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total + // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 + // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index f5057417b..5738828a7 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -8,6 +8,7 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include __attribute__((always_inline)) static inline void InnerProductOp(uint8x16_t &v1, uint8x16_t &v2, @@ -26,8 +27,12 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { pVect2 += 16; } +// Returns the raw integer total, and is static and always_inline; see the NEON header for why each +// of those three matters. The internal linkage is what keeps this body and the NEON one apart +// despite the shared name. template // 0..63 -uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -98,8 +103,8 @@ uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t uint32x4_t total_sum = vaddq_u32(sum0, sum1); // ADDV, unsigned. The total reaches 255*255*dim, so the previous int32_t receiving this - // wrapped negative from dimension 33,027. Exact through spaces::MAX_EXACT_UINT8_SIMD_DIM; - // above that the chooser selects the scalar kernel instead of this one. + // wrapped negative from dimension 33,027. Exact for up to spaces::UINT8_CHUNK_ELEMENTS + // elements, which is what the caller guarantees. return vaddvq_u32(total_sum); } @@ -116,3 +121,45 @@ float UINT8_CosineSIMD_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, si const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } + +// See the NEON header for why each chunk's 32-bit total is exact and why the first chunk absorbs +// the residual. +template // 0..63 +static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + const size_t first = residual + (chunk - residual) / 64 * 64; + uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < chunk ? remaining : chunk; + total += UINT8_InnerProductImp<0>(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return total; +} + +template // 0..63 +float UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + return 1.0f - + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); +} + +template // 0..63 +float UINT8_CosineSIMD_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + float ip = + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); + const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); + return 1.0f - ip / (norm_v1 * norm_v2); +} diff --git a/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h index 4628d3d5d..215d20e5e 100644 --- a/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h @@ -43,9 +43,12 @@ float SQ8_SQ8_InnerProductSIMD64_NEON_IMP(const void *pVec1v, const void *pVec2v // UINT8_InnerProductImp processes 16 elements at a time using native uint8 instructions // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int - // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is - // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the - // fence belongs with SQ8 index creation (#1007) rather than here. + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. + // + // Note this calls the helper directly rather than through a uint8 chooser, so it does not get + // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total + // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 + // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of pVec1 diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index c486d3598..e2375852e 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -8,6 +8,7 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include __attribute__((always_inline)) static inline void InnerProductOp(uint8x16_t &v1, uint8x16_t &v2, @@ -34,8 +35,20 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { pVect2 += 16; } +// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing +// the float results per chunk would round each one. +// +// static: the NEON and NEON_DOTPROD headers define this same name with the same signature and +// different bodies, and both are compiled into an ARM build. At -O2 both are fully inlined today, +// so no symbol is emitted and nothing collides, but nothing guarantees that: outline either one and +// both objects define one mangled name, and a NEON call site could reach the DOTPROD body, which +// needs the dotprod feature. Internal linkage costs nothing and removes the possibility, which +// matters more now that the chunked wrapper below adds call sites. always_inline for the same +// reason from the other direction: GCC does outline this once it has several callers, and that also +// costs the plain wrapper its inlining. template // 0..63 -uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -106,8 +119,8 @@ uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t uint32x4_t total_sum = vaddq_u32(sum0, sum1); // ADDV, unsigned. The total reaches 255*255*dim, so the previous int32_t receiving this - // wrapped negative from dimension 33,027. Exact through spaces::MAX_EXACT_UINT8_SIMD_DIM; - // above that the chooser selects the scalar kernel instead of this one. + // wrapped negative from dimension 33,027. Exact for up to spaces::UINT8_CHUNK_ELEMENTS + // elements, which is what the caller guarantees. return vaddvq_u32(total_sum); } @@ -123,3 +136,49 @@ float UINT8_CosineSIMD_NEON(const void *pVect1v, const void *pVect2v, size_t dim const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } + +// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit +// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is +// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole +// correctness argument; no reasoning about how work spreads across lanes is needed. +// +// The first chunk absorbs the residual, so every later chunk is a whole multiple of 64 and matches +// the residual-0 kernel's precondition. +template // 0..63 +static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + const size_t first = residual + (chunk - residual) / 64 * 64; + uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < chunk ? remaining : chunk; + total += UINT8_InnerProductImp<0>(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return total; +} + +template // 0..63 +float UINT8_InnerProductSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + return 1.0f - + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); +} + +template // 0..63 +float UINT8_CosineSIMD_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { + float ip = + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); + const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); + return 1.0f - ip / (norm_v1 * norm_v2); +} diff --git a/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h index 8751aac9f..e13a88823 100644 --- a/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h @@ -42,9 +42,12 @@ float SQ8_SQ8_InnerProductSIMD_SVE_IMP(const void *pVec1v, const void *pVec2v, s // UINT8_InnerProductImp uses svdot_u32 for native uint8 dot product // uint32_t, matching what the helper returns. This kernel is reachable at any dimension: unlike // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int - // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. SQ8 itself is - // capped at spaces::MAX_EXACT_UINT8_SIMD_DIM by its uint32 q_sum_squares metadata slot, so the - // fence belongs with SQ8 index creation (#1007) rather than here. + // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. + // + // Note this calls the helper directly rather than through a uint8 chooser, so it does not get + // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total + // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 + // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index b827ddb20..fe3043692 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -8,6 +8,7 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include inline void InnerProductStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t &offset, @@ -23,8 +24,14 @@ inline void InnerProductStep(const uint8_t *&pVect1, const uint8_t *&pVect2, siz offset += chunk; // Move to the next set of uint8 elements } +// Split so the chunked wrapper below can fold each chunk's total in 64 bits; summing the float +// results per chunk would round each one. always_inline because the chunked wrapper calls this +// twice, and GCC outlines a template once it has several callers, which also costs the plain +// wrapper its inlining. static keeps each translation unit's copy to itself: SVE.cpp and SVE2.cpp +// both include this header, and other headers define the same name with different bodies. template -uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { const uint8_t *pVect1 = reinterpret_cast(pVect1v); const uint8_t *pVect2 = reinterpret_cast(pVect2v); @@ -83,8 +90,8 @@ uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t sum2 = svadd_u32_x(svptrue_b32(), sum2, sum3); // svaddv_u32 reduces into a 64-bit scalar; the previous int32_t truncated it, which wrapped - // negative from dimension 33,027. Narrowed to uint32_t, which is exact through - // spaces::MAX_EXACT_UINT8_SIMD_DIM; above that the chooser selects the scalar kernel. + // negative from dimension 33,027. Narrowed to uint32_t, which is exact for up to + // spaces::UINT8_CHUNK_ELEMENTS elements, and that is what the caller guarantees. return static_cast(svaddv_u32(svptrue_b32(), svadd_u32_x(svptrue_b32(), sum0, sum2))); } @@ -102,3 +109,56 @@ float UINT8_CosineSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dime const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } + +// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit +// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is +// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole +// correctness argument; no reasoning about how work spreads across lanes is needed. +template +static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + // The SVE vector length is a runtime value, so unlike the fixed-width kernels the split is + // computed here rather than at compile time. chunk_size matches the kernel's 4-accumulator main + // loop, and tail is the part the template parameters describe. + const size_t chunk_size = 4 * svcntb(); + const size_t tail = dimension % chunk_size; + const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; + const size_t first = tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; + + // first keeps this instantiation's own residual shape: it is congruent to dimension modulo + // chunk_size, so partial_chunk and additional_steps still describe its tail. + uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + // remaining is a whole multiple of chunk_size, and so is every step, which is the + // shape: no partial vector and no leftover single steps. + while (remaining) { + const size_t step = remaining < max_step ? remaining : max_step; + total += UINT8_InnerProductImp(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return total; +} + +template +float UINT8_InnerProductSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + return 1.0f - static_cast(UINT8_InnerProductChunkedImp( + pVect1v, pVect2v, dimension)); +} + +template +float UINT8_CosineSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { + float ip = static_cast( + UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); + const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); + return 1.0f - ip / (norm_v1 * norm_v2); +} diff --git a/src/VecSim/spaces/IP_space.cpp b/src/VecSim/spaces/IP_space.cpp index 44c495787..6706d8f31 100644 --- a/src/VecSim/spaces/IP_space.cpp +++ b/src/VecSim/spaces/IP_space.cpp @@ -770,13 +770,6 @@ dist_func_t IP_UINT8_GetDistFunc(size_t dim, unsigned char *alignment, dist_func_t ret_dist_func = UINT8_InnerProduct; - // Above this dimension the 32-bit horizontal reduce in every uint8 SIMD kernel wraps, so fall - // back to the scalar kernel, which accumulates into a 64-bit ret_t and stays exact. See - // spaces::MAX_EXACT_UINT8_SIMD_DIM for why this is a fallback rather than a wider reduce. - if (dim > MAX_EXACT_UINT8_SIMD_DIM) { - return ret_dist_func; - } - [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); #ifdef CPU_FEATURES_ARCH_AARCH64 @@ -825,13 +818,6 @@ dist_func_t Cosine_UINT8_GetDistFunc(size_t dim, unsigned char *alignment dist_func_t ret_dist_func = UINT8_Cosine; - // Above this dimension the 32-bit horizontal reduce in every uint8 SIMD kernel wraps, so fall - // back to the scalar kernel, which accumulates into a 64-bit ret_t and stays exact. See - // spaces::MAX_EXACT_UINT8_SIMD_DIM for why this is a fallback rather than a wider reduce. - if (dim > MAX_EXACT_UINT8_SIMD_DIM) { - return ret_dist_func; - } - [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); #ifdef CPU_FEATURES_ARCH_AARCH64 diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 2b99c5b40..1dc768c64 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -7,6 +7,7 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS static inline void L2SqrStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { __m512i va = _mm512_loadu_epi8(pVect1); // AVX512BW @@ -31,9 +32,13 @@ static inline void L2SqrStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { // with the corresponding 32-bit integer in src, and store the packed 32-bit results in dst. } +// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing +// the float results per chunk would round each one. always_inline, not merely inline: the chunked +// wrapper calls this twice, and without the attribute GCC outlines it once it has several callers, +// which costs the plain wrapper its inlining too. template // 0..63 -float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, - size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -92,11 +97,45 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVec } while (pVect1 < pEnd1); } - // The lanes hold sums of squared byte differences, so the horizontal total is unsigned and - // reaches 255*255*dim. Reading it as a signed int wrapped it negative from dimension 33,026. - // Still a 32-bit reduce, so this stays exact only to dimension 66,051 (65025 * 66052 exceeds - // UINT32_MAX). Unlike the inner product, which returns uint64, widening here would mean - // splitting the reduce; left as is because no dimension near that is realistic, but the bound - // is real and undocumented bounds are how the signed version survived this long. - return static_cast(static_cast(_mm512_reduce_add_epi32(sum))); + // Unsigned. The lanes hold sums of squared byte differences, so the total is unsigned and + // reaches 255*255*dim; reading it as a signed int wrapped it negative from dimension 33,026. + // Exact for up to spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + return static_cast(_mm512_reduce_add_epi32(sum)); +} + +template // 0..63 +float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, + size_t dimension) { + return static_cast( + UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1v, pVect2v, dimension)); +} + +// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit +// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is +// non-negative, so no individual lane can exceed the chunk total either. That is the whole +// correctness argument; no lane-distribution reasoning is needed. +// +// The first chunk absorbs the residual, leaving the remaining length a whole multiple of 64, so +// every later chunk satisfies the residual-0 kernel's precondition. +template // 0..63 +float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + const size_t first = residual + (chunk - residual) / 64 * 64; + uint64_t total = UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < chunk ? remaining : chunk; + total += UINT8_L2SqrImp_AVX512F_BW_VL_VNNI<0>(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return static_cast(total); } diff --git a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h index 654c0b3b1..5cd79f8bf 100644 --- a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h @@ -7,6 +7,7 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include __attribute__((always_inline)) static inline void @@ -50,8 +51,13 @@ L2SquareStep32(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum1, uint32x4_t pVect2 += 32; } +// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing +// the float results per chunk would round each one. always_inline because the chunked wrapper calls +// this twice, and GCC outlines a template once it has several callers, which also costs the plain +// wrapper its inlining. template // 0..63 -float UINT8_L2SqrSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_L2SqrImp_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -121,9 +127,43 @@ float UINT8_L2SqrSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, s total_sum = vaddq_u32(total_sum, sum2); total_sum = vaddq_u32(total_sum, sum3); - // Horizontal sum of the 4 elements in the combined sum register - uint32_t result = vaddvq_u32(total_sum); + // Horizontal sum of the 4 elements in the combined sum register. + // Unsigned: the total is a sum of squared byte differences, reaching 255*255*dim. Exact for up + // to spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + return vaddvq_u32(total_sum); +} + +template // 0..63 +float UINT8_L2SqrSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { + return static_cast(UINT8_L2SqrImp_NEON_DOTPROD(pVect1v, pVect2v, dimension)); +} - // Return the L2 squared distance as a float - return static_cast(result); +// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit +// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is +// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole +// correctness argument; no reasoning about how work spreads across lanes is needed. +// +// The first chunk absorbs the residual, so every later chunk is a whole multiple of 64 and matches +// the residual-0 kernel's precondition. +template // 0..63 +float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + const size_t first = residual + (chunk - residual) / 64 * 64; + uint64_t total = UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < chunk ? remaining : chunk; + total += UINT8_L2SqrImp_NEON_DOTPROD<0>(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return static_cast(total); } diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index feb485308..76601da69 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -7,6 +7,7 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include __attribute__((always_inline)) static inline void @@ -52,8 +53,13 @@ L2SquareStep32(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum1, uint32x4_t pVect2 += 32; } +// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing +// the float results per chunk would round each one. always_inline because the chunked wrapper calls +// this twice, and GCC outlines a template once it has several callers, which also costs the plain +// wrapper its inlining. template // 0..63 -float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_L2SqrImp_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { uint8_t *pVect1 = (uint8_t *)pVect1v; uint8_t *pVect2 = (uint8_t *)pVect2v; @@ -125,12 +131,43 @@ float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t di total_sum = vaddq_u32(total_sum, sum2); total_sum = vaddq_u32(total_sum, sum3); - // Horizontal sum of the 4 elements in the combined sum register + // Horizontal sum of the 4 elements in the combined sum register. // Unsigned: the total is a sum of squared byte differences, reaching 255*255*dim. As a signed - // int32 this wrapped negative from dimension 33,026. Still 32-bit, so exact only to dimension - // 66,051; see the AVX512 variant for why that is left rather than widened. - uint32_t result = vaddvq_u32(total_sum); + // int32 this wrapped negative from dimension 33,026. Exact for up to + // spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + return vaddvq_u32(total_sum); +} + +template // 0..63 +float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { + return static_cast(UINT8_L2SqrImp_NEON(pVect1v, pVect2v, dimension)); +} - // Return the L2 squared distance as a float - return static_cast(result); +// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit +// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is +// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole +// correctness argument; no reasoning about how work spreads across lanes is needed. +// +// The first chunk absorbs the residual, so every later chunk is a whole multiple of 64 and matches +// the residual-0 kernel's precondition. +template // 0..63 +float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + const size_t first = residual + (chunk - residual) / 64 * 64; + uint64_t total = UINT8_L2SqrImp_NEON(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < chunk ? remaining : chunk; + total += UINT8_L2SqrImp_NEON<0>(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return static_cast(total); } diff --git a/src/VecSim/spaces/L2/L2_SVE_UINT8.h b/src/VecSim/spaces/L2/L2_SVE_UINT8.h index 553db2169..221348355 100644 --- a/src/VecSim/spaces/L2/L2_SVE_UINT8.h +++ b/src/VecSim/spaces/L2/L2_SVE_UINT8.h @@ -7,6 +7,7 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include // Aligned step using svptrue_b8() @@ -26,8 +27,14 @@ inline void L2SquareStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t offset += chunk; // Move to the next set of uint8 elements } +// Split so the chunked wrapper below can fold each chunk's total in 64 bits; summing the float +// results per chunk would round each one. always_inline because the chunked wrapper calls this +// twice, and GCC outlines a template once it has several callers, which also costs the plain +// wrapper its inlining. static keeps each translation unit's copy to itself: SVE.cpp and SVE2.cpp +// both include this header, and other headers define the same name with different bodies. template -float UINT8_L2SqrSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { +__attribute__((always_inline)) static inline uint32_t +UINT8_L2SqrImp_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { const uint8_t *pVect1 = reinterpret_cast(pVect1v); const uint8_t *pVect2 = reinterpret_cast(pVect2v); @@ -85,5 +92,50 @@ float UINT8_L2SqrSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimen sum0 = svadd_u32_x(all, sum0, sum1); sum2 = svadd_u32_x(all, sum2, sum3); svuint32_t sum_all = svadd_u32_x(all, sum0, sum2); - return svaddv_u32(svptrue_b32(), sum_all); + // svaddv_u32 reduces into a 64-bit scalar. The total is a sum of squared byte differences, + // reaching 255*255*dim, and is exact for up to spaces::UINT8_CHUNK_ELEMENTS elements, which is + // what the caller guarantees. + return static_cast(svaddv_u32(svptrue_b32(), sum_all)); +} + +template +float UINT8_L2SqrSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { + return static_cast( + UINT8_L2SqrImp_SVE(pVect1v, pVect2v, dimension)); +} + +// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit +// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is +// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole +// correctness argument; no reasoning about how work spreads across lanes is needed. +template +float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + // The SVE vector length is a runtime value, so unlike the fixed-width kernels the split is + // computed here rather than at compile time. chunk_size matches the kernel's 4-accumulator main + // loop, and tail is the part the template parameters describe. + const size_t chunk_size = 4 * svcntb(); + const size_t tail = dimension % chunk_size; + const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; + const size_t first = tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; + + // first keeps this instantiation's own residual shape: it is congruent to dimension modulo + // chunk_size, so partial_chunk and additional_steps still describe its tail. + uint64_t total = UINT8_L2SqrImp_SVE(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + // remaining is a whole multiple of chunk_size, and so is every step, which is the + // shape: no partial vector and no leftover single steps. + while (remaining) { + const size_t step = remaining < max_step ? remaining : max_step; + total += UINT8_L2SqrImp_SVE(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return static_cast(total); } diff --git a/src/VecSim/spaces/L2_space.cpp b/src/VecSim/spaces/L2_space.cpp index 41c394331..b4ca3669a 100644 --- a/src/VecSim/spaces/L2_space.cpp +++ b/src/VecSim/spaces/L2_space.cpp @@ -468,12 +468,6 @@ dist_func_t L2_UINT8_GetDistFunc(size_t dim, unsigned char *alignment, dist_func_t ret_dist_func = UINT8_L2Sqr; - // Above this dimension the 32-bit horizontal reduce in every uint8 SIMD kernel wraps, so fall - // back to the scalar kernel, which accumulates into a 64-bit ret_t and stays exact. See - // spaces::MAX_EXACT_UINT8_SIMD_DIM for why this is a fallback rather than a wider reduce. - if (dim > MAX_EXACT_UINT8_SIMD_DIM) { - return ret_dist_func; - } // Optimizations assume at least 32 uint8. If we have less, we use the naive implementation. [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); diff --git a/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp b/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp index 97da55546..c9f73c9ed 100644 --- a/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp +++ b/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp @@ -42,21 +42,43 @@ dist_func_t Choose_INT8_Cosine_implementation_AVX512F_BW_VL_VNNI(size_t d return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays what it was. dist_func_t Choose_UINT8_L2_implementation_AVX512F_BW_VL_VNNI(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI); + } return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant, which folds each chunk's +// exact 32-bit total into 64 bits. Chosen here, once per index, so the plain kernel below carries +// no branch and stays what it was. dist_func_t Choose_UINT8_IP_implementation_AVX512F_BW_VL_VNNI(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, + UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI); + } return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant, which folds each chunk's +// exact 32-bit total into 64 bits. Chosen here, once per index, so the plain kernel below carries +// no branch and stays what it was. dist_func_t Choose_UINT8_Cosine_implementation_AVX512F_BW_VL_VNNI(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, + UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI); + } return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/NEON.cpp b/src/VecSim/spaces/functions/NEON.cpp index 0c9a286e3..d33dc0f6e 100644 --- a/src/VecSim/spaces/functions/NEON.cpp +++ b/src/VecSim/spaces/functions/NEON.cpp @@ -30,9 +30,15 @@ dist_func_t Choose_INT8_IP_implementation_NEON(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON); + } return ret_dist_func; } @@ -54,9 +60,15 @@ dist_func_t Choose_INT8_Cosine_implementation_NEON(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON); + } return ret_dist_func; } @@ -71,9 +83,15 @@ dist_func_t Choose_INT8_L2_implementation_NEON(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON); + } return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/NEON_DOTPROD.cpp b/src/VecSim/spaces/functions/NEON_DOTPROD.cpp index 12f762093..54fd74ad4 100644 --- a/src/VecSim/spaces/functions/NEON_DOTPROD.cpp +++ b/src/VecSim/spaces/functions/NEON_DOTPROD.cpp @@ -24,9 +24,16 @@ dist_func_t Choose_INT8_IP_implementation_NEON_DOTPROD(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON_DOTPROD); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, + UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON_DOTPROD); + } return ret_dist_func; } @@ -36,9 +43,15 @@ dist_func_t Choose_INT8_Cosine_implementation_NEON_DOTPROD(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_DOTPROD); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_DOTPROD_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_DOTPROD); + } return ret_dist_func; } @@ -48,9 +61,15 @@ dist_func_t Choose_INT8_L2_implementation_NEON_DOTPROD(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_DOTPROD); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked); + } else { + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_DOTPROD); + } return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/SVE.cpp b/src/VecSim/spaces/functions/SVE.cpp index bd197c84c..c473a7ed3 100644 --- a/src/VecSim/spaces/functions/SVE.cpp +++ b/src/VecSim/spaces/functions/SVE.cpp @@ -86,21 +86,39 @@ dist_func_t Choose_INT8_Cosine_implementation_SVE(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE_Chunked, dim, svcntb); + } else { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); + } return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE_Chunked, dim, svcntb); + } else { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); + } return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE_Chunked, dim, svcntb); + } else { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); + } return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/SVE2.cpp b/src/VecSim/spaces/functions/SVE2.cpp index 9eea81523..0f59bb721 100644 --- a/src/VecSim/spaces/functions/SVE2.cpp +++ b/src/VecSim/spaces/functions/SVE2.cpp @@ -82,21 +82,39 @@ dist_func_t Choose_INT8_Cosine_implementation_SVE2(size_t dim) { return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE_Chunked, dim, svcntb); + } else { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); + } return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE_Chunked, dim, svcntb); + } else { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); + } return ret_dist_func; } +// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per +// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); + if (dim > spaces::UINT8_CHUNK_ELEMENTS) { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE_Chunked, dim, svcntb); + } else { + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); + } return ret_dist_func; } diff --git a/src/VecSim/spaces/spaces.h b/src/VecSim/spaces/spaces.h index 482cb3693..6419f9772 100644 --- a/src/VecSim/spaces/spaces.h +++ b/src/VecSim/spaces/spaces.h @@ -52,40 +52,24 @@ static int inline is_little_endian() { return *(char *)&x; } -// Largest dimension for which a uint8 SIMD kernel may be selected. +// A full-range uint8 product is at most 255 * 255 = 65,025, so a 32-bit accumulator is exact +// through floor(UINT32_MAX / 65,025) = 66,051 terms. Twice the old signed limit of 33,025: the +// accumulation was always fine, the top bit was being read as a sign. // -// The uint8 kernels accumulate products or squared differences of bytes into 32-bit SIMD lanes and -// finish with a 32-bit horizontal reduce. Per element that caps at 255 * 255 = 65025, so the total -// is 65025 * dim, exactly representable in uint32 through dim 66,051 and wrapping at 66,052. Note -// this is twice the old signed limit of 33,025: nothing about the accumulation changed there, the -// top bit was simply being read as a sign, which is why the unsigned reduce costs nothing. +// Rather than cap the dimension there, the kernels accumulate in chunks of this many elements and +// fold each chunk's exact 32-bit total into a 64-bit scalar, which makes them exact at any +// dimension. 65,536 is chosen because it is under 66,051, so the existing 32-bit reduce needs no +// change, and because it is a whole number of 64-byte blocks, so a chunk boundary always lands on +// one. // -// Above this dimension the choosers hand back the scalar kernel, which accumulates into a 64-bit -// ret_t and is exact to roughly dim 2.8e14. One comparison at index creation, no branch in any -// kernel, and no second set of instantiations. Two alternatives were considered and rejected: -// -// * Widening the horizontal reduce. Measured on an Ice Lake-SP Xeon it costs 4 extra uops in the -// epilogue: +20% at dim 32, +8-11% across dim 55-200, +4-5% at dim 900-1024, on byte-identical -// loop code. That is a dependency-chain cost rather than a throughput one, which is why an -// instruction count understates it. It also only moves the limit rather than removing it, and -// to a different place per ISA: roughly dim 1,056,816 on AVX-512, but only 264,204 on NEON, -// where four accumulators are combined with vaddq_u32 in 32 bits before the widening reduce -// ever sees them. -// -// * Chunking the accumulation and flushing into a 64-bit total. Exact at any dimension and cheap -// if the chunk loop lives in the wrapper rather than the kernel (+2 instructions on the fast -// path, versus +12 to +21 when placed inside the kernel). Deferred rather than dismissed: it -// is only worth the restructuring if dimensions above 66,051 become a real workload. -// -// Nothing comparable supports that range today. Lucene caps its scalar-quantized format at 1,024 -// dimensions and Elasticsearch caps dense vectors at 4,096, both of which keep a 32-bit -// accumulator safe by contract. Faiss's QT_8bit_direct path accumulates full-range bytes into -// int/32-bit lanes with no widening, so it carries the same theoretical limit. Qdrant quantizes to -// 0..127 instead of 0..255, which lowers the per-element cap to 16,129 and pushes the signed limit -// out to dim 133,144, and its raw uint8 metric still sums into i32. So the scalar fallback here is -// already stricter than the alternatives, and optimizing the SIMD path beyond 66,051 would be -// optimizing a range none of them accept. -static constexpr size_t MAX_EXACT_UINT8_SIMD_DIM = 66051; +// The margins are deliberately loose, because the tight version was wrong. A previous attempt +// widened the reduce and bounded the dimension at 4 * 66,051, on the assumption that products +// spread evenly across NEON's four lanes after its 32-bit vaddq_u32 merge. The even case already +// sat within 1,020 of UINT32_MAX while a masked residual load can put 1,040,400 into a single lane, +// so lanes wrapped before the widened reduce saw them. At this chunk size the per-chunk total has +// 33 million to spare and a NEON lane has 3.2 billion, so neither constraint is close and no +// per-ISA lane audit is needed. +static constexpr size_t UINT8_CHUNK_ELEMENTS = 65536; static inline auto getCpuOptimizationFeatures(const void *arch_opt = nullptr) { diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 83da413ea..bfc198059 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -2222,20 +2222,64 @@ TEST_F(SpacesTest, UINT8_L2Sqr_and_InnerProduct_are_exact_past_int32) { } } -// Past spaces::MAX_EXACT_UINT8_SIMD_DIM the 32-bit horizontal reduce in every uint8 SIMD kernel -// wraps, so the choosers must hand back the scalar kernel, which accumulates into a 64-bit ret_t. -// Asserting the returned pointer is the point of this test: on a host with no uint8 SIMD support -// the value comparisons below would pass either way, but the pointer identity would not. -TEST_F(SpacesTest, UINT8_dispatchers_fall_back_to_scalar_past_the_exact_dim) { - constexpr size_t dim = spaces::MAX_EXACT_UINT8_SIMD_DIM + 1; - unsigned char alignment = 0; - - EXPECT_EQ(L2_UINT8_GetDistFunc(dim, &alignment, nullptr), UINT8_L2Sqr); - EXPECT_EQ(IP_UINT8_GetDistFunc(dim, &alignment, nullptr), UINT8_InnerProduct); - EXPECT_EQ(Cosine_UINT8_GetDistFunc(dim, &alignment, nullptr), UINT8_Cosine); +// Past spaces::UINT8_CHUNK_ELEMENTS the uint8 SIMD kernels accumulate in chunks: each chunk's total +// still fits the 32-bit accumulators (65025 * 65536 <= UINT32_MAX), and the per-chunk totals are +// folded in 64 bits. The dispatched kernel must therefore agree exactly with the scalar kernel, +// which accumulates the whole vector into a 64-bit ret_t. Exact equality is the right assertion +// because both paths convert the same integer total to float once, at the end. +// +// All-255 against all-0 is the worst case and puts the L2 total past UINT32_MAX from dimension +// 66,052, so the multi-chunk dimensions below genuinely exercise the 64-bit fold. The existing +// UINT8 suites stop at dim 128, which is why the wrap went unseen. +TEST_F(SpacesTest, UINT8_dispatched_kernels_are_exact_across_the_chunk_boundary) { + // Below the boundary, on it, one past it (whose last chunk is a single 64-element block), an + // exact multiple of it, and dimensions spanning two and three chunks. + for (const size_t dim : {65535UL, 65536UL, 65537UL, 65600UL, 131072UL, 131109UL, 200000UL}) { + // The cosine kernels read a float norm from just past the payload, so size for both. + std::vector ones(dim + sizeof(float), 255); + std::vector zeros(dim + sizeof(float), 0); + std::vector ramp(dim + sizeof(float)); + for (size_t i = 0; i < dim; i++) { + ramp[i] = static_cast(i % 256); + } + const float norm = std::sqrt(255.0f * 255.0f * static_cast(dim)); + memcpy(ones.data() + dim, &norm, sizeof(float)); + memcpy(ramp.data() + dim, &norm, sizeof(float)); + + unsigned char alignment = 0; + auto l2 = L2_UINT8_GetDistFunc(dim, &alignment, nullptr); + auto ip = IP_UINT8_GetDistFunc(dim, &alignment, nullptr); + auto cosine = Cosine_UINT8_GetDistFunc(dim, &alignment, nullptr); + + // Worst case: the largest total the byte range allows. + EXPECT_EQ(UINT8_L2Sqr(ones.data(), zeros.data(), dim), l2(ones.data(), zeros.data(), dim)) + << "L2 all-255 vs all-0, dim " << dim; + EXPECT_EQ(UINT8_InnerProduct(ones.data(), ones.data(), dim), + ip(ones.data(), ones.data(), dim)) + << "IP all-255, dim " << dim; + EXPECT_EQ(UINT8_Cosine(ones.data(), ones.data(), dim), + cosine(ones.data(), ones.data(), dim)) + << "Cosine all-255, dim " << dim; + + // A varying pattern, so the residual and chunk seams have to line up element for element + // rather than merely produce the right sum of identical values. + EXPECT_EQ(UINT8_L2Sqr(ramp.data(), ones.data(), dim), l2(ramp.data(), ones.data(), dim)) + << "L2 ramp vs all-255, dim " << dim; + EXPECT_EQ(UINT8_InnerProduct(ramp.data(), ones.data(), dim), + ip(ramp.data(), ones.data(), dim)) + << "IP ramp vs all-255, dim " << dim; + EXPECT_EQ(UINT8_Cosine(ramp.data(), ones.data(), dim), + cosine(ramp.data(), ones.data(), dim)) + << "Cosine ramp vs all-255, dim " << dim; + } +} - // And one dimension below the threshold the SIMD kernel is still eligible, so the guard is a - // boundary rather than a blanket disable. Only assert this where a uint8 SIMD tier exists. +// The chooser picks the chunked kernel once per index rather than branching per call, so assert the +// switch actually happens. Both dimensions are a multiple of 64, so they map to the same residual +// instantiation: any difference in the returned pointer can only come from the chunked family being +// chosen. Only meaningful where a uint8 SIMD tier exists, since otherwise both are the scalar +// kernel. +TEST_F(SpacesTest, UINT8_choosers_switch_to_the_chunked_kernel_past_the_chunk_size) { const auto features = getCpuOptimizationFeatures(); const bool has_uint8_simd = #ifdef CPU_FEATURES_ARCH_X86_64 @@ -2243,19 +2287,26 @@ TEST_F(SpacesTest, UINT8_dispatchers_fall_back_to_scalar_past_the_exact_dim) { #else features.sve2 || features.sve || features.asimddp || features.asimd; #endif - if (has_uint8_simd) { - EXPECT_NE(L2_UINT8_GetDistFunc(spaces::MAX_EXACT_UINT8_SIMD_DIM, &alignment, nullptr), - UINT8_L2Sqr); + if (!has_uint8_simd) { + GTEST_SKIP() << "no uint8 SIMD tier on this host"; } - // The scalar path must actually be exact here, which is the reason the fallback is safe. - // All-255 against all-0 gives 255^2 * dim, past UINT32_MAX at this dimension. - std::vector v1(dim + sizeof(float), 255); - std::vector v2(dim + sizeof(float), 0); - const double expected_l2 = 255.0 * 255.0 * static_cast(dim); - EXPECT_GT(expected_l2, 4294967295.0) << "test would not exercise the 32-bit wrap"; - const double l2 = static_cast(UINT8_L2Sqr(v1.data(), v2.data(), dim)); - EXPECT_LT(std::abs(l2 - expected_l2) / expected_l2, 1e-6); + constexpr size_t plain = spaces::UINT8_CHUNK_ELEMENTS; // on the boundary, not chunked + constexpr size_t chunked = spaces::UINT8_CHUNK_ELEMENTS * 2; // same residual, chunked + unsigned char alignment = 0; + + EXPECT_NE(L2_UINT8_GetDistFunc(plain, &alignment, nullptr), + L2_UINT8_GetDistFunc(chunked, &alignment, nullptr)); + EXPECT_NE(IP_UINT8_GetDistFunc(plain, &alignment, nullptr), + IP_UINT8_GetDistFunc(chunked, &alignment, nullptr)); + EXPECT_NE(Cosine_UINT8_GetDistFunc(plain, &alignment, nullptr), + Cosine_UINT8_GetDistFunc(chunked, &alignment, nullptr)); + + // And the SIMD kernel is still what gets chosen past the boundary: the chunked variant replaces + // the plain one, it does not fall back to scalar. + EXPECT_NE(L2_UINT8_GetDistFunc(chunked, &alignment, nullptr), UINT8_L2Sqr); + EXPECT_NE(IP_UINT8_GetDistFunc(chunked, &alignment, nullptr), UINT8_InnerProduct); + EXPECT_NE(Cosine_UINT8_GetDistFunc(chunked, &alignment, nullptr), UINT8_Cosine); } class SQ8_FP32_SpacesOptimizationTest : public testing::TestWithParam {}; From 73bfc37faede07fb9525c519937e511c52444319 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Mon, 17 Aug 2026 16:42:00 +0300 Subject: [PATCH 03/19] test(uint8): sweep every residual past the chunk boundary, and every tier The boundary test sampled seven dimensions and went through the generic dispatcher. Two gaps followed from that. First, only seven of the 64 residual instantiations were covered past the boundary, so a seam between the residual-bearing first chunk and the residual-0 chunks after it could have survived in the other 57. 65,600 and 196,608 are both multiples of 64, so base + r has residual r; sweeping r over 0..63 at both bases covers every shape one chunk past the boundary and again three chunks past it. A ramp against all-255 is position sensitive, so a seam that skips or double-counts elements changes the total rather than cancelling out, and the total stays above UINT32_MAX so the 64-bit fold is under test throughout. Second, the dispatcher only ever returns the best tier the host supports, so on a machine with SVE the NEON and NEON_DOTPROD chunked kernels never ran at all. The new tier test calls each compiled-in chooser directly, still gated on the CPU supporting it. Co-Authored-By: Claude Opus 5 --- tests/unit/test_spaces.cpp | 113 +++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index bfc198059..42ef4fc14 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -2274,6 +2274,119 @@ TEST_F(SpacesTest, UINT8_dispatched_kernels_are_exact_across_the_chunk_boundary) } } +// The boundary test above samples dimensions; this sweeps every residual instantiation. 65,600 and +// 196,608 are both multiples of 64, so base + r has residual r: one chunk past the boundary, then +// three chunks past it, so the seam between the residual-bearing first chunk and the residual-0 +// chunks after it is exercised for all 64 shapes. A ramp against all-255 is position sensitive, so +// a seam that double-counts or skips elements changes the total rather than cancelling out, and the +// total still passes UINT32_MAX (about 32,500 * dim) so the 64-bit fold is under test throughout. +TEST_F(SpacesTest, UINT8_dispatched_kernels_are_exact_at_every_residual_past_the_chunk_boundary) { + constexpr size_t max_dim = 196608 + 63; + std::vector ones(max_dim + sizeof(float), 255); + std::vector ramp(max_dim + sizeof(float)); + for (size_t i = 0; i < max_dim; i++) { + ramp[i] = static_cast(i % 256); + } + + for (const size_t base : {65600UL, 196608UL}) { + for (size_t r = 0; r < 64; r++) { + const size_t dim = base + r; + // The cosine kernels read a float norm from just past the payload, which moves with + // dim. + const float norm = std::sqrt(255.0f * 255.0f * static_cast(dim)); + memcpy(ones.data() + dim, &norm, sizeof(float)); + memcpy(ramp.data() + dim, &norm, sizeof(float)); + + unsigned char alignment = 0; + const void *a = ramp.data(); + const void *b = ones.data(); + + EXPECT_EQ(UINT8_L2Sqr(a, b, dim), + L2_UINT8_GetDistFunc(dim, &alignment, nullptr)(a, b, dim)) + << "L2 at dim " << dim << " (residual " << r << ")"; + EXPECT_EQ(UINT8_InnerProduct(a, b, dim), + IP_UINT8_GetDistFunc(dim, &alignment, nullptr)(a, b, dim)) + << "IP at dim " << dim << " (residual " << r << ")"; + EXPECT_EQ(UINT8_Cosine(a, b, dim), + Cosine_UINT8_GetDistFunc(dim, &alignment, nullptr)(a, b, dim)) + << "Cosine at dim " << dim << " (residual " << r << ")"; + } + } +} + +// The tests above go through the generic dispatcher, which only ever returns the best tier this +// host supports, so on an ARM machine with SVE the NEON and NEON_DOTPROD chunked kernels are never +// executed. Reach every compiled-in tier directly instead. Each tier is still gated on the CPU +// actually supporting it, since calling an unsupported kernel faults. +TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { + // Boundary (plain family), one past it with residuals 0/1/63, two chunks, and a ragged + // multiple. + const std::vector dims = {65536, 65600, 65601, 65663, 131072, 200000}; + const auto opt = getCpuOptimizationFeatures(); + + constexpr size_t max_dim = 200000; + std::vector ones(max_dim + sizeof(float), 255); + std::vector ramp(max_dim + sizeof(float)); + for (size_t i = 0; i < max_dim; i++) { + ramp[i] = static_cast(i % 256); + } + + for (const size_t dim : dims) { + const float norm = std::sqrt(255.0f * 255.0f * static_cast(dim)); + memcpy(ones.data() + dim, &norm, sizeof(float)); + memcpy(ramp.data() + dim, &norm, sizeof(float)); + const void *a = ramp.data(); + const void *b = ones.data(); + + const float want_l2 = UINT8_L2Sqr(a, b, dim); + const float want_ip = UINT8_InnerProduct(a, b, dim); + const float want_cos = UINT8_Cosine(a, b, dim); + + auto check = [&](const char *tier, dist_func_t l2, dist_func_t ip, + dist_func_t cosine) { + EXPECT_EQ(want_l2, l2(a, b, dim)) << "L2 " << tier << " dim " << dim; + EXPECT_EQ(want_ip, ip(a, b, dim)) << "IP " << tier << " dim " << dim; + EXPECT_EQ(want_cos, cosine(a, b, dim)) << "Cosine " << tier << " dim " << dim; + }; + +#ifdef OPT_AVX512_F_BW_VL_VNNI + if (opt.avx512f && opt.avx512bw && opt.avx512vl && opt.avx512vnni) { + check("AVX512F_BW_VL_VNNI", Choose_UINT8_L2_implementation_AVX512F_BW_VL_VNNI(dim), + Choose_UINT8_IP_implementation_AVX512F_BW_VL_VNNI(dim), + Choose_UINT8_Cosine_implementation_AVX512F_BW_VL_VNNI(dim)); + } +#endif +#ifdef OPT_SVE2 + if (opt.sve2) { + check("SVE2", Choose_UINT8_L2_implementation_SVE2(dim), + Choose_UINT8_IP_implementation_SVE2(dim), + Choose_UINT8_Cosine_implementation_SVE2(dim)); + } +#endif +#ifdef OPT_SVE + if (opt.sve) { + check("SVE", Choose_UINT8_L2_implementation_SVE(dim), + Choose_UINT8_IP_implementation_SVE(dim), + Choose_UINT8_Cosine_implementation_SVE(dim)); + } +#endif +#ifdef OPT_NEON_DOTPROD + if (opt.asimddp) { + check("NEON_DOTPROD", Choose_UINT8_L2_implementation_NEON_DOTPROD(dim), + Choose_UINT8_IP_implementation_NEON_DOTPROD(dim), + Choose_UINT8_Cosine_implementation_NEON_DOTPROD(dim)); + } +#endif +#ifdef OPT_NEON + if (opt.asimd) { + check("NEON", Choose_UINT8_L2_implementation_NEON(dim), + Choose_UINT8_IP_implementation_NEON(dim), + Choose_UINT8_Cosine_implementation_NEON(dim)); + } +#endif + } +} + // The chooser picks the chunked kernel once per index rather than branching per call, so assert the // switch actually happens. Both dimensions are a multiple of 64, so they map to the same residual // instantiation: any difference in the returned pointer can only come from the chunked family being From e7ae3bd3ebedcc6f6968d5ce3744e34518878100 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Mon, 17 Aug 2026 17:45:25 +0300 Subject: [PATCH 04/19] test(uint8): report which tiers the per-tier test actually exercised A tier the CPU does not support is skipped by a plain if, so on a host with no uint8 SIMD at all the test passed without checking a single SIMD kernel, and the log gave no way to tell. Record and print the tiers covered per dimension, so a green run states what it proved rather than leaving it to be inferred from the host's feature flags. Co-Authored-By: Claude Opus 5 --- tests/unit/test_spaces.cpp | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 42ef4fc14..85fafe297 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -13,6 +13,8 @@ #include #include #include +#include +#include #include "gtest/gtest.h" #include "VecSim/spaces/space_includes.h" @@ -2342,8 +2344,13 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { const float want_ip = UINT8_InnerProduct(a, b, dim); const float want_cos = UINT8_Cosine(a, b, dim); + // Track which tiers ran. A tier the CPU lacks is skipped silently, so without this the + // whole test would pass vacuously on a host with no uint8 SIMD at all, and the log would + // not say which kernels were actually covered. + std::vector tiers_checked; auto check = [&](const char *tier, dist_func_t l2, dist_func_t ip, dist_func_t cosine) { + tiers_checked.emplace_back(tier); EXPECT_EQ(want_l2, l2(a, b, dim)) << "L2 " << tier << " dim " << dim; EXPECT_EQ(want_ip, ip(a, b, dim)) << "IP " << tier << " dim " << dim; EXPECT_EQ(want_cos, cosine(a, b, dim)) << "Cosine " << tier << " dim " << dim; @@ -2384,6 +2391,15 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { Choose_UINT8_Cosine_implementation_NEON(dim)); } #endif + + std::string covered; + for (const auto &t : tiers_checked) { + covered += covered.empty() ? t : ", " + t; + } + RecordProperty("tiers_at_dim_" + std::to_string(dim), covered); + std::cout << " dim " << dim << " covered tiers: " + << (covered.empty() ? "" : covered) + << std::endl; } } From 068c76b9489cac0d276b4fc6e98d245f0fef71b2 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 08:16:20 +0300 Subject: [PATCH 05/19] docs(uint8): correct the linkage comment, the collision is live on gcc 12.3 The comment claimed both copies of UINT8_InnerProductImp are fully inlined at -O2 so nothing collides, and that static was therefore defensive. Measured on an aarch64 host with gcc 12.3: not inlined. Both objects emit one weak COMDAT symbol per residual under the same mangled name, the linker keeps a single body, and on main the plain NEON inner product and cosine wrappers branch into the NEON_DOTPROD body and execute udot. That faults on a core with asimd but without asimddp, which includes Neoverse-N1 and Graviton2. x86-64 gcc 13/14 and aarch64 clang 18 do inline it and do not collide, which is why the earlier check came back clean and why this cannot be left to the toolchain. static is load-bearing on at least one shipping compiler. Co-Authored-By: Claude Opus 5 --- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index e2375852e..e29314694 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -38,14 +38,14 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { // Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing // the float results per chunk would round each one. // -// static: the NEON and NEON_DOTPROD headers define this same name with the same signature and -// different bodies, and both are compiled into an ARM build. At -O2 both are fully inlined today, -// so no symbol is emitted and nothing collides, but nothing guarantees that: outline either one and -// both objects define one mangled name, and a NEON call site could reach the DOTPROD body, which -// needs the dotprod feature. Internal linkage costs nothing and removes the possibility, which -// matters more now that the chunked wrapper below adds call sites. always_inline for the same -// reason from the other direction: GCC does outline this once it has several callers, and that also -// costs the plain wrapper its inlining. +// static and always_inline are both load-bearing here, not hygiene. The NEON and NEON_DOTPROD +// headers define this same name with the same signature and different bodies, and both are compiled +// into an ARM build. Whether that collides depends on whether the compiler outlines: aarch64 +// gcc 12.3 at -O2 does, emitting one weak COMDAT symbol per residual from both objects, and the +// linker keeps a single body for both. Measured on main with that compiler, the plain NEON inner +// product and cosine wrappers branched into the DOTPROD body and executed udot, which faults on a +// core that has asimd but not asimddp. x86-64 gcc 13/14 and aarch64 clang 18 inline it and do not +// collide, which is precisely why this cannot be left to the toolchain. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { From 7a70200d0eb325cc1d6ede63e2eca748fd1aa212 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 08:25:34 +0300 Subject: [PATCH 06/19] perf(uint8): stop the chunked wrapper pessimising its first chunk, and cheapen the IP epilogue Two findings from benchmark runs on an Ice Lake-SP host and an ARM host, both measured rather than inferred. First, the chunked wrapper was 8-9.5% slower per element than the plain kernel on the stretch its first chunk covers. Cause: the first chunk's length was a compile-time constant, so its loop had a compile-time trip count, and GCC then split that loop's accumulator and copied it in and out every 64 elements. The inner product loop went 12 instructions with no register moves to 13 with two; L2 went 14/0 to 15/2. The later-chunks loop was always fine. Fixed by two changes to the chunked wrappers only, leaving the plain path alone: - the first chunk's length is now a runtime min against the dimension, so the trip count is not a constant. Both loops are back to 12/0 and 14/0, matching the plain kernels exactly, at every residual. The min also makes the chunked wrapper correct at any dimension rather than only past the chunk size. - the residual-0 chunks now go through one out-of-line copy of the kernel instead of being inlined into all 64 chunked wrappers. One call per 65,536 elements is unmeasurable, and it drops the AVX512 inner product family from 41,267 to 34,229 bytes of text, against 14,735 for the plain family alone. Second, the inner product epilogue. Converting the unsigned total to float and subtracting in float costs one instruction more than main's integer subtract plus a single signed convert, and rounds twice instead of once; it measured about 1% slower across all four IP benchmark groups. Restored the integer form, which is also what INT8_InnerProduct already does. The cast to int64_t is required because ret_t is unsigned for uint8, so 1 minus the total would otherwise wrap. Applied to the scalar kernel too, so scalar and SIMD stay bit-identical, which the exactness tests assert. Co-Authored-By: Claude Opus 5 --- src/VecSim/spaces/IP/IP.cpp | 5 +++- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 23 +++++++++++++++---- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 23 +++++++++++++++---- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 22 ++++++++++++++---- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 23 ++++++++++++++----- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 17 ++++++++++++-- src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h | 16 +++++++++++-- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 16 +++++++++++-- src/VecSim/spaces/L2/L2_SVE_UINT8.h | 15 ++++++++++-- 9 files changed, 130 insertions(+), 30 deletions(-) diff --git a/src/VecSim/spaces/IP/IP.cpp b/src/VecSim/spaces/IP/IP.cpp index cefafb773..1b2bc7523 100644 --- a/src/VecSim/spaces/IP/IP.cpp +++ b/src/VecSim/spaces/IP/IP.cpp @@ -277,7 +277,10 @@ float INT8_Cosine(const void *pVect1v, const void *pVect2v, size_t dimension) { float UINT8_InnerProduct(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto *pVect1 = static_cast(pVect1v); const auto *pVect2 = static_cast(pVect2v); - return 1.0f - static_cast(INTEGER_InnerProductImp(pVect1, pVect2, dimension)); + // Integer subtract then a single conversion, matching INT8_InnerProduct above. The cast to a + // signed type is required because ret_t is unsigned for uint8: 1 - an unsigned total would + // wrap. The SIMD kernels do the same, so their results stay bit-identical to this one. + return 1 - static_cast(INTEGER_InnerProductImp(pVect1, pVect2, dimension)); } float UINT8_Cosine(const void *pVect1v, const void *pVect2v, size_t dimension) { diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index 3c08844c1..18c91ac78 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -103,7 +103,7 @@ template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return 1 - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); } template // 0..63 float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, @@ -114,6 +114,15 @@ float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVe return 1.0f - ip / (norm_v1 * norm_v2); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pVect2, + size_t dimension) { + return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); +} + // Chunked variants, selected by the chooser for dimensions past spaces::UINT8_CHUNK_ELEMENTS. Each // chunk's 32-bit total is exact, and the chunks are folded in 64 bits, so these are exact at any // dimension. The plain wrappers above are left untouched so their inlining is unaffected, and the @@ -128,7 +137,11 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - const size_t first = residual + (chunk - residual) / 64 * 64; + // Runtime min rather than the constant alone: with a compile-time trip count GCC split this + // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice + // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. + constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; + const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); pVect1 += first; pVect2 += first; @@ -136,7 +149,7 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v while (remaining) { const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_InnerProductImp<0>(pVect1, pVect2, step); + total += UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; @@ -147,8 +160,8 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return 1 - static_cast( + UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); } template // 0..63 diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index 5738828a7..c09239878 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -111,7 +111,7 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return 1 - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); } template // 0..63 @@ -122,6 +122,15 @@ float UINT8_CosineSIMD_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, si return 1.0f - ip / (norm_v1 * norm_v2); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_InnerProductFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, + size_t dimension) { + return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); +} + // See the NEON header for why each chunk's 32-bit total is exact and why the first chunk absorbs // the residual. template // 0..63 @@ -131,7 +140,11 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - const size_t first = residual + (chunk - residual) / 64 * 64; + // Runtime min rather than the constant alone: with a compile-time trip count GCC split this + // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice + // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. + constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; + const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); pVect1 += first; pVect2 += first; @@ -139,7 +152,7 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v while (remaining) { const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_InnerProductImp<0>(pVect1, pVect2, step); + total += UINT8_InnerProductFullChunk_NEON_DOTPROD(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; @@ -150,8 +163,8 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return 1 - static_cast( + UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); } template // 0..63 diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index e29314694..8ea2acda9 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -126,7 +126,7 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension template // 0..15 float UINT8_InnerProductSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return 1 - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); } template // 0..63 @@ -137,6 +137,14 @@ float UINT8_CosineSIMD_NEON(const void *pVect1v, const void *pVect2v, size_t dim return 1.0f - ip / (norm_v1 * norm_v2); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_InnerProductFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); +} + // Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit // total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is // non-negative, so no accumulator lane can exceed the chunk total either. That is the whole @@ -151,7 +159,11 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - const size_t first = residual + (chunk - residual) / 64 * 64; + // Runtime min rather than the constant alone: with a compile-time trip count GCC split this + // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice + // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. + constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; + const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); pVect1 += first; pVect2 += first; @@ -159,7 +171,7 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v while (remaining) { const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_InnerProductImp<0>(pVect1, pVect2, step); + total += UINT8_InnerProductFullChunk_NEON(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; @@ -170,8 +182,8 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template // 0..63 float UINT8_InnerProductSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return 1 - static_cast( + UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); } template // 0..63 diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index fe3043692..a9a23e103 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -97,8 +97,8 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension template float UINT8_InnerProductSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - static_cast(UINT8_InnerProductImp( - pVect1v, pVect2v, dimension)); + return 1 - static_cast(UINT8_InnerProductImp( + pVect1v, pVect2v, dimension)); } template @@ -110,6 +110,14 @@ float UINT8_CosineSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dime return 1.0f - ip / (norm_v1 * norm_v2); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_InnerProductFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductImp(pVect1, pVect2, dimension); +} + // Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit // total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is // non-negative, so no accumulator lane can exceed the chunk total either. That is the whole @@ -126,7 +134,10 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const size_t chunk_size = 4 * svcntb(); const size_t tail = dimension % chunk_size; const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; - const size_t first = tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; + const size_t first_chunk = + tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; + // Clamped so this wrapper is correct at any dimension, not only past the chunk size. + const size_t first = dimension < first_chunk ? dimension : first_chunk; // first keeps this instantiation's own residual shape: it is congruent to dimension modulo // chunk_size, so partial_chunk and additional_steps still describe its tail. @@ -139,7 +150,7 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v // shape: no partial vector and no leftover single steps. while (remaining) { const size_t step = remaining < max_step ? remaining : max_step; - total += UINT8_InnerProductImp(pVect1, pVect2, step); + total += UINT8_InnerProductFullChunk_SVE(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; @@ -150,8 +161,8 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template float UINT8_InnerProductSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1.0f - static_cast(UINT8_InnerProductChunkedImp( - pVect1v, pVect2v, dimension)); + return 1 - static_cast(UINT8_InnerProductChunkedImp( + pVect1v, pVect2v, dimension)); } template diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 1dc768c64..e0c1e0550 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -110,6 +110,15 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVec UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1v, pVect2v, dimension)); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pVect2, + size_t dimension) { + return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI<0>(pVect1, pVect2, dimension); +} + // Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit // total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is // non-negative, so no individual lane can exceed the chunk total either. That is the whole @@ -124,7 +133,11 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const vo const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - const size_t first = residual + (chunk - residual) / 64 * 64; + // Runtime min rather than the constant alone: with a compile-time trip count GCC split this + // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice + // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. + constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; + const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, first); pVect1 += first; pVect2 += first; @@ -132,7 +145,7 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const vo while (remaining) { const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_L2SqrImp_AVX512F_BW_VL_VNNI<0>(pVect1, pVect2, step); + total += UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; diff --git a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h index 5cd79f8bf..c9e9aef68 100644 --- a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h @@ -138,6 +138,14 @@ float UINT8_L2SqrSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, s return static_cast(UINT8_L2SqrImp_NEON_DOTPROD(pVect1v, pVect2v, dimension)); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_L2SqrFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_NEON_DOTPROD<0>(pVect1, pVect2, dimension); +} + // Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit // total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is // non-negative, so no accumulator lane can exceed the chunk total either. That is the whole @@ -152,7 +160,11 @@ float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pV const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - const size_t first = residual + (chunk - residual) / 64 * 64; + // Runtime min rather than the constant alone: with a compile-time trip count GCC split this + // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice + // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. + constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; + const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, first); pVect1 += first; pVect2 += first; @@ -160,7 +172,7 @@ float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pV while (remaining) { const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_L2SqrImp_NEON_DOTPROD<0>(pVect1, pVect2, step); + total += UINT8_L2SqrFullChunk_NEON_DOTPROD(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index 76601da69..89ca27dd0 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -143,6 +143,14 @@ float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t di return static_cast(UINT8_L2SqrImp_NEON(pVect1v, pVect2v, dimension)); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_L2SqrFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_NEON<0>(pVect1, pVect2, dimension); +} + // Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit // total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is // non-negative, so no accumulator lane can exceed the chunk total either. That is the whole @@ -156,7 +164,11 @@ float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, s const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - const size_t first = residual + (chunk - residual) / 64 * 64; + // Runtime min rather than the constant alone: with a compile-time trip count GCC split this + // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice + // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. + constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; + const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_L2SqrImp_NEON(pVect1, pVect2, first); pVect1 += first; pVect2 += first; @@ -164,7 +176,7 @@ float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, s while (remaining) { const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_L2SqrImp_NEON<0>(pVect1, pVect2, step); + total += UINT8_L2SqrFullChunk_NEON(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; diff --git a/src/VecSim/spaces/L2/L2_SVE_UINT8.h b/src/VecSim/spaces/L2/L2_SVE_UINT8.h index 221348355..7ca7efe3c 100644 --- a/src/VecSim/spaces/L2/L2_SVE_UINT8.h +++ b/src/VecSim/spaces/L2/L2_SVE_UINT8.h @@ -104,6 +104,14 @@ float UINT8_L2SqrSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimen UINT8_L2SqrImp_SVE(pVect1v, pVect2v, dimension)); } +// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into +// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and +// keeping it out of line leaves the first chunk's register allocation alone. +__attribute__((noinline)) static uint32_t +UINT8_L2SqrFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_SVE(pVect1, pVect2, dimension); +} + // Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit // total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is // non-negative, so no accumulator lane can exceed the chunk total either. That is the whole @@ -119,7 +127,10 @@ float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size const size_t chunk_size = 4 * svcntb(); const size_t tail = dimension % chunk_size; const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; - const size_t first = tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; + const size_t first_chunk = + tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; + // Clamped so this wrapper is correct at any dimension, not only past the chunk size. + const size_t first = dimension < first_chunk ? dimension : first_chunk; // first keeps this instantiation's own residual shape: it is congruent to dimension modulo // chunk_size, so partial_chunk and additional_steps still describe its tail. @@ -132,7 +143,7 @@ float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size // shape: no partial vector and no leftover single steps. while (remaining) { const size_t step = remaining < max_step ? remaining : max_step; - total += UINT8_L2SqrImp_SVE(pVect1, pVect2, step); + total += UINT8_L2SqrFullChunk_SVE(pVect1, pVect2, step); pVect1 += step; pVect2 += step; remaining -= step; From 362ed00fba0d40ed9d5c4897d67ac812721c0164 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 08:48:04 +0300 Subject: [PATCH 07/19] style(uint8): spell out the inner product's integer subtract and conversion The epilogue read as `return 1 - static_cast(...)` from a function declared to return float, which relies on an implicit narrowing conversion to carry the intent and reads like a type error. Same arithmetic, written out: hold the total in a signed local, subtract in integer, convert once explicitly. GCC folds the two forms to the same code, so this is readability only. The rationale now appears once per header on the plain wrapper rather than on every wrapper. Co-Authored-By: Claude Opus 5 --- src/VecSim/spaces/IP/IP.cpp | 3 ++- src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 11 ++++++++--- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 11 ++++++++--- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 11 ++++++++--- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 12 ++++++++---- 5 files changed, 34 insertions(+), 14 deletions(-) diff --git a/src/VecSim/spaces/IP/IP.cpp b/src/VecSim/spaces/IP/IP.cpp index 1b2bc7523..83be88044 100644 --- a/src/VecSim/spaces/IP/IP.cpp +++ b/src/VecSim/spaces/IP/IP.cpp @@ -280,7 +280,8 @@ float UINT8_InnerProduct(const void *pVect1v, const void *pVect2v, size_t dimens // Integer subtract then a single conversion, matching INT8_InnerProduct above. The cast to a // signed type is required because ret_t is unsigned for uint8: 1 - an unsigned total would // wrap. The SIMD kernels do the same, so their results stay bit-identical to this one. - return 1 - static_cast(INTEGER_InnerProductImp(pVect1, pVect2, dimension)); + const auto ip = static_cast(INTEGER_InnerProductImp(pVect1, pVect2, dimension)); + return static_cast(1 - ip); } float UINT8_Cosine(const void *pVect1v, const void *pVect2v, size_t dimension) { diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index 18c91ac78..9aa434a36 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -103,7 +103,11 @@ template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + // Subtract in integer and convert once: one rounding rather than two, and a signed cast + // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. + const auto ip = + static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template // 0..63 float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, @@ -160,8 +164,9 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast( - UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const auto ip = + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template // 0..63 diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index c09239878..fe90297a4 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -111,7 +111,11 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + // Subtract in integer and convert once: one rounding rather than two, and a signed cast + // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. + const auto ip = + static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template // 0..63 @@ -163,8 +167,9 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast( - UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const auto ip = + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template // 0..63 diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index 8ea2acda9..a654b4dcf 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -126,7 +126,11 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension template // 0..15 float UINT8_InnerProductSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + // Subtract in integer and convert once: one rounding rather than two, and a signed cast + // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. + const auto ip = + static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template // 0..63 @@ -182,8 +186,9 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template // 0..63 float UINT8_InnerProductSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast( - UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const auto ip = + static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template // 0..63 diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index a9a23e103..6fad3dab2 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -97,8 +97,11 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension template float UINT8_InnerProductSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast(UINT8_InnerProductImp( - pVect1v, pVect2v, dimension)); + // Subtract in integer and convert once: one rounding rather than two, and a signed cast + // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. + const auto ip = static_cast( + UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template @@ -161,8 +164,9 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v template float UINT8_InnerProductSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return 1 - static_cast(UINT8_InnerProductChunkedImp( - pVect1v, pVect2v, dimension)); + const auto ip = static_cast( + UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + return static_cast(1 - ip); } template From 15d0ca4c1dbae7f90287bf603fef641926d687de Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 08:56:08 +0300 Subject: [PATCH 08/19] docs(uint8): collect the kernel notes into one block per file The rationale had spread to a block above almost every definition, and some of it was repeated across headers and across the choosers. Collected into a single block at the top of each kernel header, with spaces.h left as the one place that carries the chunk-size argument, and the three-line reduce notes cut to one line. The chooser note went from three copies per file to one. Comment lines across the eight uint8 kernel headers: 249 to 161. Codegen is byte-identical. Co-Authored-By: Claude Opus 5 --- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 32 ++++++-------- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 29 ++++++------- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 42 ++++++------------- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 37 ++++++---------- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 31 +++++--------- src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h | 31 +++++--------- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 32 +++++--------- src/VecSim/spaces/L2/L2_SVE_UINT8.h | 34 +++++---------- src/VecSim/spaces/functions/NEON.cpp | 4 -- src/VecSim/spaces/functions/NEON_DOTPROD.cpp | 4 -- src/VecSim/spaces/functions/SVE.cpp | 4 -- src/VecSim/spaces/functions/SVE2.cpp | 4 -- 12 files changed, 94 insertions(+), 190 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index 9aa434a36..826353373 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -10,6 +10,17 @@ #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +// uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser +// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h +// carries the chunk-size argument. +// +// Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has +// several callers. +// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against +// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks +// share one out-of-line copy of the kernel. +// The inner product subtracts in integer and converts once, signed because the total is not. + static inline void InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { __m512i va = _mm512_loadu_epi8(pVect1); // AVX512BW pVect1 += 64; @@ -91,11 +102,7 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension } while (pVect1 < pEnd1); } - // Unsigned reduce. The lanes are in range individually, but their total reaches 255*255*dim, - // which passes INT_MAX from dimension 33,027, so reading the result as a signed int wrapped it. - // The intrinsic's adds are vector operations, so the bit pattern is already correct modulo - // 2^32 and this cast simply reads it as unsigned. Exact for up to - // spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. return static_cast(_mm512_reduce_add_epi32(sum)); } @@ -103,8 +110,6 @@ template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { - // Subtract in integer and convert once: one rounding rather than two, and a signed cast - // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. const auto ip = static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); return static_cast(1 - ip); @@ -118,22 +123,12 @@ float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVe return 1.0f - ip / (norm_v1 * norm_v2); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); } -// Chunked variants, selected by the chooser for dimensions past spaces::UINT8_CHUNK_ELEMENTS. Each -// chunk's 32-bit total is exact, and the chunks are folded in 64 bits, so these are exact at any -// dimension. The plain wrappers above are left untouched so their inlining is unaffected, and the -// choice is made once per index rather than per call. -// -// The first chunk absorbs the residual, which leaves the remaining length a whole multiple of 64, -// so every later chunk satisfies the residual-0 kernel's precondition. template // 0..63 static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -141,9 +136,6 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Runtime min rather than the constant alone: with a compile-time trip count GCC split this - // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice - // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index fe90297a4..6e64e93b5 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -11,6 +11,18 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +// uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser +// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h +// carries the chunk-size argument. +// +// Imp is static because IP_NEON_UINT8.h defines the same name with a different body and +// aarch64 gcc 12.3 outlines it, so shared linkage lets a NEON call site execute udot and fault +// where asimddp is absent. always_inline keeps the plain wrapper's codegen unchanged. +// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against +// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks +// share one out-of-line copy of the kernel. +// The inner product subtracts in integer and converts once, signed because the total is not. + __attribute__((always_inline)) static inline void InnerProductOp(uint8x16_t &v1, uint8x16_t &v2, uint32x4_t &sum) { sum = vdotq_u32(sum, v1, v2); @@ -27,9 +39,6 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { pVect2 += 16; } -// Returns the raw integer total, and is static and always_inline; see the NEON header for why each -// of those three matters. The internal linkage is what keeps this body and the NEON one apart -// despite the shared name. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -102,17 +111,13 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension uint32x4_t total_sum = vaddq_u32(sum0, sum1); - // ADDV, unsigned. The total reaches 255*255*dim, so the previous int32_t receiving this - // wrapped negative from dimension 33,027. Exact for up to spaces::UINT8_CHUNK_ELEMENTS - // elements, which is what the caller guarantees. + // ADDV, unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. return vaddvq_u32(total_sum); } template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { - // Subtract in integer and convert once: one rounding rather than two, and a signed cast - // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. const auto ip = static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); return static_cast(1 - ip); @@ -126,17 +131,12 @@ float UINT8_CosineSIMD_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, si return 1.0f - ip / (norm_v1 * norm_v2); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_InnerProductFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); } -// See the NEON header for why each chunk's 32-bit total is exact and why the first chunk absorbs -// the residual. template // 0..63 static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -144,9 +144,6 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Runtime min rather than the constant alone: with a compile-time trip count GCC split this - // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice - // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index a654b4dcf..274c87059 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -11,6 +11,18 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +// uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser +// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h +// carries the chunk-size argument. +// +// Imp is static because IP_NEON_DOTPROD_UINT8.h defines the same name with a different body and +// aarch64 gcc 12.3 outlines it, so shared linkage lets a NEON call site execute udot and fault +// where asimddp is absent. always_inline keeps the plain wrapper's codegen unchanged. +// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against +// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks +// share one out-of-line copy of the kernel. +// The inner product subtracts in integer and converts once, signed because the total is not. + __attribute__((always_inline)) static inline void InnerProductOp(uint8x16_t &v1, uint8x16_t &v2, uint32x4_t &sum) { // Multiply and accumulate low 8 elements (first half) @@ -35,17 +47,6 @@ InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum) { pVect2 += 16; } -// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing -// the float results per chunk would round each one. -// -// static and always_inline are both load-bearing here, not hygiene. The NEON and NEON_DOTPROD -// headers define this same name with the same signature and different bodies, and both are compiled -// into an ARM build. Whether that collides depends on whether the compiler outlines: aarch64 -// gcc 12.3 at -O2 does, emitting one weak COMDAT symbol per residual from both objects, and the -// linker keeps a single body for both. Measured on main with that compiler, the plain NEON inner -// product and cosine wrappers branched into the DOTPROD body and executed udot, which faults on a -// core that has asimd but not asimddp. x86-64 gcc 13/14 and aarch64 clang 18 inline it and do not -// collide, which is precisely why this cannot be left to the toolchain. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -118,16 +119,12 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension uint32x4_t total_sum = vaddq_u32(sum0, sum1); - // ADDV, unsigned. The total reaches 255*255*dim, so the previous int32_t receiving this - // wrapped negative from dimension 33,027. Exact for up to spaces::UINT8_CHUNK_ELEMENTS - // elements, which is what the caller guarantees. + // ADDV, unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. return vaddvq_u32(total_sum); } template // 0..15 float UINT8_InnerProductSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { - // Subtract in integer and convert once: one rounding rather than two, and a signed cast - // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. const auto ip = static_cast(UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); return static_cast(1 - ip); @@ -141,21 +138,11 @@ float UINT8_CosineSIMD_NEON(const void *pVect1v, const void *pVect2v, size_t dim return 1.0f - ip / (norm_v1 * norm_v2); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_InnerProductFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); } -// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit -// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is -// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole -// correctness argument; no reasoning about how work spreads across lanes is needed. -// -// The first chunk absorbs the residual, so every later chunk is a whole multiple of 64 and matches -// the residual-0 kernel's precondition. template // 0..63 static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -163,9 +150,6 @@ static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const v const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Runtime min rather than the constant alone: with a compile-time trip count GCC split this - // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice - // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index 6fad3dab2..42a610c77 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -11,6 +11,17 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +// uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser +// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h +// carries the chunk-size argument. +// +// Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has +// several callers. +// The chunked wrapper's first chunk keeps this instantiation's residual shape, clamped to the +// dimension; the vector length is a runtime value so that split is computed rather than folded. +// Later chunks share one out-of-line copy of the kernel. +// The inner product subtracts in integer and converts once, signed because the total is not. + inline void InnerProductStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t &offset, svuint32_t &sum, const size_t chunk) { svbool_t pg = svptrue_b8(); @@ -24,11 +35,6 @@ inline void InnerProductStep(const uint8_t *&pVect1, const uint8_t *&pVect2, siz offset += chunk; // Move to the next set of uint8 elements } -// Split so the chunked wrapper below can fold each chunk's total in 64 bits; summing the float -// results per chunk would round each one. always_inline because the chunked wrapper calls this -// twice, and GCC outlines a template once it has several callers, which also costs the plain -// wrapper its inlining. static keeps each translation unit's copy to itself: SVE.cpp and SVE2.cpp -// both include this header, and other headers define the same name with different bodies. template __attribute__((always_inline)) static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -89,16 +95,12 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension sum0 = svadd_u32_x(svptrue_b32(), sum0, sum1); sum2 = svadd_u32_x(svptrue_b32(), sum2, sum3); - // svaddv_u32 reduces into a 64-bit scalar; the previous int32_t truncated it, which wrapped - // negative from dimension 33,027. Narrowed to uint32_t, which is exact for up to - // spaces::UINT8_CHUNK_ELEMENTS elements, and that is what the caller guarantees. + // Exact for up to spaces::UINT8_CHUNK_ELEMENTS elements; narrowed from svaddv_u32. return static_cast(svaddv_u32(svptrue_b32(), svadd_u32_x(svptrue_b32(), sum0, sum2))); } template float UINT8_InnerProductSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { - // Subtract in integer and convert once: one rounding rather than two, and a signed cast - // because the total is unsigned, so 1 - total would wrap. Same form as INT8_InnerProduct. const auto ip = static_cast( UINT8_InnerProductImp(pVect1v, pVect2v, dimension)); return static_cast(1 - ip); @@ -113,44 +115,29 @@ float UINT8_CosineSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dime return 1.0f - ip / (norm_v1 * norm_v2); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_InnerProductFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp(pVect1, pVect2, dimension); } -// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit -// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is -// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole -// correctness argument; no reasoning about how work spreads across lanes is needed. template static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto *pVect1 = static_cast(pVect1v); const auto *pVect2 = static_cast(pVect2v); - // The SVE vector length is a runtime value, so unlike the fixed-width kernels the split is - // computed here rather than at compile time. chunk_size matches the kernel's 4-accumulator main - // loop, and tail is the part the template parameters describe. const size_t chunk_size = 4 * svcntb(); const size_t tail = dimension % chunk_size; const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; const size_t first_chunk = tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; - // Clamped so this wrapper is correct at any dimension, not only past the chunk size. const size_t first = dimension < first_chunk ? dimension : first_chunk; - // first keeps this instantiation's own residual shape: it is congruent to dimension modulo - // chunk_size, so partial_chunk and additional_steps still describe its tail. uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); pVect1 += first; pVect2 += first; size_t remaining = dimension - first; - // remaining is a whole multiple of chunk_size, and so is every step, which is the - // shape: no partial vector and no leftover single steps. while (remaining) { const size_t step = remaining < max_step ? remaining : max_step; total += UINT8_InnerProductFullChunk_SVE(pVect1, pVect2, step); diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index e0c1e0550..6958757ad 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -9,6 +9,16 @@ #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks +// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries +// the chunk-size argument. +// +// Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has +// several callers. +// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against +// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks +// share one out-of-line copy of the kernel. + static inline void L2SqrStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { __m512i va = _mm512_loadu_epi8(pVect1); // AVX512BW pVect1 += 64; @@ -32,10 +42,6 @@ static inline void L2SqrStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { // with the corresponding 32-bit integer in src, and store the packed 32-bit results in dst. } -// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing -// the float results per chunk would round each one. always_inline, not merely inline: the chunked -// wrapper calls this twice, and without the attribute GCC outlines it once it has several callers, -// which costs the plain wrapper its inlining too. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -97,9 +103,7 @@ UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size } while (pVect1 < pEnd1); } - // Unsigned. The lanes hold sums of squared byte differences, so the total is unsigned and - // reaches 255*255*dim; reading it as a signed int wrapped it negative from dimension 33,026. - // Exact for up to spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. return static_cast(_mm512_reduce_add_epi32(sum)); } @@ -110,22 +114,12 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVec UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1v, pVect2v, dimension)); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI<0>(pVect1, pVect2, dimension); } -// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit -// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is -// non-negative, so no individual lane can exceed the chunk total either. That is the whole -// correctness argument; no lane-distribution reasoning is needed. -// -// The first chunk absorbs the residual, leaving the remaining length a whole multiple of 64, so -// every later chunk satisfies the residual-0 kernel's precondition. template // 0..63 float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -133,9 +127,6 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const vo const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Runtime min rather than the constant alone: with a compile-time trip count GCC split this - // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice - // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, first); diff --git a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h index c9e9aef68..db1365690 100644 --- a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h @@ -10,6 +10,16 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks +// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries +// the chunk-size argument. +// +// Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has +// several callers. +// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against +// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks +// share one out-of-line copy of the kernel. + __attribute__((always_inline)) static inline void L2SquareOp(const uint8x16_t &v1, const uint8x16_t &v2, uint32x4_t &sum) { // Explicitly reinterpret the int8 vectors as uint8 for vabdq_u8 @@ -51,10 +61,6 @@ L2SquareStep32(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum1, uint32x4_t pVect2 += 32; } -// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing -// the float results per chunk would round each one. always_inline because the chunked wrapper calls -// this twice, and GCC outlines a template once it has several callers, which also costs the plain -// wrapper its inlining. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_L2SqrImp_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -127,9 +133,7 @@ UINT8_L2SqrImp_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dim total_sum = vaddq_u32(total_sum, sum2); total_sum = vaddq_u32(total_sum, sum3); - // Horizontal sum of the 4 elements in the combined sum register. - // Unsigned: the total is a sum of squared byte differences, reaching 255*255*dim. Exact for up - // to spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. return vaddvq_u32(total_sum); } @@ -138,21 +142,11 @@ float UINT8_L2SqrSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, s return static_cast(UINT8_L2SqrImp_NEON_DOTPROD(pVect1v, pVect2v, dimension)); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_L2SqrFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_NEON_DOTPROD<0>(pVect1, pVect2, dimension); } -// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit -// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is -// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole -// correctness argument; no reasoning about how work spreads across lanes is needed. -// -// The first chunk absorbs the residual, so every later chunk is a whole multiple of 64 and matches -// the residual-0 kernel's precondition. template // 0..63 float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -160,9 +154,6 @@ float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pV const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Runtime min rather than the constant alone: with a compile-time trip count GCC split this - // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice - // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, first); diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index 89ca27dd0..20967f15e 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -10,6 +10,16 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks +// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries +// the chunk-size argument. +// +// Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has +// several callers. +// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against +// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks +// share one out-of-line copy of the kernel. + __attribute__((always_inline)) static inline void L2SquareOp(const uint8x16_t &v1, const uint8x16_t &v2, uint32x4_t &sum) { // Compute absolute differences and widen to 16-bit in one step @@ -53,10 +63,6 @@ L2SquareStep32(uint8_t *&pVect1, uint8_t *&pVect2, uint32x4_t &sum1, uint32x4_t pVect2 += 32; } -// Returns the raw integer total so the chunked wrapper below can fold chunks in 64 bits; summing -// the float results per chunk would round each one. always_inline because the chunked wrapper calls -// this twice, and GCC outlines a template once it has several callers, which also costs the plain -// wrapper its inlining. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_L2SqrImp_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -131,10 +137,7 @@ UINT8_L2SqrImp_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) total_sum = vaddq_u32(total_sum, sum2); total_sum = vaddq_u32(total_sum, sum3); - // Horizontal sum of the 4 elements in the combined sum register. - // Unsigned: the total is a sum of squared byte differences, reaching 255*255*dim. As a signed - // int32 this wrapped negative from dimension 33,026. Exact for up to - // spaces::UINT8_CHUNK_ELEMENTS elements, which is what the caller guarantees. + // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. return vaddvq_u32(total_sum); } @@ -143,30 +146,17 @@ float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t di return static_cast(UINT8_L2SqrImp_NEON(pVect1v, pVect2v, dimension)); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_L2SqrFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_NEON<0>(pVect1, pVect2, dimension); } -// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit -// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is -// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole -// correctness argument; no reasoning about how work spreads across lanes is needed. -// -// The first chunk absorbs the residual, so every later chunk is a whole multiple of 64 and matches -// the residual-0 kernel's precondition. template // 0..63 float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto *pVect1 = static_cast(pVect1v); const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Runtime min rather than the constant alone: with a compile-time trip count GCC split this - // loop's accumulator and copied it in and out every 64 elements, measured at 8-9.5% on Ice - // Lake. The min also makes this wrapper correct at any dimension, not only past the chunk size. constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; const size_t first = dimension < first_chunk ? dimension : first_chunk; uint64_t total = UINT8_L2SqrImp_NEON(pVect1, pVect2, first); diff --git a/src/VecSim/spaces/L2/L2_SVE_UINT8.h b/src/VecSim/spaces/L2/L2_SVE_UINT8.h index 7ca7efe3c..9f968d7c8 100644 --- a/src/VecSim/spaces/L2/L2_SVE_UINT8.h +++ b/src/VecSim/spaces/L2/L2_SVE_UINT8.h @@ -10,6 +10,16 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks +// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries +// the chunk-size argument. +// +// Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has +// several callers. +// The chunked wrapper's first chunk keeps this instantiation's residual shape, clamped to the +// dimension; the vector length is a runtime value so that split is computed rather than folded. +// Later chunks share one out-of-line copy of the kernel. + // Aligned step using svptrue_b8() inline void L2SquareStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t &offset, svuint32_t &sum, const size_t chunk) { @@ -27,11 +37,6 @@ inline void L2SquareStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t offset += chunk; // Move to the next set of uint8 elements } -// Split so the chunked wrapper below can fold each chunk's total in 64 bits; summing the float -// results per chunk would round each one. always_inline because the chunked wrapper calls this -// twice, and GCC outlines a template once it has several callers, which also costs the plain -// wrapper its inlining. static keeps each translation unit's copy to itself: SVE.cpp and SVE2.cpp -// both include this header, and other headers define the same name with different bodies. template __attribute__((always_inline)) static inline uint32_t UINT8_L2SqrImp_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -92,9 +97,7 @@ UINT8_L2SqrImp_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { sum0 = svadd_u32_x(all, sum0, sum1); sum2 = svadd_u32_x(all, sum2, sum3); svuint32_t sum_all = svadd_u32_x(all, sum0, sum2); - // svaddv_u32 reduces into a 64-bit scalar. The total is a sum of squared byte differences, - // reaching 255*255*dim, and is exact for up to spaces::UINT8_CHUNK_ELEMENTS elements, which is - // what the caller guarantees. + // Exact for up to spaces::UINT8_CHUNK_ELEMENTS elements; narrowed from svaddv_u32. return static_cast(svaddv_u32(svptrue_b32(), sum_all)); } @@ -104,43 +107,28 @@ float UINT8_L2SqrSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimen UINT8_L2SqrImp_SVE(pVect1v, pVect2v, dimension)); } -// One out-of-line copy of the residual-0 kernel, called once per whole chunk. Inlining it into -// every chunked wrapper cost text for nothing: one call per 65,536 elements is unmeasurable, and -// keeping it out of line leaves the first chunk's register allocation alone. __attribute__((noinline)) static uint32_t UINT8_L2SqrFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_SVE(pVect1, pVect2, dimension); } -// Chunked variant, selected by the chooser past spaces::UINT8_CHUNK_ELEMENTS. Each chunk's 32-bit -// total is exact because 65025 * 65536 = 4,261,478,400 <= UINT32_MAX, and every contribution is -// non-negative, so no accumulator lane can exceed the chunk total either. That is the whole -// correctness argument; no reasoning about how work spreads across lanes is needed. template float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto *pVect1 = static_cast(pVect1v); const auto *pVect2 = static_cast(pVect2v); - // The SVE vector length is a runtime value, so unlike the fixed-width kernels the split is - // computed here rather than at compile time. chunk_size matches the kernel's 4-accumulator main - // loop, and tail is the part the template parameters describe. const size_t chunk_size = 4 * svcntb(); const size_t tail = dimension % chunk_size; const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; const size_t first_chunk = tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; - // Clamped so this wrapper is correct at any dimension, not only past the chunk size. const size_t first = dimension < first_chunk ? dimension : first_chunk; - // first keeps this instantiation's own residual shape: it is congruent to dimension modulo - // chunk_size, so partial_chunk and additional_steps still describe its tail. uint64_t total = UINT8_L2SqrImp_SVE(pVect1, pVect2, first); pVect1 += first; pVect2 += first; size_t remaining = dimension - first; - // remaining is a whole multiple of chunk_size, and so is every step, which is the - // shape: no partial vector and no leftover single steps. while (remaining) { const size_t step = remaining < max_step ? remaining : max_step; total += UINT8_L2SqrFullChunk_SVE(pVect1, pVect2, step); diff --git a/src/VecSim/spaces/functions/NEON.cpp b/src/VecSim/spaces/functions/NEON.cpp index d33dc0f6e..d50bc28d5 100644 --- a/src/VecSim/spaces/functions/NEON.cpp +++ b/src/VecSim/spaces/functions/NEON.cpp @@ -60,8 +60,6 @@ dist_func_t Choose_INT8_Cosine_implementation_NEON(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { @@ -83,8 +81,6 @@ dist_func_t Choose_INT8_L2_implementation_NEON(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { diff --git a/src/VecSim/spaces/functions/NEON_DOTPROD.cpp b/src/VecSim/spaces/functions/NEON_DOTPROD.cpp index 54fd74ad4..bcf5b8d59 100644 --- a/src/VecSim/spaces/functions/NEON_DOTPROD.cpp +++ b/src/VecSim/spaces/functions/NEON_DOTPROD.cpp @@ -43,8 +43,6 @@ dist_func_t Choose_INT8_Cosine_implementation_NEON_DOTPROD(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { @@ -61,8 +59,6 @@ dist_func_t Choose_INT8_L2_implementation_NEON_DOTPROD(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { diff --git a/src/VecSim/spaces/functions/SVE.cpp b/src/VecSim/spaces/functions/SVE.cpp index c473a7ed3..5f8c1b625 100644 --- a/src/VecSim/spaces/functions/SVE.cpp +++ b/src/VecSim/spaces/functions/SVE.cpp @@ -98,8 +98,6 @@ dist_func_t Choose_UINT8_L2_implementation_SVE(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { @@ -110,8 +108,6 @@ dist_func_t Choose_UINT8_IP_implementation_SVE(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { diff --git a/src/VecSim/spaces/functions/SVE2.cpp b/src/VecSim/spaces/functions/SVE2.cpp index 0f59bb721..7c1a662ab 100644 --- a/src/VecSim/spaces/functions/SVE2.cpp +++ b/src/VecSim/spaces/functions/SVE2.cpp @@ -94,8 +94,6 @@ dist_func_t Choose_UINT8_L2_implementation_SVE2(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { @@ -106,8 +104,6 @@ dist_func_t Choose_UINT8_IP_implementation_SVE2(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_Cosine_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; if (dim > spaces::UINT8_CHUNK_ELEMENTS) { From a133b649cbc223163e60766850ea92c7c724b30d Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 09:29:41 +0300 Subject: [PATCH 09/19] Add shared uint8 chunked-accumulation driver Extracts the chunked-accumulation formula duplicated across the eight uint8 SIMD kernel headers into a single templated driver, so each kernel's 32-bit per-chunk total stays exact regardless of dimension. --- src/VecSim/spaces/uint8_chunking.h | 64 ++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 src/VecSim/spaces/uint8_chunking.h diff --git a/src/VecSim/spaces/uint8_chunking.h b/src/VecSim/spaces/uint8_chunking.h new file mode 100644 index 000000000..afc7e829b --- /dev/null +++ b/src/VecSim/spaces/uint8_chunking.h @@ -0,0 +1,64 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#pragma once + +// Shared chunked-accumulation driver for the uint8 SIMD kernels. It splits a distance +// computation into chunks of at most UINT8_CHUNK_ELEMENTS elements so each chunk's 32-bit +// SIMD total stays exact, and folds the per-chunk totals into a 64-bit scalar. The same +// formula serves both fixed-width kernels (granule 64) and SVE (granule 4 * svcntb()); only +// the granule differs. The caller supplies a Kernel adapter with: +// static size_t granule() - the kernel's block size +// static uint32_t first(const uint8_t *, const uint8_t *, size_t) - the residual-bearing +// kernel, shape already +// bound +// static uint32_t rest(const uint8_t *, const uint8_t *, size_t) - the out-of-line +// residual-0 kernel +// Invariants: first <= dimension always, so this is correct at any dimension, including ones +// below the chunk size (the loop then does not execute). first is congruent to dimension modulo +// granule, so Kernel::first's residual shape still describes it. remaining is therefore a whole +// multiple of granule, and so is every step, which is Kernel::rest's precondition. No single call +// ever gets more than UINT8_CHUNK_ELEMENTS elements, which is what keeps each chunk's 32-bit +// total exact. + +#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS + +#include +#include + +namespace spaces { + +template +static inline uint64_t uint8_chunked_total(const void *pVect1v, const void *pVect2v, + size_t dimension) { + const auto *pVect1 = static_cast(pVect1v); + const auto *pVect2 = static_cast(pVect2v); + + constexpr size_t chunk = UINT8_CHUNK_ELEMENTS; + const size_t granule = Kernel::granule(); + const size_t tail = dimension % granule; + const size_t first_chunk = tail + ((chunk - tail) / granule) * granule; + const size_t first = dimension < first_chunk ? dimension : first_chunk; + const size_t max_step = (chunk / granule) * granule; + + uint64_t total = Kernel::first(pVect1, pVect2, first); + pVect1 += first; + pVect2 += first; + size_t remaining = dimension - first; + + while (remaining) { + const size_t step = remaining < max_step ? remaining : max_step; + total += Kernel::rest(pVect1, pVect2, step); + pVect1 += step; + pVect2 += step; + remaining -= step; + } + return total; +} + +} // namespace spaces From c471891a2743d44a2945b1cd46a1d44ad38c7002 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 09:35:00 +0300 Subject: [PATCH 10/19] Migrate SVE uint8 IP/L2 kernels to the shared chunking driver Replace the hand-written chunked-accumulation logic in IP_SVE_UINT8.h and L2_SVE_UINT8.h with adapters (UINT8_IPChunkKernel_SVE, UINT8_L2ChunkKernel_SVE) over spaces::uint8_chunked_total, using granule() = 4 * svcntb() for SVE's runtime block size. --- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 41 ++++++++++------------------- src/VecSim/spaces/L2/L2_SVE_UINT8.h | 39 +++++++++++---------------- 2 files changed, 30 insertions(+), 50 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index 42a610c77..7dd963903 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -9,6 +9,7 @@ #pragma once #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" #include // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser @@ -121,45 +122,31 @@ UINT8_InnerProductFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, si } template -static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - const size_t chunk_size = 4 * svcntb(); - const size_t tail = dimension % chunk_size; - const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; - const size_t first_chunk = - tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - - uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < max_step ? remaining : max_step; - total += UINT8_InnerProductFullChunk_SVE(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; +struct UINT8_IPChunkKernel_SVE { + static size_t granule() { return 4 * svcntb(); } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductImp(pVect1, pVect2, dimension); } - return total; -} + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductFullChunk_SVE(pVect1, pVect2, dimension); + } +}; template float UINT8_InnerProductSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto ip = static_cast( - UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); return static_cast(1 - ip); } template float UINT8_CosineSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { float ip = static_cast( - UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/L2/L2_SVE_UINT8.h b/src/VecSim/spaces/L2/L2_SVE_UINT8.h index 9f968d7c8..8f05a7253 100644 --- a/src/VecSim/spaces/L2/L2_SVE_UINT8.h +++ b/src/VecSim/spaces/L2/L2_SVE_UINT8.h @@ -8,6 +8,7 @@ */ #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" #include // uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks @@ -113,28 +114,20 @@ UINT8_L2SqrFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t di } template -float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - const size_t chunk_size = 4 * svcntb(); - const size_t tail = dimension % chunk_size; - const size_t max_step = spaces::UINT8_CHUNK_ELEMENTS / chunk_size * chunk_size; - const size_t first_chunk = - tail + (spaces::UINT8_CHUNK_ELEMENTS - tail) / chunk_size * chunk_size; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - - uint64_t total = UINT8_L2SqrImp_SVE(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < max_step ? remaining : max_step; - total += UINT8_L2SqrFullChunk_SVE(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; +struct UINT8_L2ChunkKernel_SVE { + static size_t granule() { return 4 * svcntb(); } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_SVE(pVect1, pVect2, dimension); + } + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrFullChunk_SVE(pVect1, pVect2, dimension); } - return static_cast(total); +}; + +template +float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { + return static_cast( + spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); } From 1097aaaea1d8ebc69f306813ee38d88610c5ed64 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 09:35:37 +0300 Subject: [PATCH 11/19] Migrate IP kernels to shared uint8_chunking driver Replace local UINT8_InnerProductChunkedImp function templates in three inner-product kernel headers with adapter structs that call the shared chunked-accumulation driver in uint8_chunking.h. Reduces duplication across AVX512F_BW_VL_VNNI_UINT8, NEON_UINT8, and NEON_DOTPROD_UINT8. Co-Authored-By: Claude Opus 5 --- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 41 ++++++++----------- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 39 +++++++----------- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 38 +++++++---------- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 35 ++++++++-------- src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h | 35 ++++++++-------- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 34 +++++++-------- 6 files changed, 92 insertions(+), 130 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index 826353373..c3778152e 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -9,6 +9,7 @@ #pragma once #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser // picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h @@ -130,42 +131,32 @@ UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint } template // 0..63 -static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; +struct UINT8_IPChunkKernel_AVX512F_BW_VL_VNNI { + static size_t granule() { return 64; } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductImp(pVect1, pVect2, dimension); } - return total; -} + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); + } +}; template // 0..63 float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const auto ip = - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const auto ip = static_cast( + spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); return static_cast(1 - ip); } template // 0..63 float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const float ip = - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const float ip = static_cast( + spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index 6e64e93b5..1e620eb76 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -9,6 +9,7 @@ #pragma once #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" #include // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser @@ -138,34 +139,23 @@ UINT8_InnerProductFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *p } template // 0..63 -static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_InnerProductFullChunk_NEON_DOTPROD(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; +struct UINT8_IPChunkKernel_NEON_DOTPROD { + static size_t granule() { return 64; } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductImp(pVect1, pVect2, dimension); } - return total; -} + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductFullChunk_NEON_DOTPROD(pVect1, pVect2, dimension); + } +}; template // 0..63 float UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const auto ip = - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + const auto ip = static_cast( + spaces::uint8_chunked_total>(pVect1v, pVect2v, + dimension)); return static_cast(1 - ip); } @@ -173,7 +163,8 @@ template // 0..63 float UINT8_CosineSIMD_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { float ip = - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + static_cast(spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index 274c87059..7bb78cc44 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -9,6 +9,7 @@ #pragma once #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" #include // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser @@ -144,41 +145,30 @@ UINT8_InnerProductFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, s } template // 0..63 -static inline uint64_t UINT8_InnerProductChunkedImp(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - uint64_t total = UINT8_InnerProductImp(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_InnerProductFullChunk_NEON(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; +struct UINT8_IPChunkKernel_NEON { + static size_t granule() { return 64; } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductImp(pVect1, pVect2, dimension); } - return total; -} + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_InnerProductFullChunk_NEON(pVect1, pVect2, dimension); + } +}; template // 0..63 float UINT8_InnerProductSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { const auto ip = - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + static_cast(spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); return static_cast(1 - ip); } template // 0..63 float UINT8_CosineSIMD_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = - static_cast(UINT8_InnerProductChunkedImp(pVect1v, pVect2v, dimension)); + float ip = static_cast(spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 6958757ad..178cb75e1 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -8,6 +8,7 @@ */ #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" // uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks // plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries @@ -120,26 +121,22 @@ UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pV return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI<0>(pVect1, pVect2, dimension); } +template // 0..63 +struct UINT8_L2ChunkKernel_AVX512F_BW_VL_VNNI { + static size_t granule() { return 64; } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); + } + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); + } +}; + template // 0..63 float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - uint64_t total = UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; - } - return static_cast(total); + return static_cast( + spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); } diff --git a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h index db1365690..ba662ca7c 100644 --- a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h @@ -8,6 +8,7 @@ */ #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" #include // uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks @@ -147,26 +148,22 @@ UINT8_L2SqrFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, return UINT8_L2SqrImp_NEON_DOTPROD<0>(pVect1, pVect2, dimension); } +template // 0..63 +struct UINT8_L2ChunkKernel_NEON_DOTPROD { + static size_t granule() { return 64; } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, dimension); + } + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrFullChunk_NEON_DOTPROD(pVect1, pVect2, dimension); + } +}; + template // 0..63 float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - uint64_t total = UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_L2SqrFullChunk_NEON_DOTPROD(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; - } - return static_cast(total); + return static_cast( + spaces::uint8_chunked_total>(pVect1v, pVect2v, + dimension)); } diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index 20967f15e..fa407afb0 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -8,6 +8,7 @@ */ #include "VecSim/spaces/space_includes.h" #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include "VecSim/spaces/uint8_chunking.h" #include // uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks @@ -152,24 +153,19 @@ UINT8_L2SqrFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t d } template // 0..63 -float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - constexpr size_t first_chunk = residual + (chunk - residual) / 64 * 64; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - uint64_t total = UINT8_L2SqrImp_NEON(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < chunk ? remaining : chunk; - total += UINT8_L2SqrFullChunk_NEON(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; +struct UINT8_L2ChunkKernel_NEON { + static size_t granule() { return 64; } + __attribute__((always_inline)) static inline uint32_t + first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrImp_NEON(pVect1, pVect2, dimension); + } + static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { + return UINT8_L2SqrFullChunk_NEON(pVect1, pVect2, dimension); } - return static_cast(total); +}; + +template // 0..63 +float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { + return static_cast(spaces::uint8_chunked_total>( + pVect1v, pVect2v, dimension)); } From b10e1cc60c90c9fdcb27c4d8b317628b1d95a369 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 09:39:21 +0300 Subject: [PATCH 12/19] Add architecture-independent test for uint8 chunked-accumulation driver Exercises spaces::uint8_chunked_total directly with a mock kernel so the chunk-tiling invariants (exact tiling, granule-multiple steps, chunk-size cap) are verified on any host, not only through whatever SIMD kernel a given CPU happens to support. Co-Authored-By: Claude Opus 5 --- tests/unit/test_spaces.cpp | 132 +++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 85fafe297..62b76d5c6 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -48,6 +48,7 @@ #include "VecSim/spaces/functions/SVE.h" #include "VecSim/spaces/functions/SVE_BF16.h" #include "VecSim/spaces/functions/SVE2.h" +#include "VecSim/spaces/uint8_chunking.h" #include "tests_utils.h" using bfloat16 = vecsim_types::bfloat16; @@ -4936,6 +4937,137 @@ TEST(SQ8_SQ8_EdgeCases, L2ExtremeValuesTest) { ASSERT_NEAR(result, baseline, 0.01f) << "Extreme values L2 should match baseline"; } +// spaces::uint8_chunked_total (uint8_chunking.h) is the chunked-accumulation driver shared by +// every uint8 SIMD kernel: it tiles a vector into segments no larger than UINT8_CHUNK_ELEMENTS +// so each segment's 32-bit SIMD partial sum stays exact, then folds the per-segment totals into +// a 64-bit scalar. The driver only ever calls Kernel::granule/first/rest, so it can be exercised +// directly with a mock kernel instead of a real SIMD kernel, which means this test needs no +// architecture-specific build flag and runs the same way on every host: an x86 box without +// AVX512 and an ARM box exercise identical logic here, closing the gap where this driver was +// previously only reachable through whichever hardware-specific kernel happened to be present. +// The mock kernel below records the (offset, length) of every call it receives and returns 0; +// the test then checks that the recorded segments exactly tile the vector for a sweep of +// granules (64, 128, 192, 256 and 1024, standing in for fixed-width kernels and SVE vector +// lengths of 32/48/64/256 bytes) and dimensions chosen to cover every residue class modulo the +// granule across one-, two- and three-segment cases, plus the boundary around +// UINT8_CHUNK_ELEMENTS itself. +TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { + struct RecordedCall { + size_t offset; + size_t length; + }; + + // Local mock adapter matching the Kernel contract from uint8_chunking.h. All state lives in + // function-local statics reached through static member functions (a local class cannot have + // static data members), so `reset` must be called before each driver invocation. + struct RecordingKernel { + static size_t granule() { return granule_ref(); } + + static uint32_t first(const uint8_t *v1, const uint8_t *, size_t length) { + record(v1, length); + return 0; + } + + static uint32_t rest(const uint8_t *v1, const uint8_t *, size_t length) { + record(v1, length); + return 0; + } + + static void reset(const uint8_t *base, size_t granule) { + calls_ref().clear(); + base_ref() = base; + granule_ref() = granule; + } + + static const std::vector &calls_seen() { return calls_ref(); } + + private: + static void record(const uint8_t *v1, size_t length) { + calls_ref().push_back({static_cast(v1 - base_ref()), length}); + } + + static std::vector &calls_ref() { + static std::vector calls; + return calls; + } + + static const uint8_t *&base_ref() { + static const uint8_t *base = nullptr; + return base; + } + + static size_t &granule_ref() { + static size_t granule = 0; + return granule; + } + }; + + constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; + // Large enough for the biggest dimension exercised below (first segment plus two full + // max-size segments, bounded by chunk), with slack. Contents are irrelevant: the mock kernel + // never dereferences the pointers, it only computes offsets from them. + constexpr size_t buffer_size = 3 * chunk + 4096; + std::vector v1(buffer_size, 0); + std::vector v2(buffer_size, 0); + + const std::array granules = {64, 128, 192, 256, 1024}; + + for (size_t granule : granules) { + const size_t max_step = (chunk / granule) * granule; + auto first_chunk_for = [&](size_t tail) { + return tail + ((chunk - tail) / granule) * granule; + }; + + std::vector dims; + for (size_t r = 0; r < granule; r++) { + const size_t fc = first_chunk_for(r); + dims.push_back(r == 0 ? granule : r); // one segment: dim <= chunk + dims.push_back(fc + max_step); // two segments + dims.push_back(fc + 2 * max_step); // three segments + } + dims.push_back(chunk - 1); + dims.push_back(chunk); + dims.push_back(chunk + 1); + + for (size_t dimension : dims) { + ASSERT_LE(dimension, buffer_size) + << "granule=" << granule << " dimension=" << dimension; + + RecordingKernel::reset(v1.data(), granule); + const uint64_t total = + spaces::uint8_chunked_total(v1.data(), v2.data(), dimension); + (void)total; + + SCOPED_TRACE("granule=" + std::to_string(granule) + + " dimension=" + std::to_string(dimension)); + const auto &calls = RecordingKernel::calls_seen(); + ASSERT_FALSE(calls.empty()); + + size_t sum = 0; + size_t expected_offset = 0; + for (size_t i = 0; i < calls.size(); i++) { + EXPECT_EQ(calls[i].offset, expected_offset) + << "call " << i << " does not tile contiguously (gap or overlap)"; + EXPECT_LE(calls[i].length, chunk) + << "call " << i << " exceeds UINT8_CHUNK_ELEMENTS"; + if (i > 0) { + EXPECT_EQ(calls[i].length % granule, 0u) + << "call " << i << " length is not a whole multiple of granule"; + } + sum += calls[i].length; + expected_offset += calls[i].length; + } + EXPECT_EQ(sum, dimension) << "recorded lengths do not sum to the dimension"; + EXPECT_EQ(calls[0].length % granule, dimension % granule) + << "first call length is not congruent to dimension modulo granule"; + if (dimension <= chunk) { + EXPECT_EQ(calls.size(), 1u) + << "dimension at or below UINT8_CHUNK_ELEMENTS should need exactly one call"; + } + } + } +} + // Assert the exact alignment-hint values published by the SQ8 distance dispatchers. // The hint refers to the SQ8 (first / storage) operand per the GetDistFunc contract documented // in spaces/spaces.h. These tests guard against silent regressions of the per-kernel hints used From c1b99ee362ae7181bffc1c823c55b1b5888e9021 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 09:41:58 +0300 Subject: [PATCH 13/19] Assert the granule precondition in the uint8 chunking driver Kernel::granule() must be in (0, UINT8_CHUNK_ELEMENTS] for the chunking arithmetic to hold; assert it instead of dividing by zero or silently underflowing chunk - tail. Documents the precondition that the invariants already relied on. --- src/VecSim/spaces/uint8_chunking.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/src/VecSim/spaces/uint8_chunking.h b/src/VecSim/spaces/uint8_chunking.h index afc7e829b..2aecacec3 100644 --- a/src/VecSim/spaces/uint8_chunking.h +++ b/src/VecSim/spaces/uint8_chunking.h @@ -19,15 +19,18 @@ // bound // static uint32_t rest(const uint8_t *, const uint8_t *, size_t) - the out-of-line // residual-0 kernel -// Invariants: first <= dimension always, so this is correct at any dimension, including ones -// below the chunk size (the loop then does not execute). first is congruent to dimension modulo -// granule, so Kernel::first's residual shape still describes it. remaining is therefore a whole -// multiple of granule, and so is every step, which is Kernel::rest's precondition. No single call -// ever gets more than UINT8_CHUNK_ELEMENTS elements, which is what keeps each chunk's 32-bit -// total exact. +// Invariants below hold for any Kernel whose granule() is in (0, UINT8_CHUNK_ELEMENTS]; all +// current adapters return 64 (fixed-width) or 4 * svcntb() (SVE, 64 to 1024 for a 16 to 256 byte +// vector length), so both stay within that range. Given that precondition: first <= dimension +// always, so this is correct at any dimension, including ones below the chunk size (the loop +// then does not execute). first is congruent to dimension modulo granule, so Kernel::first's +// residual shape still describes it. remaining is therefore a whole multiple of granule, and so +// is every step, which is Kernel::rest's precondition. No single call ever gets more than +// UINT8_CHUNK_ELEMENTS elements, which is what keeps each chunk's 32-bit total exact. #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS +#include #include #include @@ -41,6 +44,7 @@ static inline uint64_t uint8_chunked_total(const void *pVect1v, const void *pVec constexpr size_t chunk = UINT8_CHUNK_ELEMENTS; const size_t granule = Kernel::granule(); + assert(granule > 0 && granule <= chunk); const size_t tail = dimension % granule; const size_t first_chunk = tail + ((chunk - tail) / granule) * granule; const size_t first = dimension < first_chunk ? dimension : first_chunk; From e7193124090e444c8add28c03c333bc4ff3c761d Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 09:47:47 +0300 Subject: [PATCH 14/19] Add value-based end-to-end check to the uint8 chunked-driver test The tiling offset check compared a recorded offset to the cumulative sum of prior recorded lengths, both derived from the same driver-advanced pointer, so it could not fail on its own. Keep it (it still guards the coupling between the length passed to the kernel and the pointer advance) and add a real correctness check: the mock kernel now returns the sum of the bytes in the slice it was handed, read from a position-dependent fill, and the driver's total is compared against an independent sum over the whole buffer. That catches skipped, duplicated or mis-sized chunks that the offset check cannot. Co-Authored-By: Claude Opus 5 --- tests/unit/test_spaces.cpp | 61 ++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 12 deletions(-) diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 62b76d5c6..883c83d4a 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -4945,12 +4945,21 @@ TEST(SQ8_SQ8_EdgeCases, L2ExtremeValuesTest) { // architecture-specific build flag and runs the same way on every host: an x86 box without // AVX512 and an ARM box exercise identical logic here, closing the gap where this driver was // previously only reachable through whichever hardware-specific kernel happened to be present. -// The mock kernel below records the (offset, length) of every call it receives and returns 0; -// the test then checks that the recorded segments exactly tile the vector for a sweep of -// granules (64, 128, 192, 256 and 1024, standing in for fixed-width kernels and SVE vector -// lengths of 32/48/64/256 bytes) and dimensions chosen to cover every residue class modulo the -// granule across one-, two- and three-segment cases, plus the boundary around -// UINT8_CHUNK_ELEMENTS itself. +// Real coverage of "did the driver visit every element exactly once, correctly" comes from the +// value check below: the mock's first()/rest() return the sum of the bytes in the slice they +// were handed (read from a position-dependent, non-constant fill), and the total the driver +// returns is compared against an independent, trivially-correct sum over the whole buffer. A +// skipped element, a double-counted element, or a mis-sized chunk changes that sum; it cannot +// cancel out the way it could with a constant fill or a return value of 0. The mock also records +// the (offset, length) of every call; the tiling check on those recordings does not by itself +// prove the driver visited the right elements (offset is derived from the same length the driver +// just advanced its pointer by, so "no gap/overlap" holds by construction), but it does guard the +// coupling between the length passed to the kernel and the distance the pointer is advanced, plus +// the length-shape properties below (granule multiples, chunk-size cap, congruence). Together the +// two checks cover a sweep of granules (64, 128, 192, 256 and 1024, standing in for fixed-width +// kernels and SVE vector lengths of 32/48/64/256 bytes) and dimensions chosen to cover every +// residue class modulo the granule across one-, two- and three-segment cases, plus the boundary +// around UINT8_CHUNK_ELEMENTS itself. TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { struct RecordedCall { size_t offset; @@ -4963,14 +4972,17 @@ TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { struct RecordingKernel { static size_t granule() { return granule_ref(); } + // Returns the sum of the bytes in [v1, v1 + length), not 0: combined with a + // position-dependent fill, this makes the driver's return value an end-to-end proof + // that every element was visited exactly once, not just a coupling check on lengths. static uint32_t first(const uint8_t *v1, const uint8_t *, size_t length) { record(v1, length); - return 0; + return sum_of(v1, length); } static uint32_t rest(const uint8_t *v1, const uint8_t *, size_t length) { record(v1, length); - return 0; + return sum_of(v1, length); } static void reset(const uint8_t *base, size_t granule) { @@ -4982,6 +4994,14 @@ TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { static const std::vector &calls_seen() { return calls_ref(); } private: + static uint32_t sum_of(const uint8_t *v1, size_t length) { + uint32_t sum = 0; + for (size_t i = 0; i < length; i++) { + sum += v1[i]; + } + return sum; + } + static void record(const uint8_t *v1, size_t length) { calls_ref().push_back({static_cast(v1 - base_ref()), length}); } @@ -5004,10 +5024,17 @@ TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; // Large enough for the biggest dimension exercised below (first segment plus two full - // max-size segments, bounded by chunk), with slack. Contents are irrelevant: the mock kernel - // never dereferences the pointers, it only computes offsets from them. + // max-size segments, bounded by chunk), with slack. constexpr size_t buffer_size = 3 * chunk + 4096; - std::vector v1(buffer_size, 0); + // v1 is filled with a position-dependent, non-constant pattern so a skipped, duplicated or + // mis-sized element changes the summed value rather than cancelling out (a constant fill, or + // returning 0 from the mock, would not catch that). Values stay under 251 and dimensions + // stay well under 600,000, so the accumulated uint64_t sum cannot overflow. v2 is unused by + // the mock kernel and left zero-filled. + std::vector v1(buffer_size); + for (size_t i = 0; i < buffer_size; i++) { + v1[i] = static_cast((i * 31 + 7) % 251); + } std::vector v2(buffer_size, 0); const std::array granules = {64, 128, 192, 256, 1024}; @@ -5036,10 +5063,20 @@ TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { RecordingKernel::reset(v1.data(), granule); const uint64_t total = spaces::uint8_chunked_total(v1.data(), v2.data(), dimension); - (void)total; SCOPED_TRACE("granule=" + std::to_string(granule) + " dimension=" + std::to_string(dimension)); + + // Value check: independently sum the same byte range the driver was asked to cover. + // This is what actually proves every element was visited exactly once (a skipped, + // duplicated or mis-sized chunk changes this sum); the tiling check below only + // proves the length passed to the kernel matches how far the pointer advanced. + uint64_t expected = 0; + for (size_t i = 0; i < dimension; i++) { + expected += v1[i]; + } + EXPECT_EQ(total, expected) << "driver total does not match independent byte sum"; + const auto &calls = RecordingKernel::calls_seen(); ASSERT_FALSE(calls.empty()); From 0fafc78cc886e459d6fffaab0b47f67378429e38 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 10:26:14 +0300 Subject: [PATCH 15/19] test(uint8): make a zero-tier run skip, let CI demand its tier, and enforce the granule at compile time Three review points, all of which made a green result mean less than it looked. A run that exercised no SIMD tier now reports skipped rather than passed. On a host without the relevant instruction set the per-tier test executed no chunked kernel at all, and reporting that as a pass reads as coverage that does not exist. Hardware-specific CI can now name the tier it exists to cover by setting VECSIM_REQUIRE_UINT8_TIER to AVX512F_BW_VL_VNNI, SVE2, SVE, NEON_DOTPROD or NEON. The requirement is checked before the skip, so a mislabeled or silently downgraded runner fails instead of quietly skipping. Verified both ways: without the variable the test skips on this host, with it set the test fails and names the tier it could not reach. The granule precondition was guarded only by assert, which disappears under NDEBUG, so it protected nobody in a release build. granule() is now constexpr in the six fixed-width adapters, and the driver static_asserts the bound whenever the adapter can express it as a constant. Verified that granule 0 and granule 70000 both fail to compile under -DNDEBUG while 64 compiles. SVE keeps the runtime assert because its granule depends on the vector length and genuinely cannot be constant. Co-Authored-By: Claude Opus 5 --- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 2 +- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 2 +- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 2 +- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 2 +- src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h | 2 +- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 2 +- src/VecSim/spaces/uint8_chunking.h | 8 ++++++ tests/unit/test_spaces.cpp | 27 +++++++++++++++++-- 8 files changed, 39 insertions(+), 8 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index c3778152e..71123af65 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -132,7 +132,7 @@ UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint template // 0..63 struct UINT8_IPChunkKernel_AVX512F_BW_VL_VNNI { - static size_t granule() { return 64; } + static constexpr size_t granule() { return 64; } __attribute__((always_inline)) static inline uint32_t first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp(pVect1, pVect2, dimension); diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index 1e620eb76..504fd912c 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -140,7 +140,7 @@ UINT8_InnerProductFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *p template // 0..63 struct UINT8_IPChunkKernel_NEON_DOTPROD { - static size_t granule() { return 64; } + static constexpr size_t granule() { return 64; } __attribute__((always_inline)) static inline uint32_t first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp(pVect1, pVect2, dimension); diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index 7bb78cc44..d012e47e1 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -146,7 +146,7 @@ UINT8_InnerProductFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, s template // 0..63 struct UINT8_IPChunkKernel_NEON { - static size_t granule() { return 64; } + static constexpr size_t granule() { return 64; } __attribute__((always_inline)) static inline uint32_t first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_InnerProductImp(pVect1, pVect2, dimension); diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 178cb75e1..6ee1a1777 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -123,7 +123,7 @@ UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pV template // 0..63 struct UINT8_L2ChunkKernel_AVX512F_BW_VL_VNNI { - static size_t granule() { return 64; } + static constexpr size_t granule() { return 64; } __attribute__((always_inline)) static inline uint32_t first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); diff --git a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h index ba662ca7c..a0e5a9ebb 100644 --- a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h @@ -150,7 +150,7 @@ UINT8_L2SqrFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, template // 0..63 struct UINT8_L2ChunkKernel_NEON_DOTPROD { - static size_t granule() { return 64; } + static constexpr size_t granule() { return 64; } __attribute__((always_inline)) static inline uint32_t first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, dimension); diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index fa407afb0..f1e461c18 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -154,7 +154,7 @@ UINT8_L2SqrFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t d template // 0..63 struct UINT8_L2ChunkKernel_NEON { - static size_t granule() { return 64; } + static constexpr size_t granule() { return 64; } __attribute__((always_inline)) static inline uint32_t first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { return UINT8_L2SqrImp_NEON(pVect1, pVect2, dimension); diff --git a/src/VecSim/spaces/uint8_chunking.h b/src/VecSim/spaces/uint8_chunking.h index 2aecacec3..88c1f279d 100644 --- a/src/VecSim/spaces/uint8_chunking.h +++ b/src/VecSim/spaces/uint8_chunking.h @@ -31,6 +31,7 @@ #include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS #include +#include #include #include @@ -43,6 +44,13 @@ static inline uint64_t uint8_chunked_total(const void *pVect1v, const void *pVec const auto *pVect2 = static_cast(pVect2v); constexpr size_t chunk = UINT8_CHUNK_ELEMENTS; + // Enforce the granule precondition at compile time when the adapter can express it, which is + // every fixed-width kernel. A plain assert would vanish under NDEBUG, so it is the fallback + // only for SVE, whose granule depends on the runtime vector length and cannot be constant. + if constexpr (requires { std::integral_constant{}; }) { + static_assert(Kernel::granule() > 0 && Kernel::granule() <= UINT8_CHUNK_ELEMENTS, + "Kernel::granule() must be in (0, UINT8_CHUNK_ELEMENTS]"); + } const size_t granule = Kernel::granule(); assert(granule > 0 && granule <= chunk); const size_t tail = dimension % granule; diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 883c83d4a..09bf601e3 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -14,6 +14,8 @@ #include #include #include +#include +#include #include #include "gtest/gtest.h" @@ -2326,6 +2328,7 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { // multiple. const std::vector dims = {65536, 65600, 65601, 65663, 131072, 200000}; const auto opt = getCpuOptimizationFeatures(); + std::set all_tiers; constexpr size_t max_dim = 200000; std::vector ones(max_dim + sizeof(float), 255); @@ -2398,9 +2401,29 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { covered += covered.empty() ? t : ", " + t; } RecordProperty("tiers_at_dim_" + std::to_string(dim), covered); - std::cout << " dim " << dim << " covered tiers: " - << (covered.empty() ? "" : covered) + std::cout << " dim " << dim << " covered tiers: " << (covered.empty() ? "" : covered) << std::endl; + for (const auto &t : tiers_checked) { + all_tiers.insert(t); + } + } + + // Order matters. A stated requirement must be able to FAIL, so it is checked before the skip: + // hardware-specific CI sets VECSIM_REQUIRE_UINT8_TIER to the tier that job exists to cover + // (AVX512F_BW_VL_VNNI, SVE2, SVE, NEON_DOTPROD or NEON), and a mislabeled or silently + // downgraded runner then fails instead of quietly skipping. + if (const char *required = std::getenv("VECSIM_REQUIRE_UINT8_TIER")) { + EXPECT_TRUE(all_tiers.count(required) > 0) + << "VECSIM_REQUIRE_UINT8_TIER=" << required << " but that tier was not exercised. " + << "This host reached " << all_tiers.size() << " tier(s), so the run proves nothing " + << "about " << required; + return; + } + + // With no requirement stated, a run that exercised no tier proves nothing. Report it skipped + // rather than passed, because passing reads as coverage on a host that has none. + if (all_tiers.empty()) { + GTEST_SKIP() << "no uint8 SIMD tier on this host, no chunked kernel was executed"; } } From 0ab1c67c09ebf8155264eb5342c98b0499cff5b7 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 10:39:34 +0300 Subject: [PATCH 16/19] test(uint8): assert worst-case inputs against an independent 64-bit oracle, and let CI demand its tier Closes the two coverage gaps that this PR's own changes created, rather than deferring them. Independent oracle. Every existing uint8 kernel test used the scalar kernel as its oracle, which is a different code path but not an independent one. This series changed the scalar and SIMD inner product epilogues together so they would stay bit-identical, so a test asserting only scalar == SIMD cannot catch that shared convention being wrong. The new test derives its expectation from the inputs alone in 64-bit integer arithmetic and asserts the scalar kernel and every available SIMD tier against it on the same footing. Demonstrated non-vacuous: mutating the scalar epilogue from 1 - ip to ip - 1 leaves the residual-sweep test passing and fails only the oracle test. Worst-case overflow. The residual sweep used a ramp against all-255, averaging roughly half the maximum accumulator load, and the worst case appeared only at seven sampled dimensions through the dispatched tier. The new test sweeps every residual with all-255 against all-255, which puts 65,025 per element into the inner product accumulator, and all-255 against all-0, which does the same for L2. It does so at two bases: 65024+r, which stays under the chunk size and so drives the plain kernel's 32-bit reduce to about 4.23e9, just under UINT32_MAX; and 131072+r, whose total near 8.5e9 only a 64-bit fold can carry. A ramp pair is kept alongside because constant data lets a gap and an overlap of equal size cancel. The test asserts the worst-case totals actually exceed UINT32_MAX, so it cannot quietly stop exercising the fold. Tier discovery is now shared between this test and the per-tier test, so a tier cannot be covered by one and missed by the other. CI. task-unit-test.yml takes a require-uint8-tier input, exported as VECSIM_REQUIRE_UINT8_TIER. The dedicated ARM job requires SVE2, which r8g Graviton4 has, and the coverage job requires AVX512F_BW_VL_VNNI on both suite runs, which c7i Sapphire Rapids has. Those two jobs now fail rather than skip if the hardware they exist to cover is absent. Generic-CPU jobs leave it unset and skip as before. Also documents why SVE keeping only a runtime assert is acceptable: the architecture bounds an SVE vector to 16 to 256 bytes, so its granule is 64 to 1024, far below the 65,536 limit. Co-Authored-By: Claude Opus 5 --- .github/workflows/arm.yml | 4 + .github/workflows/coverage.yml | 7 + .github/workflows/task-unit-test.yml | 8 + src/VecSim/spaces/uint8_chunking.h | 4 + tests/unit/test_spaces.cpp | 220 ++++++++++++++++++++------- 5 files changed, 189 insertions(+), 54 deletions(-) diff --git a/.github/workflows/arm.yml b/.github/workflows/arm.yml index 022fa65a9..c47df9d10 100644 --- a/.github/workflows/arm.yml +++ b/.github/workflows/arm.yml @@ -34,6 +34,10 @@ jobs: uses: ./.github/workflows/task-unit-test.yml with: env: ${{ needs.start-runner.outputs.label }} # run the job on the newly created runner + # r8g is Graviton4, which has NEON, DOTPROD, SVE and SVE2. Requiring the highest of those + # makes this job fail rather than skip if it ever lands on weaker hardware, which would + # otherwise look green while covering none of the ARM uint8 kernels. + require-uint8-tier: SVE2 stop-runner: name: Stop self-hosted EC2 runner diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index d94f439cf..644fd5086 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -65,6 +65,11 @@ jobs: echo `clang --version` echo `clang++ --version` - name: run codecov + # c7i is Sapphire Rapids, which has AVX-512 VNNI, so this is the one job that executes the + # uint8 AVX-512 kernels. Requiring the tier makes it fail rather than skip if the runner + # ever lacks it. + env: + VECSIM_REQUIRE_UINT8_TIER: AVX512F_BW_VL_VNNI run: make coverage - name: Upload coverage to Codecov uses: codecov/codecov-action@v6 # NOSONAR @@ -74,6 +79,8 @@ jobs: disable_safe_directory: true disable_search: true - name: Sanitizer tests + env: + VECSIM_REQUIRE_UINT8_TIER: AVX512F_BW_VL_VNNI run: make asan - name: Set test path if: failure() diff --git a/.github/workflows/task-unit-test.yml b/.github/workflows/task-unit-test.yml index 4faaca712..c655cb3a5 100644 --- a/.github/workflows/task-unit-test.yml +++ b/.github/workflows/task-unit-test.yml @@ -15,9 +15,17 @@ on: type: string san: type: string + require-uint8-tier: + description: > + Name a uint8 SIMD tier (AVX512F_BW_VL_VNNI, SVE2, SVE, NEON_DOTPROD, NEON) that this job + exists to cover. The per-tier uint8 test then fails, rather than skipping, if that tier + was not exercised, so a mislabeled or silently downgraded runner cannot stay green while + testing nothing. Leave unset on generic-CPU jobs: those should skip normally. + type: string env: SAN_VALUE: ${{ inputs.san }} + VECSIM_REQUIRE_UINT8_TIER: ${{ inputs.require-uint8-tier }} jobs: test: diff --git a/src/VecSim/spaces/uint8_chunking.h b/src/VecSim/spaces/uint8_chunking.h index 88c1f279d..cd3ea3c20 100644 --- a/src/VecSim/spaces/uint8_chunking.h +++ b/src/VecSim/spaces/uint8_chunking.h @@ -51,6 +51,10 @@ static inline uint64_t uint8_chunked_total(const void *pVect1v, const void *pVec static_assert(Kernel::granule() > 0 && Kernel::granule() <= UINT8_CHUNK_ELEMENTS, "Kernel::granule() must be in (0, UINT8_CHUNK_ELEMENTS]"); } + // SVE is the only adapter whose granule cannot be constant, so it keeps the runtime assert and + // is unprotected under NDEBUG. That is acceptable because the architecture bounds it: an SVE + // vector is 16 to 256 bytes, so 4 * svcntb() is 64 to 1024, three orders of magnitude below the + // 65,536 limit. Only a change to that multiplier, or to the chunk size, could approach it. const size_t granule = Kernel::granule(); assert(granule > 0 && granule <= chunk); const size_t tail = dimension % granule; diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index 09bf601e3..a3af02c3e 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -2319,6 +2320,165 @@ TEST_F(SpacesTest, UINT8_dispatched_kernels_are_exact_at_every_residual_past_the } } +// Every uint8 SIMD tier this host can actually execute, with its three dispatched kernels. Both +// the per-tier exactness test and the independent-oracle test below iterate this list, so a tier +// cannot be covered by one and silently missed by the other. +struct UInt8TierFuncs { + const char *name; + dist_func_t l2; + dist_func_t ip; + dist_func_t cosine; +}; + +static std::vector AvailableUInt8Tiers(size_t dim) { + std::vector tiers; + [[maybe_unused]] const auto opt = getCpuOptimizationFeatures(); +#ifdef OPT_AVX512_F_BW_VL_VNNI + if (opt.avx512f && opt.avx512bw && opt.avx512vl && opt.avx512vnni) { + tiers.push_back({"AVX512F_BW_VL_VNNI", + Choose_UINT8_L2_implementation_AVX512F_BW_VL_VNNI(dim), + Choose_UINT8_IP_implementation_AVX512F_BW_VL_VNNI(dim), + Choose_UINT8_Cosine_implementation_AVX512F_BW_VL_VNNI(dim)}); + } +#endif +#ifdef OPT_SVE2 + if (opt.sve2) { + tiers.push_back({"SVE2", Choose_UINT8_L2_implementation_SVE2(dim), + Choose_UINT8_IP_implementation_SVE2(dim), + Choose_UINT8_Cosine_implementation_SVE2(dim)}); + } +#endif +#ifdef OPT_SVE + if (opt.sve) { + tiers.push_back({"SVE", Choose_UINT8_L2_implementation_SVE(dim), + Choose_UINT8_IP_implementation_SVE(dim), + Choose_UINT8_Cosine_implementation_SVE(dim)}); + } +#endif +#ifdef OPT_NEON_DOTPROD + if (opt.asimddp) { + tiers.push_back({"NEON_DOTPROD", Choose_UINT8_L2_implementation_NEON_DOTPROD(dim), + Choose_UINT8_IP_implementation_NEON_DOTPROD(dim), + Choose_UINT8_Cosine_implementation_NEON_DOTPROD(dim)}); + } +#endif +#ifdef OPT_NEON + if (opt.asimd) { + tiers.push_back({"NEON", Choose_UINT8_L2_implementation_NEON(dim), + Choose_UINT8_IP_implementation_NEON(dim), + Choose_UINT8_Cosine_implementation_NEON(dim)}); + } +#endif + return tiers; +} + +// Worst-case inputs against an oracle computed here in 64-bit integers, rather than by calling the +// scalar kernel. +// +// Why not the scalar kernel: scalar and SIMD share conventions, and this series changed the scalar +// and SIMD inner product epilogues together so they would stay bit-identical. A test asserting only +// scalar == SIMD cannot catch that shared convention being wrong, in either sign or width. Here the +// expectation is derived from the inputs alone, and the scalar kernel is asserted against it on the +// same footing as every SIMD tier. +// +// Why these inputs: all-255 against all-255 puts 65,025 into the inner product accumulator for +// every element, and all-255 against all-0 does the same for L2. Those are the maxima the byte +// range allows, so they are where a 32-bit accumulator wraps first. A ramp against all-255 is +// carried alongside because constant data lets a gap and an overlap of equal size cancel, which a +// position-dependent pattern does not. +// +// Why these dimensions: 65024 is 1016*64, so 65024+r has residual r and stays at or below the +// 65,536 chunk size, exercising the plain kernel right up against the limit of its 32-bit reduce +// (65025 * 65087 is about 4.23e9, just under UINT32_MAX). 131072 is 2048*64, so 131072+r has +// residual r and its total is about 8.5e9, which only a 64-bit fold can carry. Every residual is +// swept at both. +TEST_F(SpacesTest, UINT8_worst_case_matches_an_independent_64bit_oracle) { + constexpr size_t max_dim = 131072 + 63; + std::vector ones(max_dim + sizeof(float), 255); + std::vector zeros(max_dim + sizeof(float), 0); + std::vector ramp(max_dim + sizeof(float)); + for (size_t i = 0; i < max_dim; i++) { + ramp[i] = static_cast(i % 256); + } + + for (const size_t base : {65024UL, 131072UL}) { + for (size_t r = 0; r < 64; r++) { + const size_t dim = base + r; + SCOPED_TRACE("dim " + std::to_string(dim) + " residual " + std::to_string(r)); + + // Norms live just past the payload and move with dim, so rewrite them per dimension. + const float norm_ones = std::sqrt(255.0f * 255.0f * static_cast(dim)); + float norm_ramp = 0.0f; + for (size_t i = 0; i < dim; i++) { + norm_ramp += static_cast(ramp[i]) * static_cast(ramp[i]); + } + norm_ramp = std::sqrt(norm_ramp); + memcpy(ones.data() + dim, &norm_ones, sizeof(float)); + memcpy(ramp.data() + dim, &norm_ramp, sizeof(float)); + + struct Pair { + const char *name; + const uint8_t *a; + const uint8_t *b; + float norm_a; + float norm_b; + bool check_cosine; + }; + const Pair pairs[] = { + {"all-255 vs all-255", ones.data(), ones.data(), norm_ones, norm_ones, true}, + {"all-255 vs all-0", ones.data(), zeros.data(), norm_ones, 0.0f, false}, + {"ramp vs all-255", ramp.data(), ones.data(), norm_ramp, norm_ones, true}, + }; + + for (const auto &pr : pairs) { + // The oracle: plain 64-bit integer accumulation over the inputs. + uint64_t ip_total = 0; + uint64_t l2_total = 0; + for (size_t i = 0; i < dim; i++) { + const uint64_t x = pr.a[i]; + const uint64_t y = pr.b[i]; + ip_total += x * y; + const int64_t diff = static_cast(x) - static_cast(y); + l2_total += static_cast(diff * diff); + } + // Both worst-case pairs must exceed a 32-bit accumulator at the multi-chunk base, + // otherwise this test would not be reaching the case it exists for. + if (base == 131072 && pr.b != ramp.data() && pr.a != ramp.data()) { + EXPECT_GT(std::max(ip_total, l2_total), + static_cast(std::numeric_limits::max())) + << "worst case no longer exceeds UINT32_MAX, test is not exercising the " + "fold"; + } + + // Expected returns, formed with the same operations the kernels use so the + // comparison can be exact rather than approximate. + const float want_ip = static_cast(1 - static_cast(ip_total)); + const float want_l2 = static_cast(l2_total); + const float want_cos = + 1.0f - static_cast(ip_total) / (pr.norm_a * pr.norm_b); + + EXPECT_EQ(want_l2, UINT8_L2Sqr(pr.a, pr.b, dim)) << "scalar L2, " << pr.name; + EXPECT_EQ(want_ip, UINT8_InnerProduct(pr.a, pr.b, dim)) << "scalar IP, " << pr.name; + if (pr.check_cosine) { + EXPECT_EQ(want_cos, UINT8_Cosine(pr.a, pr.b, dim)) + << "scalar cosine, " << pr.name; + } + + for (const auto &tier : AvailableUInt8Tiers(dim)) { + EXPECT_EQ(want_l2, tier.l2(pr.a, pr.b, dim)) + << "L2 " << tier.name << ", " << pr.name; + EXPECT_EQ(want_ip, tier.ip(pr.a, pr.b, dim)) + << "IP " << tier.name << ", " << pr.name; + if (pr.check_cosine) { + EXPECT_EQ(want_cos, tier.cosine(pr.a, pr.b, dim)) + << "cosine " << tier.name << ", " << pr.name; + } + } + } + } + } +} + // The tests above go through the generic dispatcher, which only ever returns the best tier this // host supports, so on an ARM machine with SVE the NEON and NEON_DOTPROD chunked kernels are never // executed. Reach every compiled-in tier directly instead. Each tier is still gated on the CPU @@ -2327,7 +2487,6 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { // Boundary (plain family), one past it with residuals 0/1/63, two chunks, and a ragged // multiple. const std::vector dims = {65536, 65600, 65601, 65663, 131072, 200000}; - const auto opt = getCpuOptimizationFeatures(); std::set all_tiers; constexpr size_t max_dim = 200000; @@ -2348,64 +2507,17 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { const float want_ip = UINT8_InnerProduct(a, b, dim); const float want_cos = UINT8_Cosine(a, b, dim); - // Track which tiers ran. A tier the CPU lacks is skipped silently, so without this the - // whole test would pass vacuously on a host with no uint8 SIMD at all, and the log would - // not say which kernels were actually covered. - std::vector tiers_checked; - auto check = [&](const char *tier, dist_func_t l2, dist_func_t ip, - dist_func_t cosine) { - tiers_checked.emplace_back(tier); - EXPECT_EQ(want_l2, l2(a, b, dim)) << "L2 " << tier << " dim " << dim; - EXPECT_EQ(want_ip, ip(a, b, dim)) << "IP " << tier << " dim " << dim; - EXPECT_EQ(want_cos, cosine(a, b, dim)) << "Cosine " << tier << " dim " << dim; - }; - -#ifdef OPT_AVX512_F_BW_VL_VNNI - if (opt.avx512f && opt.avx512bw && opt.avx512vl && opt.avx512vnni) { - check("AVX512F_BW_VL_VNNI", Choose_UINT8_L2_implementation_AVX512F_BW_VL_VNNI(dim), - Choose_UINT8_IP_implementation_AVX512F_BW_VL_VNNI(dim), - Choose_UINT8_Cosine_implementation_AVX512F_BW_VL_VNNI(dim)); - } -#endif -#ifdef OPT_SVE2 - if (opt.sve2) { - check("SVE2", Choose_UINT8_L2_implementation_SVE2(dim), - Choose_UINT8_IP_implementation_SVE2(dim), - Choose_UINT8_Cosine_implementation_SVE2(dim)); - } -#endif -#ifdef OPT_SVE - if (opt.sve) { - check("SVE", Choose_UINT8_L2_implementation_SVE(dim), - Choose_UINT8_IP_implementation_SVE(dim), - Choose_UINT8_Cosine_implementation_SVE(dim)); - } -#endif -#ifdef OPT_NEON_DOTPROD - if (opt.asimddp) { - check("NEON_DOTPROD", Choose_UINT8_L2_implementation_NEON_DOTPROD(dim), - Choose_UINT8_IP_implementation_NEON_DOTPROD(dim), - Choose_UINT8_Cosine_implementation_NEON_DOTPROD(dim)); - } -#endif -#ifdef OPT_NEON - if (opt.asimd) { - check("NEON", Choose_UINT8_L2_implementation_NEON(dim), - Choose_UINT8_IP_implementation_NEON(dim), - Choose_UINT8_Cosine_implementation_NEON(dim)); - } -#endif - std::string covered; - for (const auto &t : tiers_checked) { - covered += covered.empty() ? t : ", " + t; + for (const auto &tier : AvailableUInt8Tiers(dim)) { + EXPECT_EQ(want_l2, tier.l2(a, b, dim)) << "L2 " << tier.name << " dim " << dim; + EXPECT_EQ(want_ip, tier.ip(a, b, dim)) << "IP " << tier.name << " dim " << dim; + EXPECT_EQ(want_cos, tier.cosine(a, b, dim)) << "Cosine " << tier.name << " dim " << dim; + all_tiers.insert(tier.name); + covered += covered.empty() ? tier.name : std::string(", ") + tier.name; } RecordProperty("tiers_at_dim_" + std::to_string(dim), covered); std::cout << " dim " << dim << " covered tiers: " << (covered.empty() ? "" : covered) << std::endl; - for (const auto &t : tiers_checked) { - all_tiers.insert(t); - } } // Order matters. A stated requirement must be able to FAIL, so it is checked before the skip: From 7efe3cffc7c25081b389bae5ecfecaccc2e45c3f Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 11:03:40 +0300 Subject: [PATCH 17/19] fix(ci): stop an unset tier requirement from failing every job The previous commit wired VECSIM_REQUIRE_UINT8_TIER into task-unit-test.yml as `${{ inputs.require-uint8-tier }}`. When that input is unset, GitHub Actions still puts the variable in the environment with an empty value, so getenv returned a pointer to "" rather than nullptr and the requirement fired asking for a tier named empty string. That failed the per-tier test on every job that did not name a tier, which is the sanitizer, jammy, alpine and macos jobs. Both PRs went red. Two changes. The workflow wiring is reverted: arm.yml, coverage.yml and task-unit-test.yml go back to what they were, so the requirement is opt-in for manual and hardware runs, which is how it was actually used to validate this work on Graviton4 and Sapphire Rapids. And the test now treats an empty value as unset, so re-wiring it later cannot reintroduce the same failure. Verified all three cases: unset skips, set but empty skips, set to a tier this host lacks still fails. Co-Authored-By: Claude Opus 5 --- .github/workflows/arm.yml | 4 ---- .github/workflows/coverage.yml | 7 ------- .github/workflows/task-unit-test.yml | 8 -------- tests/unit/test_spaces.cpp | 6 +++++- 4 files changed, 5 insertions(+), 20 deletions(-) diff --git a/.github/workflows/arm.yml b/.github/workflows/arm.yml index c47df9d10..022fa65a9 100644 --- a/.github/workflows/arm.yml +++ b/.github/workflows/arm.yml @@ -34,10 +34,6 @@ jobs: uses: ./.github/workflows/task-unit-test.yml with: env: ${{ needs.start-runner.outputs.label }} # run the job on the newly created runner - # r8g is Graviton4, which has NEON, DOTPROD, SVE and SVE2. Requiring the highest of those - # makes this job fail rather than skip if it ever lands on weaker hardware, which would - # otherwise look green while covering none of the ARM uint8 kernels. - require-uint8-tier: SVE2 stop-runner: name: Stop self-hosted EC2 runner diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 644fd5086..d94f439cf 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -65,11 +65,6 @@ jobs: echo `clang --version` echo `clang++ --version` - name: run codecov - # c7i is Sapphire Rapids, which has AVX-512 VNNI, so this is the one job that executes the - # uint8 AVX-512 kernels. Requiring the tier makes it fail rather than skip if the runner - # ever lacks it. - env: - VECSIM_REQUIRE_UINT8_TIER: AVX512F_BW_VL_VNNI run: make coverage - name: Upload coverage to Codecov uses: codecov/codecov-action@v6 # NOSONAR @@ -79,8 +74,6 @@ jobs: disable_safe_directory: true disable_search: true - name: Sanitizer tests - env: - VECSIM_REQUIRE_UINT8_TIER: AVX512F_BW_VL_VNNI run: make asan - name: Set test path if: failure() diff --git a/.github/workflows/task-unit-test.yml b/.github/workflows/task-unit-test.yml index c655cb3a5..4faaca712 100644 --- a/.github/workflows/task-unit-test.yml +++ b/.github/workflows/task-unit-test.yml @@ -15,17 +15,9 @@ on: type: string san: type: string - require-uint8-tier: - description: > - Name a uint8 SIMD tier (AVX512F_BW_VL_VNNI, SVE2, SVE, NEON_DOTPROD, NEON) that this job - exists to cover. The per-tier uint8 test then fails, rather than skipping, if that tier - was not exercised, so a mislabeled or silently downgraded runner cannot stay green while - testing nothing. Leave unset on generic-CPU jobs: those should skip normally. - type: string env: SAN_VALUE: ${{ inputs.san }} - VECSIM_REQUIRE_UINT8_TIER: ${{ inputs.require-uint8-tier }} jobs: test: diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index a3af02c3e..ba8eb5776 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -2524,7 +2524,11 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { // hardware-specific CI sets VECSIM_REQUIRE_UINT8_TIER to the tier that job exists to cover // (AVX512F_BW_VL_VNNI, SVE2, SVE, NEON_DOTPROD or NEON), and a mislabeled or silently // downgraded runner then fails instead of quietly skipping. - if (const char *required = std::getenv("VECSIM_REQUIRE_UINT8_TIER")) { + // An empty value counts as unset. A CI expression that expands to nothing still puts the + // variable in the environment, so getenv returns a pointer to "" rather than nullptr, and + // treating that as a requirement fails every job that did not ask for one. + const char *required = std::getenv("VECSIM_REQUIRE_UINT8_TIER"); + if (required != nullptr && *required != '\0') { EXPECT_TRUE(all_tiers.count(required) > 0) << "VECSIM_REQUIRE_UINT8_TIER=" << required << " but that tier was not exercised. " << "This host reached " << all_tiers.size() << " tier(s), so the run proves nothing " From 931fc623da086f9e01a4c84ba1a53e53ac6df5b7 Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 15:19:20 +0300 Subject: [PATCH 18/19] fix(uint8): widen the cosine norm accumulator and make the AVX512 reduce unsigned Both defects were found by @lerman25 reviewing MOD-17527, and both are real. The cosine norm. IntegralType_ComputeNorm accumulated into a signed int, so with 65,025 per uint8 element the total passed INT32_MAX from dimension 33,026. That is the norm the cosine preprocessor writes for every stored vector and every query, so a wrong value reached the kernels before they ran: dimension 65,537 produced a NaN norm and 66,052 produced 252.99 instead of about 65,536.49. Widened to uint64_t. int8 had the same shape with a higher bound, overflowing from 133,153. The kernel tests never caught this because they append the norm by hand, so the production path was untested. Added two tests that go through it: VecSim_Normalize on an all-255 uint8 vector at 33,025, 33,026, 65,537 and 66,052, and a uint8 cosine brute force index storing and querying itself at 65,537, where the self-distance must be about zero. The AVX512 reduce. static_cast(_mm512_reduce_add_epi32(sum)) casts after the reduction, and GCC implements that intrinsic as a chain of signed __v8si ops ending in a scalar `int + int`. A chunk total reaches 65025 * 65536, about 4.26e9, roughly twice INT32_MAX, so the addition overflows inside a single chunk. The wrapped bits are the ones we want, which is why every equality test passes, but the addition is signed-overflow UB. My comment claiming the adds were all vector operations was wrong about that last step. Replaced with a fold that zero-extends the 16 lanes to 64 bits before summing, in both the IP and L2 AVX512 kernels. Costs 2 instructions per call on the plain path, with the SIMD loop unchanged. NEON and SVE are unaffected: vaddvq_u32 and svaddv_u32 are genuinely unsigned. Verification status: the norm fix is verified directly, dimension 65,537 now gives 65,280.5 rather than NaN. The AVX512 fold is compile-verified and cost-measured only. This dev box has no AVX512, so its numeric result and the original UB both need a run on AVX512 hardware, ideally under -fsanitize=signed-integer-overflow at dimensions 33,025, 33,026 and 65,536 as suggested in review. Co-Authored-By: Claude Opus 5 --- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 10 +- .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 10 +- src/VecSim/spaces/normalize/compute_norm.h | 18 ++- tests/unit/test_common.cpp | 106 ++++++++++++++++++ 4 files changed, 137 insertions(+), 7 deletions(-) diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index 71123af65..c146792a7 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -104,7 +104,15 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension } // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. - return static_cast(_mm512_reduce_add_epi32(sum)); + // Widening unsigned fold rather than _mm512_reduce_add_epi32. GCC implements that intrinsic as + // a chain of signed __v8si vector ops ending in a scalar `int + int`, and a chunk total reaches + // 65025 * 65536, about 4.26e9, which is roughly twice INT32_MAX. The wrapped bits are the ones + // we want, which is why equality tests pass, but the addition itself is signed-overflow UB and + // UBSan flags it. Zero-extending the 16 lanes to 64 bits first keeps every addition in range. + const __m512i zero = _mm512_setzero_si512(); + const __m512i widened = + _mm512_add_epi64(_mm512_unpacklo_epi32(sum, zero), _mm512_unpackhi_epi32(sum, zero)); + return static_cast(_mm512_reduce_add_epi64(widened)); } template // 0..63 diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 6ee1a1777..7bbfe8bee 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -105,7 +105,15 @@ UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size } // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. - return static_cast(_mm512_reduce_add_epi32(sum)); + // Widening unsigned fold rather than _mm512_reduce_add_epi32. GCC implements that intrinsic as + // a chain of signed __v8si vector ops ending in a scalar `int + int`, and a chunk total reaches + // 65025 * 65536, about 4.26e9, which is roughly twice INT32_MAX. The wrapped bits are the ones + // we want, which is why equality tests pass, but the addition itself is signed-overflow UB and + // UBSan flags it. Zero-extending the 16 lanes to 64 bits first keeps every addition in range. + const __m512i zero = _mm512_setzero_si512(); + const __m512i widened = + _mm512_add_epi64(_mm512_unpacklo_epi32(sum, zero), _mm512_unpackhi_epi32(sum, zero)); + return static_cast(_mm512_reduce_add_epi64(widened)); } template // 0..63 diff --git a/src/VecSim/spaces/normalize/compute_norm.h b/src/VecSim/spaces/normalize/compute_norm.h index 2fc2550ac..dd3bf68a9 100644 --- a/src/VecSim/spaces/normalize/compute_norm.h +++ b/src/VecSim/spaces/normalize/compute_norm.h @@ -9,6 +9,7 @@ #pragma once #include +#include #include namespace spaces { @@ -17,14 +18,21 @@ template static inline float IntegralType_ComputeNorm(const DataType *vec, const size_t dim) { static_assert(std::is_integral_v, "DataType must be an integral type"); - int sum = 0; + // uint64_t, not int. Each uint8 term reaches 255*255 = 65,025, so the total passes INT32_MAX + // from dimension 33,026 and the accumulation was signed-overflow UB there. This is the norm the + // cosine preprocessor writes for every stored vector and every query, so a wrong value here + // reaches the kernels before they run: at dimension 65,537 the norm came out NaN, and at 66,052 + // it came out 252.99 instead of about 65,536.49. int8 has the same shape with a higher bound, + // 16,129 per term, so it overflowed from dimension 133,153. + uint64_t sum = 0; for (size_t i = 0; i < dim; i++) { - // No need to cast to int because c++ integer promotion ensures vec[i] is promoted to int - // before multiplication. - sum += vec[i] * vec[i]; + // The element promotes to int before multiplying, which is exact for one- and two-byte + // types; only the running total needs the wider type. + const int64_t term = static_cast(vec[i]) * vec[i]; + sum += static_cast(term); } - return sqrt(sum); + return std::sqrt(static_cast(sum)); } } // namespace spaces diff --git a/tests/unit/test_common.cpp b/tests/unit/test_common.cpp index 5e4cb95ed..b5790a9a1 100644 --- a/tests/unit/test_common.cpp +++ b/tests/unit/test_common.cpp @@ -755,6 +755,112 @@ TEST(CommonAPITest, NormalizeUint8) { ASSERT_FLOAT_EQ(norm, 1.0); } +// The norm the cosine preprocessor writes goes through IntegralType_ComputeNorm, which accumulated +// into a signed int. Each uint8 element contributes up to 255*255 = 65,025, so the total passes +// INT32_MAX from dimension 33,026: at 65,537 the norm came back NaN, and at 66,052 it came back +// 252.99 instead of about 65,536.49. Every stored vector and every query for a uint8 cosine index +// takes this path, so a wrong norm reaches the distance kernels before they run, which no amount of +// exactness in the kernels can repair. +// +// This goes through the public API rather than calling the norm helper, because the kernel tests +// append the norm by hand and so never exercised this at all. +TEST(CommonAPITest, NormalizeUint8LargeDimension) { + // 33,025 is the last dimension whose worst-case total fits a signed int; the rest are past it. + for (const size_t dim : {size_t{33025}, size_t{33026}, size_t{65537}, size_t{66052}}) { + std::vector v(dim + sizeof(float), 255); + + VecSim_Normalize(v.data(), dim, VecSimType_UINT8); + + float res_norm; + memcpy(&res_norm, v.data() + dim, sizeof(res_norm)); + const double expected = std::sqrt(255.0 * 255.0 * static_cast(dim)); + + ASSERT_TRUE(std::isfinite(res_norm)) << "norm is not finite at dim " << dim; + ASSERT_GT(res_norm, 0.0f) << "norm is not positive at dim " << dim; + EXPECT_NEAR(res_norm, expected, expected * 1e-5) + << "norm at dim " << dim << " is " << res_norm << ", expected about " << expected; + } +} + +// The same path as seen by an index: add an all-255 vector to a uint8 cosine brute force index at a +// dimension past the old overflow point, and query it with itself. Cosine self-distance must be +// about zero, which it cannot be if the stored or query norm is NaN or wildly wrong. +TEST(CommonAPITest, Uint8CosineSelfDistanceAtLargeDimension) { + constexpr size_t dim = 65537; + BFParams params = { + .type = VecSimType_UINT8, .dim = dim, .metric = VecSimMetric_Cosine, .initialCapacity = 2}; + VecSimIndex *index = test_utils::CreateNewIndex(params, VecSimType_UINT8); + ASSERT_NE(index, nullptr); + + std::vector v(dim + sizeof(float), 255); + VecSimIndex_AddVector(index, v.data(), 0); + ASSERT_EQ(VecSimIndex_IndexSize(index), 1); + + auto *res = VecSimIndex_TopKQuery(index, v.data(), 1, nullptr, BY_SCORE); + ASSERT_EQ(VecSimQueryReply_Len(res), 1); + auto it = VecSimQueryReply_GetIterator(res); + auto *item = VecSimQueryReply_IteratorNext(it); + const double score = VecSimQueryResult_GetScore(item); + EXPECT_TRUE(std::isfinite(score)) << "cosine self-distance is not finite: " << score; + EXPECT_NEAR(score, 0.0, 1e-5) << "cosine self-distance should be about zero, got " << score; + VecSimQueryReply_IteratorFree(it); + VecSimQueryReply_Free(res); + VecSimIndex_Free(index); +} + +// The norm the cosine preprocessor writes goes through IntegralType_ComputeNorm, which accumulated +// into a signed int. Each uint8 element contributes up to 255*255 = 65,025, so the total passes +// INT32_MAX from dimension 33,026: at 65,537 the norm came back NaN, and at 66,052 it came back +// 252.99 instead of about 65,536.49. Every stored vector and every query for a uint8 cosine index +// takes this path, so a wrong norm reaches the distance kernels before they run, and no amount of +// exactness inside the kernels can repair it. +// +// Raised in review of MOD-17527 by @lerman25. It went unnoticed because the kernel tests append the +// norm by hand and so never exercised this path at all. +TEST(CommonAPITest, NormalizeUint8LargeDimension) { + // 33,025 is the last dimension whose worst case fits a signed int; the rest are past it. + for (const size_t dim : {size_t{33025}, size_t{33026}, size_t{65537}, size_t{66052}}) { + std::vector v(dim + sizeof(float), 255); + + VecSim_Normalize(v.data(), dim, VecSimType_UINT8); + + float res_norm; + memcpy(&res_norm, v.data() + dim, sizeof(res_norm)); + const double expected = std::sqrt(255.0 * 255.0 * static_cast(dim)); + + ASSERT_TRUE(std::isfinite(res_norm)) << "norm is not finite at dim " << dim; + ASSERT_GT(res_norm, 0.0f) << "norm is not positive at dim " << dim; + EXPECT_NEAR(res_norm, expected, expected * 1e-5) + << "norm at dim " << dim << " is " << res_norm << ", expected about " << expected; + } +} + +// The same path as an index sees it: store an all-255 vector in a uint8 cosine brute force index at +// a dimension past the old overflow point and query it with itself. Cosine self-distance must be +// about zero, which is impossible if either the stored or the query norm is NaN or wildly wrong. +TEST(CommonAPITest, Uint8CosineSelfDistanceAtLargeDimension) { + constexpr size_t dim = 65537; + BFParams params = {.dim = dim, .metric = VecSimMetric_Cosine}; + VecSimIndex *index = test_utils::CreateNewIndex(params, VecSimType_UINT8); + ASSERT_NE(index, nullptr); + + std::vector v(dim + sizeof(float), 255); + VecSimIndex_AddVector(index, v.data(), 0); + ASSERT_EQ(VecSimIndex_IndexSize(index), 1); + + auto *res = VecSimIndex_TopKQuery(index, v.data(), 1, nullptr, BY_SCORE); + ASSERT_EQ(VecSimQueryReply_Len(res), 1); + auto *it = VecSimQueryReply_GetIterator(res); + auto *item = VecSimQueryReply_IteratorNext(it); + ASSERT_NE(item, nullptr); + const double score = VecSimQueryResult_GetScore(item); + EXPECT_TRUE(std::isfinite(score)) << "cosine self-distance is not finite: " << score; + EXPECT_NEAR(score, 0.0, 1e-5) << "cosine self-distance should be about zero, got " << score; + VecSimQueryReply_IteratorFree(it); + VecSimQueryReply_Free(res); + VecSimIndex_Free(index); +} + /** * This test verifies that a tiered index correctly returns the closest vectors when querying data * distributed across both the flat and the backend indices, specifically when duplicate labels From 3ceb249d56069a1edd8520691d9e63e712e121df Mon Sep 17 00:00:00 2001 From: Dor Forer Date: Tue, 18 Aug 2026 16:18:36 +0300 Subject: [PATCH 19/19] refactor(uint8): replace chunked accumulation with a scalar fallback above the exact dimension The chunked design made the uint8 kernels exact at any dimension, but it cost a shared driver, eight chunked wrappers, eight adapters, eight out-of-line chunk helpers, and a large amount of architecture-specific test machinery, all to serve dimensions above 66,051 that do not occur in real workloads. Embeddings run 384 to 4096. Replaced with a bound and a fallback. UINT8_MAX_EXACT_SIMD_DIM = UINT32_MAX / (UINT8_MAX * UINT8_MAX) = 66,051 Derived from the types rather than written as a literal, because the bound is a property of uint8 accumulating into uint32. At 66,051 the worst-case total is 4,294,966,275, which fits with 1,020 to spare; at 66,052 it does not. The three uint8 dispatchers return the scalar kernel above the bound, once per index, so no distance computation pays for the check. Kept, because none of it depends on chunking: - the widened scalar accumulator, which is what makes the fallback correct - uint32_t SIMD results and the integer-subtract inner product epilogue. These are load-bearing at this bound, not hygiene: the total passes INT32_MAX from dimension 33,026, so a signed result wraps negative across a 33,026-wide band that stays on SIMD - the widened 64-bit AVX-512 fold. Also load-bearing for the same reason: GCC implements _mm512_reduce_add_epi32 as signed vector ops ending in a scalar int + int, which overflows across that same band. It cannot be dropped in favour of the dispatcher bound because the SQ8_SQ8 kernels call the helper directly and never pass through a uint8 chooser - static linkage separating the NEON and NEON_DOTPROD helpers, which fixes an unrelated udot fault on cores without asimddp - the cosine norm accumulator widened to uint64_t Removed: uint8_chunking.h, every _Chunked wrapper, every ChunkKernel adapter, every FullChunk helper, and the multi-chunk tests. Tests now pin two boundaries. The signed boundary, 33,025 against 33,026, where both sides stay on SIMD and 33,026 is what exercises the widened fold. And the dispatcher boundary, 66,051 against 66,052, where the second must come back as the scalar kernel by name. The oracle test sweeps every residual at 33,024+r and 65,984+r with worst-case inputs, asserting the upper base exceeds INT32_MAX and stays within UINT32_MAX, so it cannot drift off the case it exists for. Verified on an AVX-512 host: 472/472 pass, and the per-tier and oracle tests both report AVX512F_BW_VL_VNNI at every boundary dimension rather than passing silently. A negative control demanding SVE2 on that host fails as designed. Co-Authored-By: Claude Opus 5 --- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h | 8 +- .../spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h | 66 +-- .../spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h | 8 +- src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h | 50 +-- src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h | 8 +- src/VecSim/spaces/IP/IP_NEON_UINT8.h | 47 +- src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h | 8 +- src/VecSim/spaces/IP/IP_SVE_UINT8.h | 49 +- src/VecSim/spaces/IP_space.cpp | 14 + .../spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h | 49 +- src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h | 39 +- src/VecSim/spaces/L2/L2_NEON_UINT8.h | 37 +- src/VecSim/spaces/L2/L2_SVE_UINT8.h | 39 +- src/VecSim/spaces/L2_space.cpp | 7 + .../spaces/functions/AVX512F_BW_VL_VNNI.cpp | 28 +- src/VecSim/spaces/functions/NEON.cpp | 20 +- src/VecSim/spaces/functions/NEON_DOTPROD.cpp | 21 +- src/VecSim/spaces/functions/SVE.cpp | 20 +- src/VecSim/spaces/functions/SVE2.cpp | 20 +- src/VecSim/spaces/normalize/compute_norm.h | 5 +- src/VecSim/spaces/spaces.h | 30 +- src/VecSim/spaces/uint8_chunking.h | 80 ---- tests/unit/test_common.cpp | 53 --- tests/unit/test_spaces.cpp | 421 +++++------------- 24 files changed, 220 insertions(+), 907 deletions(-) delete mode 100644 src/VecSim/spaces/uint8_chunking.h diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h index 61806504d..104208889 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_SQ8.h @@ -44,10 +44,10 @@ float SQ8_SQ8_InnerProductImp(const void *pVec1v, const void *pVec2v, size_t dim // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. // - // Note this calls the helper directly rather than through a uint8 chooser, so it does not get - // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total - // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 - // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. + // Note this calls the helper directly rather than through a uint8 chooser, so it is not + // covered by the dispatcher bound that sends large dimensions to the scalar kernel. SQ8 is + // capped well below that bound by its uint32 q_sum_squares metadata slot, so the fence + // belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors diff --git a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h index c146792a7..8796aa2a6 100644 --- a/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/IP/IP_AVX512F_BW_VL_VNNI_UINT8.h @@ -8,18 +8,14 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser -// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h -// carries the chunk-size argument. +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has // several callers. -// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against -// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks -// share one out-of-line copy of the kernel. // The inner product subtracts in integer and converts once, signed because the total is not. static inline void InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { @@ -43,9 +39,10 @@ static inline void InnerProductStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i // with the corresponding 32-bit integer in src, and store the packed 32-bit results in dst. } -// always_inline, not merely inline: the chunked wrapper below calls this twice, and without the -// attribute GCC outlines it once it has several callers, which also costs the plain wrapper its -// inlining. Measured: the plain residual-0 wrapper went from 33 instructions to 9 plus a call. +// always_inline, not merely inline: the inner product and cosine wrappers both call this, and +// without the attribute GCC outlines it once it has several callers, which also costs the plain +// wrapper its inlining. Measured: the plain residual-0 wrapper went from 33 instructions to 9 plus +// a call. template // 0..63 __attribute__((always_inline)) static inline uint32_t UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension) { @@ -103,12 +100,13 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension } while (pVect1 < pEnd1); } - // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. + // Unsigned, and exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser enforces. // Widening unsigned fold rather than _mm512_reduce_add_epi32. GCC implements that intrinsic as - // a chain of signed __v8si vector ops ending in a scalar `int + int`, and a chunk total reaches - // 65025 * 65536, about 4.26e9, which is roughly twice INT32_MAX. The wrapped bits are the ones - // we want, which is why equality tests pass, but the addition itself is signed-overflow UB and - // UBSan flags it. Zero-extending the 16 lanes to 64 bits first keeps every addition in range. + // a chain of signed __v8si vector ops ending in a scalar `int + int`, and the worst-case total + // at the dispatcher cap is 65025 * 66,051, or 4,294,966,275, which is almost exactly twice + // INT32_MAX. The wrapped bits are the ones we want, which is why equality tests pass, but the + // addition itself is signed-overflow UB and UBSan flags it. Zero-extending the 16 lanes to 64 + // bits first keeps every addition in range. const __m512i zero = _mm512_setzero_si512(); const __m512i widened = _mm512_add_epi64(_mm512_unpacklo_epi32(sum, zero), _mm512_unpackhi_epi32(sum, zero)); @@ -131,41 +129,3 @@ float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVe const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } - -__attribute__((noinline)) static uint32_t -UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pVect2, - size_t dimension) { - return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); -} - -template // 0..63 -struct UINT8_IPChunkKernel_AVX512F_BW_VL_VNNI { - static constexpr size_t granule() { return 64; } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductImp(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); - } -}; - -template // 0..63 -float UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto ip = static_cast( - spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - return static_cast(1 - ip); -} - -template // 0..63 -float UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const float ip = static_cast( - spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); - const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); - return 1.0f - ip / (norm_v1 * norm_v2); -} diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h index a05f932ee..43af0ab3e 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_SQ8_SQ8.h @@ -45,10 +45,10 @@ float SQ8_SQ8_InnerProductSIMD64_NEON_DOTPROD_IMP(const void *pVec1v, const void // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. // - // Note this calls the helper directly rather than through a uint8 chooser, so it does not get - // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total - // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 - // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. + // Note this calls the helper directly rather than through a uint8 chooser, so it is not + // covered by the dispatcher bound that sends large dimensions to the scalar kernel. SQ8 is + // capped well below that bound by its uint32 q_sum_squares metadata slot, so the fence + // belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of vectors diff --git a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h index 504fd912c..cbc2a5497 100644 --- a/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_DOTPROD_UINT8.h @@ -8,20 +8,16 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM #include // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser -// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h -// carries the chunk-size argument. +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static because IP_NEON_UINT8.h defines the same name with a different body and // aarch64 gcc 12.3 outlines it, so shared linkage lets a NEON call site execute udot and fault // where asimddp is absent. always_inline keeps the plain wrapper's codegen unchanged. -// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against -// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks -// share one out-of-line copy of the kernel. // The inner product subtracts in integer and converts once, signed because the total is not. __attribute__((always_inline)) static inline void InnerProductOp(uint8x16_t &v1, uint8x16_t &v2, @@ -112,7 +108,7 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension uint32x4_t total_sum = vaddq_u32(sum0, sum1); - // ADDV, unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. + // ADDV, unsigned, and exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser enforces. return vaddvq_u32(total_sum); } @@ -131,41 +127,3 @@ float UINT8_CosineSIMD_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, si const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } - -__attribute__((noinline)) static uint32_t -UINT8_InnerProductFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, - size_t dimension) { - return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); -} - -template // 0..63 -struct UINT8_IPChunkKernel_NEON_DOTPROD { - static constexpr size_t granule() { return 64; } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductImp(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductFullChunk_NEON_DOTPROD(pVect1, pVect2, dimension); - } -}; - -template // 0..63 -float UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto ip = static_cast( - spaces::uint8_chunked_total>(pVect1v, pVect2v, - dimension)); - return static_cast(1 - ip); -} - -template // 0..63 -float UINT8_CosineSIMD_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - float ip = - static_cast(spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); - const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); - return 1.0f - ip / (norm_v1 * norm_v2); -} diff --git a/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h index 215d20e5e..1fe20c5c7 100644 --- a/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_NEON_SQ8_SQ8.h @@ -45,10 +45,10 @@ float SQ8_SQ8_InnerProductSIMD64_NEON_IMP(const void *pVec1v, const void *pVec2v // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. // - // Note this calls the helper directly rather than through a uint8 chooser, so it does not get - // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total - // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 - // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. + // Note this calls the helper directly rather than through a uint8 chooser, so it is not + // covered by the dispatcher bound that sends large dimensions to the scalar kernel. SQ8 is + // capped well below that bound by its uint32 q_sum_squares metadata slot, so the fence + // belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); // Get dequantization parameters and precomputed values from the end of pVec1 diff --git a/src/VecSim/spaces/IP/IP_NEON_UINT8.h b/src/VecSim/spaces/IP/IP_NEON_UINT8.h index d012e47e1..ce18f8cef 100644 --- a/src/VecSim/spaces/IP/IP_NEON_UINT8.h +++ b/src/VecSim/spaces/IP/IP_NEON_UINT8.h @@ -8,20 +8,16 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM #include // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser -// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h -// carries the chunk-size argument. +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static because IP_NEON_DOTPROD_UINT8.h defines the same name with a different body and // aarch64 gcc 12.3 outlines it, so shared linkage lets a NEON call site execute udot and fault // where asimddp is absent. always_inline keeps the plain wrapper's codegen unchanged. -// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against -// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks -// share one out-of-line copy of the kernel. // The inner product subtracts in integer and converts once, signed because the total is not. __attribute__((always_inline)) static inline void InnerProductOp(uint8x16_t &v1, uint8x16_t &v2, @@ -120,7 +116,7 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension uint32x4_t total_sum = vaddq_u32(sum0, sum1); - // ADDV, unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. + // ADDV, unsigned, and exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser enforces. return vaddvq_u32(total_sum); } @@ -138,38 +134,3 @@ float UINT8_CosineSIMD_NEON(const void *pVect1v, const void *pVect2v, size_t dim const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } - -__attribute__((noinline)) static uint32_t -UINT8_InnerProductFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductImp<0>(pVect1, pVect2, dimension); -} - -template // 0..63 -struct UINT8_IPChunkKernel_NEON { - static constexpr size_t granule() { return 64; } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductImp(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductFullChunk_NEON(pVect1, pVect2, dimension); - } -}; - -template // 0..63 -float UINT8_InnerProductSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto ip = - static_cast(spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - return static_cast(1 - ip); -} - -template // 0..63 -float UINT8_CosineSIMD_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = static_cast(spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); - const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); - return 1.0f - ip / (norm_v1 * norm_v2); -} diff --git a/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h b/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h index e13a88823..b1fb784b9 100644 --- a/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h +++ b/src/VecSim/spaces/IP/IP_SVE_SQ8_SQ8.h @@ -44,10 +44,10 @@ float SQ8_SQ8_InnerProductSIMD_SVE_IMP(const void *pVec1v, const void *pVec2v, s // the plain uint8 choosers, the SQ8_SQ8 choosers have no dimension guard, so the previous int // narrowed and wrapped past 33,025 and the float ones lost exactness past 258. // - // Note this calls the helper directly rather than through a uint8 chooser, so it does not get - // the chunked accumulation those choosers select past spaces::UINT8_CHUNK_ELEMENTS: the total - // here is still a single 32-bit reduce. SQ8 is capped well below that by its uint32 - // q_sum_squares metadata slot, so the fence belongs with SQ8 index creation (#1007), not here. + // Note this calls the helper directly rather than through a uint8 chooser, so it is not + // covered by the dispatcher bound that sends large dimensions to the scalar kernel. SQ8 is + // capped well below that bound by its uint32 q_sum_squares metadata slot, so the fence + // belongs with SQ8 index creation (#1007), not here. const uint32_t dot_product = UINT8_InnerProductImp(pVec1v, pVec2v, dimension); diff --git a/src/VecSim/spaces/IP/IP_SVE_UINT8.h b/src/VecSim/spaces/IP/IP_SVE_UINT8.h index 7dd963903..6425c928a 100644 --- a/src/VecSim/spaces/IP/IP_SVE_UINT8.h +++ b/src/VecSim/spaces/IP/IP_SVE_UINT8.h @@ -8,19 +8,15 @@ */ #pragma once #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM #include // uint8 inner product: Imp returns the raw integer total and the wrappers convert it. The chooser -// picks plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h -// carries the chunk-size argument. +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has // several callers. -// The chunked wrapper's first chunk keeps this instantiation's residual shape, clamped to the -// dimension; the vector length is a runtime value so that split is computed rather than folded. -// Later chunks share one out-of-line copy of the kernel. // The inner product subtracts in integer and converts once, signed because the total is not. inline void InnerProductStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t &offset, @@ -96,7 +92,8 @@ UINT8_InnerProductImp(const void *pVect1v, const void *pVect2v, size_t dimension sum0 = svadd_u32_x(svptrue_b32(), sum0, sum1); sum2 = svadd_u32_x(svptrue_b32(), sum2, sum3); - // Exact for up to spaces::UINT8_CHUNK_ELEMENTS elements; narrowed from svaddv_u32. + // Narrowed from svaddv_u32; exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser + // enforces. return static_cast(svaddv_u32(svptrue_b32(), svadd_u32_x(svptrue_b32(), sum0, sum2))); } @@ -115,39 +112,3 @@ float UINT8_CosineSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dime const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); return 1.0f - ip / (norm_v1 * norm_v2); } - -__attribute__((noinline)) static uint32_t -UINT8_InnerProductFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductImp(pVect1, pVect2, dimension); -} - -template -struct UINT8_IPChunkKernel_SVE { - static size_t granule() { return 4 * svcntb(); } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductImp(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_InnerProductFullChunk_SVE(pVect1, pVect2, dimension); - } -}; - -template -float UINT8_InnerProductSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto ip = static_cast( - spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - return static_cast(1 - ip); -} - -template -float UINT8_CosineSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - float ip = static_cast( - spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); - const float norm_v1 = load_unaligned(static_cast(pVect1v) + dimension); - const float norm_v2 = load_unaligned(static_cast(pVect2v) + dimension); - return 1.0f - ip / (norm_v1 * norm_v2); -} diff --git a/src/VecSim/spaces/IP_space.cpp b/src/VecSim/spaces/IP_space.cpp index 6706d8f31..9c1620a17 100644 --- a/src/VecSim/spaces/IP_space.cpp +++ b/src/VecSim/spaces/IP_space.cpp @@ -770,6 +770,13 @@ dist_func_t IP_UINT8_GetDistFunc(size_t dim, unsigned char *alignment, dist_func_t ret_dist_func = UINT8_InnerProduct; + // Above this dimension the SIMD kernels' 32-bit total is no longer exact, so hand back the + // scalar kernel, which accumulates into a 64-bit ret_t. Decided here, once per index, so no + // distance computation pays for the check. See spaces.h for how the bound is derived. + if (dim > spaces::UINT8_MAX_EXACT_SIMD_DIM) { + return ret_dist_func; + } + [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); #ifdef CPU_FEATURES_ARCH_AARCH64 @@ -818,6 +825,13 @@ dist_func_t Cosine_UINT8_GetDistFunc(size_t dim, unsigned char *alignment dist_func_t ret_dist_func = UINT8_Cosine; + // Above this dimension the SIMD kernels' 32-bit total is no longer exact, so hand back the + // scalar kernel, which accumulates into a 64-bit ret_t. Decided here, once per index, so no + // distance computation pays for the check. See spaces.h for how the bound is derived. + if (dim > spaces::UINT8_MAX_EXACT_SIMD_DIM) { + return ret_dist_func; + } + [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); #ifdef CPU_FEATURES_ARCH_AARCH64 diff --git a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h index 7bbfe8bee..128829838 100644 --- a/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h +++ b/src/VecSim/spaces/L2/L2_AVX512F_BW_VL_VNNI_UINT8.h @@ -7,18 +7,14 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM -// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks -// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries -// the chunk-size argument. +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has // several callers. -// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against -// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks -// share one out-of-line copy of the kernel. static inline void L2SqrStep(uint8_t *&pVect1, uint8_t *&pVect2, __m512i &sum) { __m512i va = _mm512_loadu_epi8(pVect1); // AVX512BW @@ -104,12 +100,13 @@ UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVect2v, size } while (pVect1 < pEnd1); } - // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. + // Unsigned, and exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser enforces. // Widening unsigned fold rather than _mm512_reduce_add_epi32. GCC implements that intrinsic as - // a chain of signed __v8si vector ops ending in a scalar `int + int`, and a chunk total reaches - // 65025 * 65536, about 4.26e9, which is roughly twice INT32_MAX. The wrapped bits are the ones - // we want, which is why equality tests pass, but the addition itself is signed-overflow UB and - // UBSan flags it. Zero-extending the 16 lanes to 64 bits first keeps every addition in range. + // a chain of signed __v8si vector ops ending in a scalar `int + int`, and the worst-case total + // at the dispatcher cap is 65025 * 66,051, or 4,294,966,275, which is almost exactly twice + // INT32_MAX. The wrapped bits are the ones we want, which is why equality tests pass, but the + // addition itself is signed-overflow UB and UBSan flags it. Zero-extending the 16 lanes to 64 + // bits first keeps every addition in range. const __m512i zero = _mm512_setzero_si512(); const __m512i widened = _mm512_add_epi64(_mm512_unpacklo_epi32(sum, zero), _mm512_unpackhi_epi32(sum, zero)); @@ -122,29 +119,3 @@ float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI(const void *pVect1v, const void *pVec return static_cast( UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1v, pVect2v, dimension)); } - -__attribute__((noinline)) static uint32_t -UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(const uint8_t *pVect1, const uint8_t *pVect2, - size_t dimension) { - return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI<0>(pVect1, pVect2, dimension); -} - -template // 0..63 -struct UINT8_L2ChunkKernel_AVX512F_BW_VL_VNNI { - static constexpr size_t granule() { return 64; } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrFullChunk_AVX512F_BW_VL_VNNI(pVect1, pVect2, dimension); - } -}; - -template // 0..63 -float UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - return static_cast( - spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); -} diff --git a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h index a0e5a9ebb..b6798f099 100644 --- a/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_DOTPROD_UINT8.h @@ -7,19 +7,15 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM #include -// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks -// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries -// the chunk-size argument. +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has // several callers. -// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against -// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks -// share one out-of-line copy of the kernel. __attribute__((always_inline)) static inline void L2SquareOp(const uint8x16_t &v1, const uint8x16_t &v2, uint32x4_t &sum) { @@ -134,7 +130,7 @@ UINT8_L2SqrImp_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dim total_sum = vaddq_u32(total_sum, sum2); total_sum = vaddq_u32(total_sum, sum3); - // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. + // Unsigned, and exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser enforces. return vaddvq_u32(total_sum); } @@ -142,28 +138,3 @@ template // 0..63 float UINT8_L2SqrSIMD16_NEON_DOTPROD(const void *pVect1v, const void *pVect2v, size_t dimension) { return static_cast(UINT8_L2SqrImp_NEON_DOTPROD(pVect1v, pVect2v, dimension)); } - -__attribute__((noinline)) static uint32_t -UINT8_L2SqrFullChunk_NEON_DOTPROD(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_NEON_DOTPROD<0>(pVect1, pVect2, dimension); -} - -template // 0..63 -struct UINT8_L2ChunkKernel_NEON_DOTPROD { - static constexpr size_t granule() { return 64; } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_NEON_DOTPROD(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrFullChunk_NEON_DOTPROD(pVect1, pVect2, dimension); - } -}; - -template // 0..63 -float UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked(const void *pVect1v, const void *pVect2v, - size_t dimension) { - return static_cast( - spaces::uint8_chunked_total>(pVect1v, pVect2v, - dimension)); -} diff --git a/src/VecSim/spaces/L2/L2_NEON_UINT8.h b/src/VecSim/spaces/L2/L2_NEON_UINT8.h index f1e461c18..60d0fb059 100644 --- a/src/VecSim/spaces/L2/L2_NEON_UINT8.h +++ b/src/VecSim/spaces/L2/L2_NEON_UINT8.h @@ -7,19 +7,15 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM #include -// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks -// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries -// the chunk-size argument. +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has // several callers. -// The chunked wrapper's first chunk absorbs the residual and its length is a runtime min against -// the dimension, because a compile-time trip count cost 8-9.5% in accumulator copies; later chunks -// share one out-of-line copy of the kernel. __attribute__((always_inline)) static inline void L2SquareOp(const uint8x16_t &v1, const uint8x16_t &v2, uint32x4_t &sum) { @@ -138,7 +134,7 @@ UINT8_L2SqrImp_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) total_sum = vaddq_u32(total_sum, sum2); total_sum = vaddq_u32(total_sum, sum3); - // Unsigned, and exact for up to spaces::UINT8_CHUNK_ELEMENTS elements. + // Unsigned, and exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser enforces. return vaddvq_u32(total_sum); } @@ -146,26 +142,3 @@ template // 0..63 float UINT8_L2SqrSIMD16_NEON(const void *pVect1v, const void *pVect2v, size_t dimension) { return static_cast(UINT8_L2SqrImp_NEON(pVect1v, pVect2v, dimension)); } - -__attribute__((noinline)) static uint32_t -UINT8_L2SqrFullChunk_NEON(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_NEON<0>(pVect1, pVect2, dimension); -} - -template // 0..63 -struct UINT8_L2ChunkKernel_NEON { - static constexpr size_t granule() { return 64; } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_NEON(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrFullChunk_NEON(pVect1, pVect2, dimension); - } -}; - -template // 0..63 -float UINT8_L2SqrSIMD16_NEON_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return static_cast(spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); -} diff --git a/src/VecSim/spaces/L2/L2_SVE_UINT8.h b/src/VecSim/spaces/L2/L2_SVE_UINT8.h index 8f05a7253..09d820c49 100644 --- a/src/VecSim/spaces/L2/L2_SVE_UINT8.h +++ b/src/VecSim/spaces/L2/L2_SVE_UINT8.h @@ -7,19 +7,15 @@ * GNU Affero General Public License v3 (AGPLv3). */ #include "VecSim/spaces/space_includes.h" -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS -#include "VecSim/spaces/uint8_chunking.h" +#include "VecSim/spaces/spaces.h" // spaces::UINT8_MAX_EXACT_SIMD_DIM #include -// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser picks -// plain up to spaces::UINT8_CHUNK_ELEMENTS and chunked above it, once per index; spaces.h carries -// the chunk-size argument. +// uint8 L2: Imp returns the raw integer total and the wrappers convert it. The chooser +// hands back the scalar kernel above spaces::UINT8_MAX_EXACT_SIMD_DIM, where a 32-bit total is +// no longer exact; spaces.h carries that bound and its derivation. // // Imp is static and always_inline so the plain wrapper's codegen is unchanged now that Imp has // several callers. -// The chunked wrapper's first chunk keeps this instantiation's residual shape, clamped to the -// dimension; the vector length is a runtime value so that split is computed rather than folded. -// Later chunks share one out-of-line copy of the kernel. // Aligned step using svptrue_b8() inline void L2SquareStep(const uint8_t *&pVect1, const uint8_t *&pVect2, size_t &offset, @@ -98,7 +94,8 @@ UINT8_L2SqrImp_SVE(const void *pVect1v, const void *pVect2v, size_t dimension) { sum0 = svadd_u32_x(all, sum0, sum1); sum2 = svadd_u32_x(all, sum2, sum3); svuint32_t sum_all = svadd_u32_x(all, sum0, sum2); - // Exact for up to spaces::UINT8_CHUNK_ELEMENTS elements; narrowed from svaddv_u32. + // Narrowed from svaddv_u32; exact up to spaces::UINT8_MAX_EXACT_SIMD_DIM, which the chooser + // enforces. return static_cast(svaddv_u32(svptrue_b32(), sum_all)); } @@ -107,27 +104,3 @@ float UINT8_L2SqrSIMD_SVE(const void *pVect1v, const void *pVect2v, size_t dimen return static_cast( UINT8_L2SqrImp_SVE(pVect1v, pVect2v, dimension)); } - -__attribute__((noinline)) static uint32_t -UINT8_L2SqrFullChunk_SVE(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_SVE(pVect1, pVect2, dimension); -} - -template -struct UINT8_L2ChunkKernel_SVE { - static size_t granule() { return 4 * svcntb(); } - __attribute__((always_inline)) static inline uint32_t - first(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrImp_SVE(pVect1, pVect2, dimension); - } - static uint32_t rest(const uint8_t *pVect1, const uint8_t *pVect2, size_t dimension) { - return UINT8_L2SqrFullChunk_SVE(pVect1, pVect2, dimension); - } -}; - -template -float UINT8_L2SqrSIMD_SVE_Chunked(const void *pVect1v, const void *pVect2v, size_t dimension) { - return static_cast( - spaces::uint8_chunked_total>( - pVect1v, pVect2v, dimension)); -} diff --git a/src/VecSim/spaces/L2_space.cpp b/src/VecSim/spaces/L2_space.cpp index b4ca3669a..5d23193a1 100644 --- a/src/VecSim/spaces/L2_space.cpp +++ b/src/VecSim/spaces/L2_space.cpp @@ -468,6 +468,13 @@ dist_func_t L2_UINT8_GetDistFunc(size_t dim, unsigned char *alignment, dist_func_t ret_dist_func = UINT8_L2Sqr; + // Above this dimension the SIMD kernels' 32-bit total is no longer exact, so hand back the + // scalar kernel, which accumulates into a 64-bit ret_t. Decided here, once per index, so no + // distance computation pays for the check. See spaces.h for how the bound is derived. + if (dim > spaces::UINT8_MAX_EXACT_SIMD_DIM) { + return ret_dist_func; + } + // Optimizations assume at least 32 uint8. If we have less, we use the naive implementation. [[maybe_unused]] auto features = getCpuOptimizationFeatures(arch_opt); diff --git a/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp b/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp index c9f73c9ed..97da55546 100644 --- a/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp +++ b/src/VecSim/spaces/functions/AVX512F_BW_VL_VNNI.cpp @@ -42,43 +42,21 @@ dist_func_t Choose_INT8_Cosine_implementation_AVX512F_BW_VL_VNNI(size_t d return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays what it was. dist_func_t Choose_UINT8_L2_implementation_AVX512F_BW_VL_VNNI(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD64_AVX512F_BW_VL_VNNI); return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant, which folds each chunk's -// exact 32-bit total into 64 bits. Chosen here, once per index, so the plain kernel below carries -// no branch and stays what it was. dist_func_t Choose_UINT8_IP_implementation_AVX512F_BW_VL_VNNI(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, - UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD64_AVX512F_BW_VL_VNNI); return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant, which folds each chunk's -// exact 32-bit total into 64 bits. Chosen here, once per index, so the plain kernel below carries -// no branch and stays what it was. dist_func_t Choose_UINT8_Cosine_implementation_AVX512F_BW_VL_VNNI(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, - UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD64_AVX512F_BW_VL_VNNI); return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/NEON.cpp b/src/VecSim/spaces/functions/NEON.cpp index d50bc28d5..0c9a286e3 100644 --- a/src/VecSim/spaces/functions/NEON.cpp +++ b/src/VecSim/spaces/functions/NEON.cpp @@ -30,15 +30,9 @@ dist_func_t Choose_INT8_IP_implementation_NEON(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON); return ret_dist_func; } @@ -62,11 +56,7 @@ dist_func_t Choose_INT8_Cosine_implementation_NEON(size_t dim) { dist_func_t Choose_UINT8_Cosine_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON); return ret_dist_func; } @@ -83,11 +73,7 @@ dist_func_t Choose_INT8_L2_implementation_NEON(size_t dim) { dist_func_t Choose_UINT8_L2_implementation_NEON(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON); return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/NEON_DOTPROD.cpp b/src/VecSim/spaces/functions/NEON_DOTPROD.cpp index bcf5b8d59..12f762093 100644 --- a/src/VecSim/spaces/functions/NEON_DOTPROD.cpp +++ b/src/VecSim/spaces/functions/NEON_DOTPROD.cpp @@ -24,16 +24,9 @@ dist_func_t Choose_INT8_IP_implementation_NEON_DOTPROD(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_IP_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, - UINT8_InnerProductSIMD16_NEON_DOTPROD_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON_DOTPROD); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_InnerProductSIMD16_NEON_DOTPROD); return ret_dist_func; } @@ -45,11 +38,7 @@ dist_func_t Choose_INT8_Cosine_implementation_NEON_DOTPROD(size_t dim) { dist_func_t Choose_UINT8_Cosine_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_DOTPROD_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_DOTPROD); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_CosineSIMD_NEON_DOTPROD); return ret_dist_func; } @@ -61,11 +50,7 @@ dist_func_t Choose_INT8_L2_implementation_NEON_DOTPROD(size_t dim) { dist_func_t Choose_UINT8_L2_implementation_NEON_DOTPROD(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_DOTPROD_Chunked); - } else { - CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_DOTPROD); - } + CHOOSE_IMPLEMENTATION(ret_dist_func, dim, 64, UINT8_L2SqrSIMD16_NEON_DOTPROD); return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/SVE.cpp b/src/VecSim/spaces/functions/SVE.cpp index 5f8c1b625..bd197c84c 100644 --- a/src/VecSim/spaces/functions/SVE.cpp +++ b/src/VecSim/spaces/functions/SVE.cpp @@ -86,35 +86,21 @@ dist_func_t Choose_INT8_Cosine_implementation_SVE(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE_Chunked, dim, svcntb); - } else { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); - } + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); return ret_dist_func; } dist_func_t Choose_UINT8_IP_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE_Chunked, dim, svcntb); - } else { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); - } + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); return ret_dist_func; } dist_func_t Choose_UINT8_Cosine_implementation_SVE(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE_Chunked, dim, svcntb); - } else { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); - } + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); return ret_dist_func; } diff --git a/src/VecSim/spaces/functions/SVE2.cpp b/src/VecSim/spaces/functions/SVE2.cpp index 7c1a662ab..9eea81523 100644 --- a/src/VecSim/spaces/functions/SVE2.cpp +++ b/src/VecSim/spaces/functions/SVE2.cpp @@ -82,35 +82,21 @@ dist_func_t Choose_INT8_Cosine_implementation_SVE2(size_t dim) { return ret_dist_func; } -// Dimensions past spaces::UINT8_CHUNK_ELEMENTS use the chunked variant. Chosen here, once per -// index, so the plain kernel carries no branch and stays exactly what it was. dist_func_t Choose_UINT8_L2_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE_Chunked, dim, svcntb); - } else { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); - } + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_L2SqrSIMD_SVE, dim, svcntb); return ret_dist_func; } dist_func_t Choose_UINT8_IP_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE_Chunked, dim, svcntb); - } else { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); - } + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_InnerProductSIMD_SVE, dim, svcntb); return ret_dist_func; } dist_func_t Choose_UINT8_Cosine_implementation_SVE2(size_t dim) { dist_func_t ret_dist_func; - if (dim > spaces::UINT8_CHUNK_ELEMENTS) { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE_Chunked, dim, svcntb); - } else { - CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); - } + CHOOSE_SVE_IMPLEMENTATION(ret_dist_func, UINT8_CosineSIMD_SVE, dim, svcntb); return ret_dist_func; } diff --git a/src/VecSim/spaces/normalize/compute_norm.h b/src/VecSim/spaces/normalize/compute_norm.h index dd3bf68a9..665080331 100644 --- a/src/VecSim/spaces/normalize/compute_norm.h +++ b/src/VecSim/spaces/normalize/compute_norm.h @@ -22,8 +22,9 @@ static inline float IntegralType_ComputeNorm(const DataType *vec, const size_t d // from dimension 33,026 and the accumulation was signed-overflow UB there. This is the norm the // cosine preprocessor writes for every stored vector and every query, so a wrong value here // reaches the kernels before they run: at dimension 65,537 the norm came out NaN, and at 66,052 - // it came out 252.99 instead of about 65,536.49. int8 has the same shape with a higher bound, - // 16,129 per term, so it overflowed from dimension 133,153. + // it came out 252.99 instead of about 65,536.49. int8 has the same shape: its largest + // squared term is (-128)^2 = 16,384, not 127^2, so the signed total overflowed from + // dimension 131,072. uint64_t sum = 0; for (size_t i = 0; i < dim; i++) { diff --git a/src/VecSim/spaces/spaces.h b/src/VecSim/spaces/spaces.h index 6419f9772..45e7f4b3a 100644 --- a/src/VecSim/spaces/spaces.h +++ b/src/VecSim/spaces/spaces.h @@ -12,6 +12,7 @@ #include "space_includes.h" #include +#include namespace spaces { @@ -52,24 +53,19 @@ static int inline is_little_endian() { return *(char *)&x; } -// A full-range uint8 product is at most 255 * 255 = 65,025, so a 32-bit accumulator is exact -// through floor(UINT32_MAX / 65,025) = 66,051 terms. Twice the old signed limit of 33,025: the -// accumulation was always fine, the top bit was being read as a sign. +// The uint8 kernels accumulate products or squared byte differences, so the worst-case total is +// 255 * 255 * dim. A 32-bit accumulator holds that exactly up to this dimension and no further, so +// the choosers hand back the scalar kernel above it, which accumulates into a 64-bit ret_t and is +// exact at any dimension. Derived from the types rather than written as a literal: the bound is a +// property of uint8 in a uint32 accumulator, and at the limit there are only 1,020 to spare. // -// Rather than cap the dimension there, the kernels accumulate in chunks of this many elements and -// fold each chunk's exact 32-bit total into a 64-bit scalar, which makes them exact at any -// dimension. 65,536 is chosen because it is under 66,051, so the existing 32-bit reduce needs no -// change, and because it is a whole number of 64-byte blocks, so a chunk boundary always lands on -// one. -// -// The margins are deliberately loose, because the tight version was wrong. A previous attempt -// widened the reduce and bounded the dimension at 4 * 66,051, on the assumption that products -// spread evenly across NEON's four lanes after its 32-bit vaddq_u32 merge. The even case already -// sat within 1,020 of UINT32_MAX while a masked residual load can put 1,040,400 into a single lane, -// so lanes wrapped before the widened reduce saw them. At this chunk size the per-chunk total has -// 33 million to spare and a NEON lane has 3.2 billion, so neither constraint is close and no -// per-ISA lane audit is needed. -static constexpr size_t UINT8_CHUNK_ELEMENTS = 65536; +// Note this is the UNSIGNED bound. The total passes INT32_MAX from dimension 33,026, so a signed +// read of the reduce wraps negative well inside the range kept on SIMD, which is why the kernels +// return uint32_t and why the AVX512 reduce folds through 64 bits: GCC implements +// _mm512_reduce_add_epi32 as signed vector ops ending in a scalar int + int. +static constexpr size_t UINT8_MAX_EXACT_SIMD_DIM = + std::numeric_limits::max() / + (std::numeric_limits::max() * std::numeric_limits::max()); static inline auto getCpuOptimizationFeatures(const void *arch_opt = nullptr) { diff --git a/src/VecSim/spaces/uint8_chunking.h b/src/VecSim/spaces/uint8_chunking.h deleted file mode 100644 index cd3ea3c20..000000000 --- a/src/VecSim/spaces/uint8_chunking.h +++ /dev/null @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2006-Present, Redis Ltd. - * All rights reserved. - * - * Licensed under your choice of the Redis Source Available License 2.0 - * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the - * GNU Affero General Public License v3 (AGPLv3). - */ -#pragma once - -// Shared chunked-accumulation driver for the uint8 SIMD kernels. It splits a distance -// computation into chunks of at most UINT8_CHUNK_ELEMENTS elements so each chunk's 32-bit -// SIMD total stays exact, and folds the per-chunk totals into a 64-bit scalar. The same -// formula serves both fixed-width kernels (granule 64) and SVE (granule 4 * svcntb()); only -// the granule differs. The caller supplies a Kernel adapter with: -// static size_t granule() - the kernel's block size -// static uint32_t first(const uint8_t *, const uint8_t *, size_t) - the residual-bearing -// kernel, shape already -// bound -// static uint32_t rest(const uint8_t *, const uint8_t *, size_t) - the out-of-line -// residual-0 kernel -// Invariants below hold for any Kernel whose granule() is in (0, UINT8_CHUNK_ELEMENTS]; all -// current adapters return 64 (fixed-width) or 4 * svcntb() (SVE, 64 to 1024 for a 16 to 256 byte -// vector length), so both stay within that range. Given that precondition: first <= dimension -// always, so this is correct at any dimension, including ones below the chunk size (the loop -// then does not execute). first is congruent to dimension modulo granule, so Kernel::first's -// residual shape still describes it. remaining is therefore a whole multiple of granule, and so -// is every step, which is Kernel::rest's precondition. No single call ever gets more than -// UINT8_CHUNK_ELEMENTS elements, which is what keeps each chunk's 32-bit total exact. - -#include "VecSim/spaces/spaces.h" // spaces::UINT8_CHUNK_ELEMENTS - -#include -#include -#include -#include - -namespace spaces { - -template -static inline uint64_t uint8_chunked_total(const void *pVect1v, const void *pVect2v, - size_t dimension) { - const auto *pVect1 = static_cast(pVect1v); - const auto *pVect2 = static_cast(pVect2v); - - constexpr size_t chunk = UINT8_CHUNK_ELEMENTS; - // Enforce the granule precondition at compile time when the adapter can express it, which is - // every fixed-width kernel. A plain assert would vanish under NDEBUG, so it is the fallback - // only for SVE, whose granule depends on the runtime vector length and cannot be constant. - if constexpr (requires { std::integral_constant{}; }) { - static_assert(Kernel::granule() > 0 && Kernel::granule() <= UINT8_CHUNK_ELEMENTS, - "Kernel::granule() must be in (0, UINT8_CHUNK_ELEMENTS]"); - } - // SVE is the only adapter whose granule cannot be constant, so it keeps the runtime assert and - // is unprotected under NDEBUG. That is acceptable because the architecture bounds it: an SVE - // vector is 16 to 256 bytes, so 4 * svcntb() is 64 to 1024, three orders of magnitude below the - // 65,536 limit. Only a change to that multiplier, or to the chunk size, could approach it. - const size_t granule = Kernel::granule(); - assert(granule > 0 && granule <= chunk); - const size_t tail = dimension % granule; - const size_t first_chunk = tail + ((chunk - tail) / granule) * granule; - const size_t first = dimension < first_chunk ? dimension : first_chunk; - const size_t max_step = (chunk / granule) * granule; - - uint64_t total = Kernel::first(pVect1, pVect2, first); - pVect1 += first; - pVect2 += first; - size_t remaining = dimension - first; - - while (remaining) { - const size_t step = remaining < max_step ? remaining : max_step; - total += Kernel::rest(pVect1, pVect2, step); - pVect1 += step; - pVect2 += step; - remaining -= step; - } - return total; -} - -} // namespace spaces diff --git a/tests/unit/test_common.cpp b/tests/unit/test_common.cpp index b5790a9a1..9b1001009 100644 --- a/tests/unit/test_common.cpp +++ b/tests/unit/test_common.cpp @@ -755,59 +755,6 @@ TEST(CommonAPITest, NormalizeUint8) { ASSERT_FLOAT_EQ(norm, 1.0); } -// The norm the cosine preprocessor writes goes through IntegralType_ComputeNorm, which accumulated -// into a signed int. Each uint8 element contributes up to 255*255 = 65,025, so the total passes -// INT32_MAX from dimension 33,026: at 65,537 the norm came back NaN, and at 66,052 it came back -// 252.99 instead of about 65,536.49. Every stored vector and every query for a uint8 cosine index -// takes this path, so a wrong norm reaches the distance kernels before they run, which no amount of -// exactness in the kernels can repair. -// -// This goes through the public API rather than calling the norm helper, because the kernel tests -// append the norm by hand and so never exercised this at all. -TEST(CommonAPITest, NormalizeUint8LargeDimension) { - // 33,025 is the last dimension whose worst-case total fits a signed int; the rest are past it. - for (const size_t dim : {size_t{33025}, size_t{33026}, size_t{65537}, size_t{66052}}) { - std::vector v(dim + sizeof(float), 255); - - VecSim_Normalize(v.data(), dim, VecSimType_UINT8); - - float res_norm; - memcpy(&res_norm, v.data() + dim, sizeof(res_norm)); - const double expected = std::sqrt(255.0 * 255.0 * static_cast(dim)); - - ASSERT_TRUE(std::isfinite(res_norm)) << "norm is not finite at dim " << dim; - ASSERT_GT(res_norm, 0.0f) << "norm is not positive at dim " << dim; - EXPECT_NEAR(res_norm, expected, expected * 1e-5) - << "norm at dim " << dim << " is " << res_norm << ", expected about " << expected; - } -} - -// The same path as seen by an index: add an all-255 vector to a uint8 cosine brute force index at a -// dimension past the old overflow point, and query it with itself. Cosine self-distance must be -// about zero, which it cannot be if the stored or query norm is NaN or wildly wrong. -TEST(CommonAPITest, Uint8CosineSelfDistanceAtLargeDimension) { - constexpr size_t dim = 65537; - BFParams params = { - .type = VecSimType_UINT8, .dim = dim, .metric = VecSimMetric_Cosine, .initialCapacity = 2}; - VecSimIndex *index = test_utils::CreateNewIndex(params, VecSimType_UINT8); - ASSERT_NE(index, nullptr); - - std::vector v(dim + sizeof(float), 255); - VecSimIndex_AddVector(index, v.data(), 0); - ASSERT_EQ(VecSimIndex_IndexSize(index), 1); - - auto *res = VecSimIndex_TopKQuery(index, v.data(), 1, nullptr, BY_SCORE); - ASSERT_EQ(VecSimQueryReply_Len(res), 1); - auto it = VecSimQueryReply_GetIterator(res); - auto *item = VecSimQueryReply_IteratorNext(it); - const double score = VecSimQueryResult_GetScore(item); - EXPECT_TRUE(std::isfinite(score)) << "cosine self-distance is not finite: " << score; - EXPECT_NEAR(score, 0.0, 1e-5) << "cosine self-distance should be about zero, got " << score; - VecSimQueryReply_IteratorFree(it); - VecSimQueryReply_Free(res); - VecSimIndex_Free(index); -} - // The norm the cosine preprocessor writes goes through IntegralType_ComputeNorm, which accumulated // into a signed int. Each uint8 element contributes up to 255*255 = 65,025, so the total passes // INT32_MAX from dimension 33,026: at 65,537 the norm came back NaN, and at 66,052 it came back diff --git a/tests/unit/test_spaces.cpp b/tests/unit/test_spaces.cpp index ba8eb5776..e6933bf8d 100644 --- a/tests/unit/test_spaces.cpp +++ b/tests/unit/test_spaces.cpp @@ -51,7 +51,6 @@ #include "VecSim/spaces/functions/SVE.h" #include "VecSim/spaces/functions/SVE_BF16.h" #include "VecSim/spaces/functions/SVE2.h" -#include "VecSim/spaces/uint8_chunking.h" #include "tests_utils.h" using bfloat16 = vecsim_types::bfloat16; @@ -2228,97 +2227,11 @@ TEST_F(SpacesTest, UINT8_L2Sqr_and_InnerProduct_are_exact_past_int32) { } } -// Past spaces::UINT8_CHUNK_ELEMENTS the uint8 SIMD kernels accumulate in chunks: each chunk's total -// still fits the 32-bit accumulators (65025 * 65536 <= UINT32_MAX), and the per-chunk totals are -// folded in 64 bits. The dispatched kernel must therefore agree exactly with the scalar kernel, -// which accumulates the whole vector into a 64-bit ret_t. Exact equality is the right assertion -// because both paths convert the same integer total to float once, at the end. +// Both dimensions sit at or below spaces::UINT8_MAX_EXACT_SIMD_DIM, so both stay on SIMD, and both +// totals exceed INT32_MAX, which is where reading the reduce as signed used to wrap them negative. +// The dispatched kernel must agree with the scalar kernel, which accumulates the whole vector into +// a 64-bit ret_t. // -// All-255 against all-0 is the worst case and puts the L2 total past UINT32_MAX from dimension -// 66,052, so the multi-chunk dimensions below genuinely exercise the 64-bit fold. The existing -// UINT8 suites stop at dim 128, which is why the wrap went unseen. -TEST_F(SpacesTest, UINT8_dispatched_kernels_are_exact_across_the_chunk_boundary) { - // Below the boundary, on it, one past it (whose last chunk is a single 64-element block), an - // exact multiple of it, and dimensions spanning two and three chunks. - for (const size_t dim : {65535UL, 65536UL, 65537UL, 65600UL, 131072UL, 131109UL, 200000UL}) { - // The cosine kernels read a float norm from just past the payload, so size for both. - std::vector ones(dim + sizeof(float), 255); - std::vector zeros(dim + sizeof(float), 0); - std::vector ramp(dim + sizeof(float)); - for (size_t i = 0; i < dim; i++) { - ramp[i] = static_cast(i % 256); - } - const float norm = std::sqrt(255.0f * 255.0f * static_cast(dim)); - memcpy(ones.data() + dim, &norm, sizeof(float)); - memcpy(ramp.data() + dim, &norm, sizeof(float)); - - unsigned char alignment = 0; - auto l2 = L2_UINT8_GetDistFunc(dim, &alignment, nullptr); - auto ip = IP_UINT8_GetDistFunc(dim, &alignment, nullptr); - auto cosine = Cosine_UINT8_GetDistFunc(dim, &alignment, nullptr); - - // Worst case: the largest total the byte range allows. - EXPECT_EQ(UINT8_L2Sqr(ones.data(), zeros.data(), dim), l2(ones.data(), zeros.data(), dim)) - << "L2 all-255 vs all-0, dim " << dim; - EXPECT_EQ(UINT8_InnerProduct(ones.data(), ones.data(), dim), - ip(ones.data(), ones.data(), dim)) - << "IP all-255, dim " << dim; - EXPECT_EQ(UINT8_Cosine(ones.data(), ones.data(), dim), - cosine(ones.data(), ones.data(), dim)) - << "Cosine all-255, dim " << dim; - - // A varying pattern, so the residual and chunk seams have to line up element for element - // rather than merely produce the right sum of identical values. - EXPECT_EQ(UINT8_L2Sqr(ramp.data(), ones.data(), dim), l2(ramp.data(), ones.data(), dim)) - << "L2 ramp vs all-255, dim " << dim; - EXPECT_EQ(UINT8_InnerProduct(ramp.data(), ones.data(), dim), - ip(ramp.data(), ones.data(), dim)) - << "IP ramp vs all-255, dim " << dim; - EXPECT_EQ(UINT8_Cosine(ramp.data(), ones.data(), dim), - cosine(ramp.data(), ones.data(), dim)) - << "Cosine ramp vs all-255, dim " << dim; - } -} - -// The boundary test above samples dimensions; this sweeps every residual instantiation. 65,600 and -// 196,608 are both multiples of 64, so base + r has residual r: one chunk past the boundary, then -// three chunks past it, so the seam between the residual-bearing first chunk and the residual-0 -// chunks after it is exercised for all 64 shapes. A ramp against all-255 is position sensitive, so -// a seam that double-counts or skips elements changes the total rather than cancelling out, and the -// total still passes UINT32_MAX (about 32,500 * dim) so the 64-bit fold is under test throughout. -TEST_F(SpacesTest, UINT8_dispatched_kernels_are_exact_at_every_residual_past_the_chunk_boundary) { - constexpr size_t max_dim = 196608 + 63; - std::vector ones(max_dim + sizeof(float), 255); - std::vector ramp(max_dim + sizeof(float)); - for (size_t i = 0; i < max_dim; i++) { - ramp[i] = static_cast(i % 256); - } - - for (const size_t base : {65600UL, 196608UL}) { - for (size_t r = 0; r < 64; r++) { - const size_t dim = base + r; - // The cosine kernels read a float norm from just past the payload, which moves with - // dim. - const float norm = std::sqrt(255.0f * 255.0f * static_cast(dim)); - memcpy(ones.data() + dim, &norm, sizeof(float)); - memcpy(ramp.data() + dim, &norm, sizeof(float)); - - unsigned char alignment = 0; - const void *a = ramp.data(); - const void *b = ones.data(); - - EXPECT_EQ(UINT8_L2Sqr(a, b, dim), - L2_UINT8_GetDistFunc(dim, &alignment, nullptr)(a, b, dim)) - << "L2 at dim " << dim << " (residual " << r << ")"; - EXPECT_EQ(UINT8_InnerProduct(a, b, dim), - IP_UINT8_GetDistFunc(dim, &alignment, nullptr)(a, b, dim)) - << "IP at dim " << dim << " (residual " << r << ")"; - EXPECT_EQ(UINT8_Cosine(a, b, dim), - Cosine_UINT8_GetDistFunc(dim, &alignment, nullptr)(a, b, dim)) - << "Cosine at dim " << dim << " (residual " << r << ")"; - } - } -} // Every uint8 SIMD tier this host can actually execute, with its three dispatched kernels. Both // the per-tier exactness test and the independent-oracle test below iterate this list, so a tier @@ -2375,25 +2288,26 @@ static std::vector AvailableUInt8Tiers(size_t dim) { // Worst-case inputs against an oracle computed here in 64-bit integers, rather than by calling the // scalar kernel. // -// Why not the scalar kernel: scalar and SIMD share conventions, and this series changed the scalar -// and SIMD inner product epilogues together so they would stay bit-identical. A test asserting only -// scalar == SIMD cannot catch that shared convention being wrong, in either sign or width. Here the -// expectation is derived from the inputs alone, and the scalar kernel is asserted against it on the -// same footing as every SIMD tier. +// Why not the scalar kernel: scalar and SIMD share conventions, and this PR changed the scalar and +// SIMD inner product epilogues together so they would stay bit-identical. A test asserting only +// scalar == SIMD cannot catch that shared convention being wrong, in sign or in width. Here the +// expectation comes from the inputs alone, and the scalar kernel is asserted against it on the same +// footing as every SIMD tier. // // Why these inputs: all-255 against all-255 puts 65,025 into the inner product accumulator for // every element, and all-255 against all-0 does the same for L2. Those are the maxima the byte -// range allows, so they are where a 32-bit accumulator wraps first. A ramp against all-255 is -// carried alongside because constant data lets a gap and an overlap of equal size cancel, which a -// position-dependent pattern does not. +// range allows. A ramp against all-255 is carried alongside because constant data lets a gap and an +// overlap of equal size cancel. // -// Why these dimensions: 65024 is 1016*64, so 65024+r has residual r and stays at or below the -// 65,536 chunk size, exercising the plain kernel right up against the limit of its 32-bit reduce -// (65025 * 65087 is about 4.23e9, just under UINT32_MAX). 131072 is 2048*64, so 131072+r has -// residual r and its total is about 8.5e9, which only a 64-bit fold can carry. Every residual is -// swept at both. +// Why these dimensions: both bases are multiples of 64, so base + r has residual r and every +// residual is swept at both. 33,024 + r spans the signed boundary: at 33,025 the total is +// 2,147,450,625, the largest that fits INT32_MAX, and at 33,026 it is 2,147,515,650, which does +// not. Both stay on SIMD, so 33,026 is what exercises the widened AVX-512 fold, since GCC's +// _mm512_reduce_add_epi32 ends in a scalar int + int. 65,984 + r sits just under the dispatcher cap +// of 66,051, where the total nears 4.29e9 and is still exact in uint32 with about a thousand to +// spare. TEST_F(SpacesTest, UINT8_worst_case_matches_an_independent_64bit_oracle) { - constexpr size_t max_dim = 131072 + 63; + constexpr size_t max_dim = 65984 + 63; std::vector ones(max_dim + sizeof(float), 255); std::vector zeros(max_dim + sizeof(float), 0); std::vector ramp(max_dim + sizeof(float)); @@ -2401,7 +2315,12 @@ TEST_F(SpacesTest, UINT8_worst_case_matches_an_independent_64bit_oracle) { ramp[i] = static_cast(i % 256); } - for (const size_t base : {65024UL, 131072UL}) { + // Which tiers this run actually reached. The loop over tiers below executes zero times on a + // host without a uint8 SIMD tier, leaving only the scalar assertions, so record it rather than + // letting a narrower run look like a full one. + std::set all_tiers; + + for (const size_t base : {33024UL, 65984UL}) { for (size_t r = 0; r < 64; r++) { const size_t dim = base + r; SCOPED_TRACE("dim " + std::to_string(dim) + " residual " + std::to_string(r)); @@ -2441,13 +2360,15 @@ TEST_F(SpacesTest, UINT8_worst_case_matches_an_independent_64bit_oracle) { const int64_t diff = static_cast(x) - static_cast(y); l2_total += static_cast(diff * diff); } - // Both worst-case pairs must exceed a 32-bit accumulator at the multi-chunk base, - // otherwise this test would not be reaching the case it exists for. - if (base == 131072 && pr.b != ramp.data() && pr.a != ramp.data()) { - EXPECT_GT(std::max(ip_total, l2_total), - static_cast(std::numeric_limits::max())) - << "worst case no longer exceeds UINT32_MAX, test is not exercising the " - "fold"; + // At the upper base the worst case must exceed INT32_MAX, or the signed fold is + // untested, and must stay within UINT32_MAX, or the dispatcher would have handed + // this dimension to scalar and these assertions would not be testing SIMD at all. + if (base == 65984 && pr.a != ramp.data() && pr.b != ramp.data()) { + const uint64_t worst = std::max(ip_total, l2_total); + EXPECT_GT(worst, static_cast(std::numeric_limits::max())) + << "worst case no longer exceeds INT32_MAX, the signed fold is untested"; + EXPECT_LE(worst, static_cast(std::numeric_limits::max())) + << "worst case exceeds UINT32_MAX, this dimension should not be on SIMD"; } // Expected returns, formed with the same operations the kernels use so the @@ -2465,6 +2386,7 @@ TEST_F(SpacesTest, UINT8_worst_case_matches_an_independent_64bit_oracle) { } for (const auto &tier : AvailableUInt8Tiers(dim)) { + all_tiers.insert(tier.name); EXPECT_EQ(want_l2, tier.l2(pr.a, pr.b, dim)) << "L2 " << tier.name << ", " << pr.name; EXPECT_EQ(want_ip, tier.ip(pr.a, pr.b, dim)) @@ -2477,19 +2399,79 @@ TEST_F(SpacesTest, UINT8_worst_case_matches_an_independent_64bit_oracle) { } } } + std::string covered; + for (const auto &t : all_tiers) { + covered += covered.empty() ? t : ", " + t; + } + std::cout << " oracle covered tiers: " << (covered.empty() ? "" : covered) + << std::endl; + RecordProperty("oracle_tiers", covered); + + // Same gate as the per-tier test: a hardware run that names the tier it exists to cover must + // fail, not quietly pass with scalar-only coverage. + const char *required = std::getenv("VECSIM_REQUIRE_UINT8_TIER"); + if (required != nullptr && *required != '\0') { + EXPECT_TRUE(all_tiers.count(required) > 0) + << "VECSIM_REQUIRE_UINT8_TIER=" << required << " but the oracle test reached " + << all_tiers.size() << " tier(s), so it proves nothing about " << required; + } +} + +// The dispatcher boundary. 66,051 is the last dimension whose worst-case total fits a uint32 +// accumulator, so it stays on SIMD; 66,052 is the first that does not, so it must come back as the +// scalar kernel, which accumulates into a 64-bit ret_t. +// +// Asserting the returned pointer is the point here. On a host with no uint8 SIMD tier both sides +// are the scalar function and the value comparisons below would pass either way, so the pointer +// identity is the only thing that distinguishes the two cases, and it is skipped where it would be +// vacuous. +TEST_F(SpacesTest, UINT8_dispatcher_falls_back_to_scalar_above_the_exact_dim) { + unsigned char alignment = 0; + constexpr size_t last_simd = 66051; + constexpr size_t first_scalar = 66052; + static_assert(last_simd == spaces::UINT8_MAX_EXACT_SIMD_DIM, + "this test pins the documented bound, update both together"); + + // Above the bound, every metric must hand back the scalar kernel by name. + EXPECT_EQ(L2_UINT8_GetDistFunc(first_scalar, &alignment, nullptr), UINT8_L2Sqr); + EXPECT_EQ(IP_UINT8_GetDistFunc(first_scalar, &alignment, nullptr), UINT8_InnerProduct); + EXPECT_EQ(Cosine_UINT8_GetDistFunc(first_scalar, &alignment, nullptr), UINT8_Cosine); + + // At the bound the SIMD kernel is still eligible, so the guard is a boundary and not a blanket + // disable. Only meaningful where a uint8 SIMD tier exists. + if (!AvailableUInt8Tiers(last_simd).empty()) { + EXPECT_NE(L2_UINT8_GetDistFunc(last_simd, &alignment, nullptr), UINT8_L2Sqr); + EXPECT_NE(IP_UINT8_GetDistFunc(last_simd, &alignment, nullptr), UINT8_InnerProduct); + EXPECT_NE(Cosine_UINT8_GetDistFunc(last_simd, &alignment, nullptr), UINT8_Cosine); + } + + // And the scalar path must actually be exact where it takes over, which is what makes the + // fallback safe rather than merely different. All-255 against all-0 gives 65,025 * dim, which + // passes UINT32_MAX at this dimension, so a 32-bit accumulator could not carry it. + std::vector ones(first_scalar + sizeof(float), 255); + std::vector zeros(first_scalar + sizeof(float), 0); + const uint64_t expected_l2 = 65025ULL * first_scalar; + EXPECT_GT(expected_l2, static_cast(std::numeric_limits::max())) + << "this dimension no longer exceeds a 32-bit accumulator, the test has lost its point"; + EXPECT_EQ(static_cast(expected_l2), + UINT8_L2Sqr(ones.data(), zeros.data(), first_scalar)); + const uint64_t expected_ip = 65025ULL * first_scalar; + EXPECT_EQ(static_cast(1 - static_cast(expected_ip)), + UINT8_InnerProduct(ones.data(), ones.data(), first_scalar)); } // The tests above go through the generic dispatcher, which only ever returns the best tier this // host supports, so on an ARM machine with SVE the NEON and NEON_DOTPROD chunked kernels are never // executed. Reach every compiled-in tier directly instead. Each tier is still gated on the CPU // actually supporting it, since calling an unsupported kernel faults. -TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { - // Boundary (plain family), one past it with residuals 0/1/63, two chunks, and a ragged - // multiple. - const std::vector dims = {65536, 65600, 65601, 65663, 131072, 200000}; +TEST_F(SpacesTest, UINT8_every_tier_is_exact_up_to_the_dispatcher_cap) { + // The signed-reduction boundary, then the dispatcher cap. All of these stay on SIMD: 33,025 is + // the last total that fits INT32_MAX, 33,026 the first that does not, 66,051 the last that fits + // UINT32_MAX. Residual-bearing dimensions are included at each end. + const std::vector dims = {33025, 33026, 33087, 66011, 66051}; std::set all_tiers; - constexpr size_t max_dim = 200000; + constexpr size_t max_dim = 66051; std::vector ones(max_dim + sizeof(float), 255); std::vector ramp(max_dim + sizeof(float)); for (size_t i = 0; i < max_dim; i++) { @@ -2543,41 +2525,6 @@ TEST_F(SpacesTest, UINT8_every_tier_is_exact_past_the_chunk_boundary) { } } -// The chooser picks the chunked kernel once per index rather than branching per call, so assert the -// switch actually happens. Both dimensions are a multiple of 64, so they map to the same residual -// instantiation: any difference in the returned pointer can only come from the chunked family being -// chosen. Only meaningful where a uint8 SIMD tier exists, since otherwise both are the scalar -// kernel. -TEST_F(SpacesTest, UINT8_choosers_switch_to_the_chunked_kernel_past_the_chunk_size) { - const auto features = getCpuOptimizationFeatures(); - const bool has_uint8_simd = -#ifdef CPU_FEATURES_ARCH_X86_64 - features.avx512f && features.avx512bw && features.avx512vl && features.avx512vnni; -#else - features.sve2 || features.sve || features.asimddp || features.asimd; -#endif - if (!has_uint8_simd) { - GTEST_SKIP() << "no uint8 SIMD tier on this host"; - } - - constexpr size_t plain = spaces::UINT8_CHUNK_ELEMENTS; // on the boundary, not chunked - constexpr size_t chunked = spaces::UINT8_CHUNK_ELEMENTS * 2; // same residual, chunked - unsigned char alignment = 0; - - EXPECT_NE(L2_UINT8_GetDistFunc(plain, &alignment, nullptr), - L2_UINT8_GetDistFunc(chunked, &alignment, nullptr)); - EXPECT_NE(IP_UINT8_GetDistFunc(plain, &alignment, nullptr), - IP_UINT8_GetDistFunc(chunked, &alignment, nullptr)); - EXPECT_NE(Cosine_UINT8_GetDistFunc(plain, &alignment, nullptr), - Cosine_UINT8_GetDistFunc(chunked, &alignment, nullptr)); - - // And the SIMD kernel is still what gets chosen past the boundary: the chunked variant replaces - // the plain one, it does not fall back to scalar. - EXPECT_NE(L2_UINT8_GetDistFunc(chunked, &alignment, nullptr), UINT8_L2Sqr); - EXPECT_NE(IP_UINT8_GetDistFunc(chunked, &alignment, nullptr), UINT8_InnerProduct); - EXPECT_NE(Cosine_UINT8_GetDistFunc(chunked, &alignment, nullptr), UINT8_Cosine); -} - class SQ8_FP32_SpacesOptimizationTest : public testing::TestWithParam {}; TEST_P(SQ8_FP32_SpacesOptimizationTest, SQ8_FP32_L2SqrTest) { @@ -5076,174 +5023,6 @@ TEST(SQ8_SQ8_EdgeCases, L2ExtremeValuesTest) { ASSERT_NEAR(result, baseline, 0.01f) << "Extreme values L2 should match baseline"; } -// spaces::uint8_chunked_total (uint8_chunking.h) is the chunked-accumulation driver shared by -// every uint8 SIMD kernel: it tiles a vector into segments no larger than UINT8_CHUNK_ELEMENTS -// so each segment's 32-bit SIMD partial sum stays exact, then folds the per-segment totals into -// a 64-bit scalar. The driver only ever calls Kernel::granule/first/rest, so it can be exercised -// directly with a mock kernel instead of a real SIMD kernel, which means this test needs no -// architecture-specific build flag and runs the same way on every host: an x86 box without -// AVX512 and an ARM box exercise identical logic here, closing the gap where this driver was -// previously only reachable through whichever hardware-specific kernel happened to be present. -// Real coverage of "did the driver visit every element exactly once, correctly" comes from the -// value check below: the mock's first()/rest() return the sum of the bytes in the slice they -// were handed (read from a position-dependent, non-constant fill), and the total the driver -// returns is compared against an independent, trivially-correct sum over the whole buffer. A -// skipped element, a double-counted element, or a mis-sized chunk changes that sum; it cannot -// cancel out the way it could with a constant fill or a return value of 0. The mock also records -// the (offset, length) of every call; the tiling check on those recordings does not by itself -// prove the driver visited the right elements (offset is derived from the same length the driver -// just advanced its pointer by, so "no gap/overlap" holds by construction), but it does guard the -// coupling between the length passed to the kernel and the distance the pointer is advanced, plus -// the length-shape properties below (granule multiples, chunk-size cap, congruence). Together the -// two checks cover a sweep of granules (64, 128, 192, 256 and 1024, standing in for fixed-width -// kernels and SVE vector lengths of 32/48/64/256 bytes) and dimensions chosen to cover every -// residue class modulo the granule across one-, two- and three-segment cases, plus the boundary -// around UINT8_CHUNK_ELEMENTS itself. -TEST_F(SpacesTest, UINT8_chunked_driver_tiles_the_vector_exactly) { - struct RecordedCall { - size_t offset; - size_t length; - }; - - // Local mock adapter matching the Kernel contract from uint8_chunking.h. All state lives in - // function-local statics reached through static member functions (a local class cannot have - // static data members), so `reset` must be called before each driver invocation. - struct RecordingKernel { - static size_t granule() { return granule_ref(); } - - // Returns the sum of the bytes in [v1, v1 + length), not 0: combined with a - // position-dependent fill, this makes the driver's return value an end-to-end proof - // that every element was visited exactly once, not just a coupling check on lengths. - static uint32_t first(const uint8_t *v1, const uint8_t *, size_t length) { - record(v1, length); - return sum_of(v1, length); - } - - static uint32_t rest(const uint8_t *v1, const uint8_t *, size_t length) { - record(v1, length); - return sum_of(v1, length); - } - - static void reset(const uint8_t *base, size_t granule) { - calls_ref().clear(); - base_ref() = base; - granule_ref() = granule; - } - - static const std::vector &calls_seen() { return calls_ref(); } - - private: - static uint32_t sum_of(const uint8_t *v1, size_t length) { - uint32_t sum = 0; - for (size_t i = 0; i < length; i++) { - sum += v1[i]; - } - return sum; - } - - static void record(const uint8_t *v1, size_t length) { - calls_ref().push_back({static_cast(v1 - base_ref()), length}); - } - - static std::vector &calls_ref() { - static std::vector calls; - return calls; - } - - static const uint8_t *&base_ref() { - static const uint8_t *base = nullptr; - return base; - } - - static size_t &granule_ref() { - static size_t granule = 0; - return granule; - } - }; - - constexpr size_t chunk = spaces::UINT8_CHUNK_ELEMENTS; - // Large enough for the biggest dimension exercised below (first segment plus two full - // max-size segments, bounded by chunk), with slack. - constexpr size_t buffer_size = 3 * chunk + 4096; - // v1 is filled with a position-dependent, non-constant pattern so a skipped, duplicated or - // mis-sized element changes the summed value rather than cancelling out (a constant fill, or - // returning 0 from the mock, would not catch that). Values stay under 251 and dimensions - // stay well under 600,000, so the accumulated uint64_t sum cannot overflow. v2 is unused by - // the mock kernel and left zero-filled. - std::vector v1(buffer_size); - for (size_t i = 0; i < buffer_size; i++) { - v1[i] = static_cast((i * 31 + 7) % 251); - } - std::vector v2(buffer_size, 0); - - const std::array granules = {64, 128, 192, 256, 1024}; - - for (size_t granule : granules) { - const size_t max_step = (chunk / granule) * granule; - auto first_chunk_for = [&](size_t tail) { - return tail + ((chunk - tail) / granule) * granule; - }; - - std::vector dims; - for (size_t r = 0; r < granule; r++) { - const size_t fc = first_chunk_for(r); - dims.push_back(r == 0 ? granule : r); // one segment: dim <= chunk - dims.push_back(fc + max_step); // two segments - dims.push_back(fc + 2 * max_step); // three segments - } - dims.push_back(chunk - 1); - dims.push_back(chunk); - dims.push_back(chunk + 1); - - for (size_t dimension : dims) { - ASSERT_LE(dimension, buffer_size) - << "granule=" << granule << " dimension=" << dimension; - - RecordingKernel::reset(v1.data(), granule); - const uint64_t total = - spaces::uint8_chunked_total(v1.data(), v2.data(), dimension); - - SCOPED_TRACE("granule=" + std::to_string(granule) + - " dimension=" + std::to_string(dimension)); - - // Value check: independently sum the same byte range the driver was asked to cover. - // This is what actually proves every element was visited exactly once (a skipped, - // duplicated or mis-sized chunk changes this sum); the tiling check below only - // proves the length passed to the kernel matches how far the pointer advanced. - uint64_t expected = 0; - for (size_t i = 0; i < dimension; i++) { - expected += v1[i]; - } - EXPECT_EQ(total, expected) << "driver total does not match independent byte sum"; - - const auto &calls = RecordingKernel::calls_seen(); - ASSERT_FALSE(calls.empty()); - - size_t sum = 0; - size_t expected_offset = 0; - for (size_t i = 0; i < calls.size(); i++) { - EXPECT_EQ(calls[i].offset, expected_offset) - << "call " << i << " does not tile contiguously (gap or overlap)"; - EXPECT_LE(calls[i].length, chunk) - << "call " << i << " exceeds UINT8_CHUNK_ELEMENTS"; - if (i > 0) { - EXPECT_EQ(calls[i].length % granule, 0u) - << "call " << i << " length is not a whole multiple of granule"; - } - sum += calls[i].length; - expected_offset += calls[i].length; - } - EXPECT_EQ(sum, dimension) << "recorded lengths do not sum to the dimension"; - EXPECT_EQ(calls[0].length % granule, dimension % granule) - << "first call length is not congruent to dimension modulo granule"; - if (dimension <= chunk) { - EXPECT_EQ(calls.size(), 1u) - << "dimension at or below UINT8_CHUNK_ELEMENTS should need exactly one call"; - } - } - } -} - // Assert the exact alignment-hint values published by the SQ8 distance dispatchers. // The hint refers to the SQ8 (first / storage) operand per the GetDistFunc contract documented // in spaces/spaces.h. These tests guard against silent regressions of the per-kernel hints used