Release v0.2.4: Stateless Searcher pipeline, contiguous batch search, and unified Python search API

- Refactor SearcherBase & SearcherImpl to be fully stateless and thread-safe with per-query eps and rerank_factor parameters
- Introduce SearchResultBatch flat contiguous container and zero-allocation span-based batch search
- Separate result unpacking and distance copying into branchless sequential loops in populate_graph_results and drain_heap_into
- Modernize candidate reranking API (deglib::search::rerank) with explicit return_distances and unsorted flags
- Unify Python Searcher.search to handle single-query vectors (1D, 1xDim, Dimx1) and multithreaded 2D batches seamlessly
- Optimize FP16 scalar quantization via 64-element chunked SIMD streaming and encapsulate internal quantizer transforms

Author: Neiko2002

RELEASE_NOTES.md

@@ -1,5 +1,35 @@
 # Release Notes
 
+## deglib v0.2.4
+
+### Overview
+deglib v0.2.4 refactors the query execution pipeline to be fully stateless and thread-safe, eliminates query-time allocations, streamlines candidate reranking, and optimizes FP16 quantization kernels. It updates `Searcher` to accept search parameters (`eps`, `rerank_factor`) directly per query, adds contiguous flat batch search (`SearchResultBatch`), separates inner loop branches for result extraction, and consolidates the Python search interface.
+
+---
+
+### ⚠️ Breaking Changes & Migration Guide
+
+* **Stateless `Searcher` API:** Removed mutable internal search parameters (`search_eps_`, `rerank_factor_`) and their setters/getters (`set_query_arguments`, `set_search_eps`, `set_rerank_factor`). `eps` and `rerank_factor` are now query-level parameters passed directly to `search()` and `search_batch()`.
+* **Rerank API Consolidation (`deglib::search::rerank`):** Removed disparate internal helpers (`rerank_into`, etc.) and exposed overloaded `deglib::search::rerank` functions with explicit `return_distances` and `unsorted` boolean flags.
+* **Unified Python `Searcher.search`:** Single entry point handling 1D vectors, row-vectors `(1, dim)`, column-vectors `(dim, 1)`, and 2D batch matrices `(N, dim)` with `threads`. Removed redundant `search_batch` from the Python class.
+
+---
+
+### 🚀 Key Features & Improvements
+
+#### Stateless & Thread-Safe Search Execution (`deglib::search`)
+* **Reentrant Querying:** Searcher instances are fully thread-safe and reentrant, allowing multiple threads to query the same instance with varying `eps` and `rerank_factor` trade-offs without data races or locks.
+* **Branch-Free Result Extraction:** Separated neighbor index unpacking and optional distance copying into distinct, branchless loops (`populate_graph_results` and `drain_heap_into`) to eliminate per-element conditional branches on the hot path.
+* **Contiguous Batch Search (`SearchResultBatch`):** Flat 1D vector layout for indices and distances (`n_queries * k`) with zero-overhead `get_indices(q)` and `get_distances(q)` row slicing. Supports zero-allocation execution with user-provided `std::span` buffers.
+* **Small Buffer Optimization (SBO):** Uses stack buffers for transformed query bytes and candidate indices for vectors $\le 512$ bytes, avoiding heap allocations during search.
+
+#### Quantization & Kernel Optimizations (`deglib::optimization::quantization`)
+* **Chunked SIMD FP16 Quantization:** Processes FP16 (`uint16_t`) vectors in fixed 64-element chunks via `fp16_to_floats`, ensuring complete auto-vectorization (AVX2/F16C/AVX-512) without dimension limits.
+* **In-Place EVP Query Quantization:** Direct query quantization into preallocated search buffers without intermediate vector allocations.
+* **API Clean-Up:** Encapsulated internal quantizer row transformations (`transform`, `transform_row`, `transform_back`) as private. Removed obsolete methods (`dequantize`, `fit_quantize`, `inv_scale`).
+
+---
+
 ## deglib v0.2.3
 
 ### Overview
@@ -13,7 +43,7 @@ deglib v0.2.3 introduces a zero-overhead C++20 templated **`Searcher`** pipeline
 * **Zero-Overhead End-to-End Querying:** Integrates query quantization (FP32/FP16 $\rightarrow$ INT8/UINT8/EVP), graph traversal on quantized indices, and SIMD candidate refinement against FP16/FP32 base features in a single C++ call.
 * **Static Template Specialization (`SearcherImpl<QuantT, RefinerT>`):**
   * Compile-time `if constexpr` branch elimination avoiding virtual method dispatch in the hot search loop.
-  * Full support for all quantizers (`NoQuantizer`, `ScalarInt8Quantizer`, `ScalarInt8PerDimQuantizer`, `ScalarUint8Quantizer`, `ScalarUint8PerDimQuantizer`, `EVPQuantizer`).
+  * Full support for all quantizers (`NoQuantizer`, `ScalarQuantizerInt8`, `ScalarQuantizerInt8PerDim`, `ScalarQuantizerUint8`, `ScalarQuantizerUint8PerDim`, `EVPQuantizer`).
   * Support for unquantized and quantized candidate refiners (`NoRefiner`, `ExactRefiner<uint16_t>`, `ExactRefiner<float>`).
 * **Modern C++20 Interface:**
   * `std::span<const T>` and `SearchResult` helpers for safe, expressive, and allocation-free querying.

cpp/deglib/include/deglib/optimization/quantization/evp_quantize.h

@@ -17,17 +17,17 @@ namespace deglib::quantization::evp {
 // ============================================================================
 
 /**
- * Quantizes a single fp32 vector to EVP bytes.
+ * Quantizes a single fp32 vector directly into a pre-allocated EVP byte buffer.
  *
  * Layout: [ones (dim/8 bytes)][negative_ones (dim/8 bytes)]
  *
  * @param embedding  Pointer to dim float values (row-major)
  * @param dim        Dimension (must be divisible by 8)
  * @param non_zeros  Number of top-K elements by absolute value
- * @return std::vector<std::byte> with 2 * dim/8 bytes
+ * @param out_bytes  Pointer to buffer of at least 2 * dim/8 bytes
  * @throws std::invalid_argument if dim % 8 != 0 or non_zeros >= dim
  */
-inline std::vector<std::byte> quantize_single(const float* embedding, uint32_t dim, uint32_t non_zeros) {
+inline void quantize_single_into(const float* embedding, uint32_t dim, uint32_t non_zeros, std::byte* out_bytes) {
     if (dim % 8 != 0) {
         throw std::invalid_argument("quantize_single: dim must be divisible by 8, got " + std::to_string(dim));
     }
@@ -36,8 +36,6 @@ inline std::vector<std::byte> quantize_single(const float* embedding, uint32_t d
     }
 
     const size_t mask_bytes = dim / 8;
-    std::vector<std::byte> result(2 * mask_bytes);
-
     std::vector<std::pair<float, uint32_t>> abs_vals(dim);
     for (uint32_t i = 0; i < dim; ++i) {
         abs_vals[i] = {std::abs(embedding[i]), i};
@@ -51,8 +49,8 @@ inline std::vector<std::byte> quantize_single(const float* embedding, uint32_t d
         is_top[abs_vals[j].second] = 1;
     }
 
-    std::byte* ones_dst = result.data();
-    std::byte* negs_dst = result.data() + mask_bytes;
+    std::byte* ones_dst = out_bytes;
+    std::byte* negs_dst = out_bytes + mask_bytes;
 
     for (uint32_t byte_idx = 0; byte_idx < mask_bytes; ++byte_idx) {
         int byte_val = 0;
@@ -77,7 +75,23 @@ inline std::vector<std::byte> quantize_single(const float* embedding, uint32_t d
         }
         negs_dst[byte_idx] = static_cast<std::byte>(byte_val);
     }
+}
 
+/**
+ * Quantizes a single fp32 vector to EVP bytes.
+ *
+ * Layout: [ones (dim/8 bytes)][negative_ones (dim/8 bytes)]
+ *
+ * @param embedding  Pointer to dim float values (row-major)
+ * @param dim        Dimension (must be divisible by 8)
+ * @param non_zeros  Number of top-K elements by absolute value
+ * @return std::vector<std::byte> with 2 * dim/8 bytes
+ * @throws std::invalid_argument if dim % 8 != 0 or non_zeros >= dim
+ */
+inline std::vector<std::byte> quantize_single(const float* embedding, uint32_t dim, uint32_t non_zeros) {
+    const size_t mask_bytes = dim / 8;
+    std::vector<std::byte> result(2 * mask_bytes);
+    quantize_single_into(embedding, dim, non_zeros, result.data());
     return result;
 }
 
@@ -378,11 +392,69 @@ inline std::vector<std::byte> quantize_batch(const std::vector<std::vector<std::
     return result;
 }
 
+/**
+ * Quantizes a single FP16 (uint16_t) vector directly into a pre-allocated EVP byte buffer.
+ *
+ * Layout: [ones (dim/8 bytes)][negative_ones (dim/8 bytes)]
+ */
+inline void quantize_single_into(const uint16_t* embedding, uint32_t dim, uint32_t non_zeros, std::byte* out_bytes) {
+    if (dim % 8 != 0) {
+        throw std::invalid_argument("quantize_single: dim must be divisible by 8, got " + std::to_string(dim));
+    }
+    if (non_zeros >= dim) {
+        throw std::invalid_argument("quantize_single: non_zeros must be < dim");
+    }
+
+    const size_t mask_bytes = dim / 8;
+    std::vector<std::pair<uint16_t, uint32_t>> abs_vals(dim);
+    for (uint32_t i = 0; i < dim; ++i) {
+        abs_vals[i] = {static_cast<uint16_t>(embedding[i] & 0x7FFFu), i};
+    }
+    std::nth_element(abs_vals.begin(), abs_vals.begin() + (dim - non_zeros), abs_vals.end(), [](const auto& a, const auto& b) {
+        return a.first != b.first ? a.first < b.first : a.second < b.second;
+    });
+
+    std::vector<uint8_t> is_top(dim, 0);
+    for (uint32_t j = dim - non_zeros; j < dim; ++j) {
+        is_top[abs_vals[j].second] = 1;
+    }
+
+    std::byte* ones_dst = out_bytes;
+    std::byte* negs_dst = out_bytes + mask_bytes;
+
+    for (uint32_t byte_idx = 0; byte_idx < mask_bytes; ++byte_idx) {
+        int bv = 0;
+        for (int bit = 0; bit < 8; ++bit) {
+            const uint32_t bit_idx = byte_idx * 8 + bit;
+            if (bit_idx >= dim) break;
+            if (is_top[bit_idx] && (embedding[bit_idx] & 0x8000u) == 0 && (embedding[bit_idx] & 0x7FFFu) > 0) {
+                bv |= (1 << bit);
+            }
+        }
+        ones_dst[byte_idx] = static_cast<std::byte>(bv);
+    }
+
+    for (uint32_t byte_idx = 0; byte_idx < mask_bytes; ++byte_idx) {
+        int bv = 0;
+        for (int bit = 0; bit < 8; ++bit) {
+            const uint32_t bit_idx = byte_idx * 8 + bit;
+            if (bit_idx >= dim) break;
+            if (is_top[bit_idx] && (embedding[bit_idx] & 0x8000u) != 0 && (embedding[bit_idx] & 0x7FFFu) > 0) {
+                bv |= (1 << bit);
+            }
+        }
+        negs_dst[byte_idx] = static_cast<std::byte>(bv);
+    }
+}
+
 /**
  * Quantizes a single FP16 (uint16_t) vector to EVP bytes.
  */
 inline std::vector<std::byte> quantize_single(const uint16_t* embedding, uint32_t dim, uint32_t non_zeros) {
-    return quantize_batch(embedding, 1, dim, non_zeros, 1);
+    const size_t mask_bytes = dim / 8;
+    std::vector<std::byte> result(2 * mask_bytes);
+    quantize_single_into(embedding, dim, non_zeros, result.data());
+    return result;
 }
 
 }  // namespace deglib::quantization::evp