Repository navigation
Native INT8 (byte vector) HNSW build + search API #665
Description
Hi! Filing this from the ArcadeDB project, which uses JVector as the HNSW backend for its LSM_VECTOR index.
Background
We're adding pre-quantized int8 ingest to ArcadeDB (ArcadeData/arcadedb#4132) so callers using providers that emit int8 directly (Cohere embed-english-v3.0, OpenAI text-embedding-3-large reduced precision, Sentence Transformers with int8 quantization) can skip a precision-losing client-side int8 → float32 round-trip.
We dug into JVector 4.0.0-rc.8 to wire the path through and found the HNSW graph API operates on VectorFloat<?> end-to-end:
RandomAccessVectorValues.getVector(int ordinal)returnsVectorFloat<?>.GraphIndexBuilderconstructors takeRandomAccessVectorValues(float-only).VectorSimilarityFunction.compare(VectorFloat<?>, VectorFloat<?>)- the abstract method signature.
So a caller with int8 input must dequantize to float32 for graph build and for every query, even when the application semantics are int8-throughout. ByteSequence<?> exists in the type system but is used only for PQ/BQ codes (sidecar against the float-vector graph), not as a primary HNSW vector type.
Ask
A native byte (int8) vector path:
- A
RandomAccessByteVectorValues(or a generalisedRandomAccessVectorValues<T>). VectorSimilarityFunctionoverloads / variants forbyte[](cosine + dot product on byte vectors with per-block min/max calibration; euclidean on bytes is also straightforward).GraphIndexBuilderconstructor(s) that accept the byte-vector RAVV + byte similarity function.- Search-side equivalent in
GraphSearcher.
Why it matters
Modern embedding providers default to int8/binary outputs at scale - Cohere binary embeddings are 1/32× the size of float32, Cohere int8 is 1/4×. Forcing dequantize-on-build/search means:
- Build cost: O(N * dim * 4) bytes of transient float32 even though the source is bytes.
- Search cost: every query vector dequantises before comparison - JVector's SIMD intrinsics for the byte-similarity case never get exercised.
- Storage cost: applications keep bytes in their primary store but JVector wants floats, so RAM and on-disk size grow 4× beyond what the application needs.
Lucene 9.x added VectorEncoding.BYTE for similar reasons; we'd love the same in JVector to close the precision/size loop.
ArcadeDB context
For reference, our matrix vs Qdrant / Milvus 2.5 (docs/arcadedb-vs-leading-vector-dbms.md) flags pre-quantized ingest as a P2 gap. We can ship an MVP that dequantizes int8 → float32 server-side (covered in ArcadeData/arcadedb#4132) and that closes the API ergonomics gap, but the full "end-to-end int8 with no float32 transient" win requires the JVector-side support.
Happy to contribute if there's a design direction the maintainers are considering. Otherwise, this is a tracking request.
Thanks!
Activity
r-devulap commentedon Jul 16, 2026
@lvca Thanks for reaching out and apologies for taking time to respond. I could start taking a look at implementing this if it is still of interest to you.
lvca commentedon Jul 16, 2026
Hi @r-devulap, thanks for following up - yes, this is definitely still of interest.
Since filing this we shipped the ArcadeDB-side MVP (ArcadeData/arcadedb#4132):
LSM_VECTORnow accepts pre-quantized INT8 ingest (byte[] put/query over the wire), which captured the bandwidth and document-storage wins on our side. Internally though we still have to dequantize int8 -> float32 before handing vectors to JVector, for both graph build and every query, because the graph API is float-only. So the remaining value is exactly the "Ask" section above: a byte-vectorRandomAccessVectorValues, byte similarity functions (dot/cosine/euclidean), andGraphIndexBuilder/GraphSearchersupport, so int8 stays int8 end-to-end with no float32 transient (4x less RAM on build/search, and the SIMD byte kernels actually get exercised).Happy to help however it's useful: glad to review a design sketch, and once there's a branch we can wire it into ArcadeDB's INT8 path and test it end-to-end against real int8 embedding workloads - we already have integration tests and benchmarks around
LSM_VECTORthat would exercise it.Thanks for offering to take this on!
r-devulap commentedon Aug 5, 2026
@lvca here is preliminary branch for you to experiment with: https://github.com/r-devulap/jvector/tree/int8-support with these changes:
Java Interfaces & Types
RandomAccessByteVectorValues,ListRandomAccessByteVectorValues, andByteVectorSimilarityFunction.VectorUtil,VectorUtilSupport, andDefaultVectorUtilSupportwith byte-vector similarity methods.Graph Index Wiring
GraphIndexBuilder.build()andaddGraphNode()overloads for byte-vector support.BuildScoreProvider.byteVectorScoreProviderfactory to integrate byte-vector scoring into the indexing pipeline.Panama and native SIMD Acceleration
ByteSequencesimilarity metrics using the Java Panama SIMD API.dot_product_i8,euclidean_i8, andcosine_i8using Highway's generic execution path and AVX-512 VNNI optimized kernels.Utilities
SiftLoader.readBvecsfor loading.bvecdatasets. Enables byte-vector datasets to be used with existing benchmarking and evaluation pipelines.lvca commentedon Aug 5, 2026
@r-devulap That's awesome! Will it be in the next release?
r-devulap commentedon Aug 7, 2026
Not sure, it still needs to be tested/reviewed by you and the team. Have you had a chance to test it yet?
r-devulap commentedon Aug 11, 2026
see #709
lvca commentedon Aug 11, 2026
@r-devulap hard for us to test it switching to a private fork. Is there any chance you can add this to the latest snapshot? Once there we can test in in 24h.
tae898 commentedon Nov 1, 2026
The INT8 against FP32 numbers I said I would post are on the PR, measured on its head
575dcee1b: #709 (comment) (SIFT 128-d, DBpedia 768-d and 1536-d; INT8 faster at p50 in every cell, recall@10 within 0.011). The PQ, BQ and NVQ comparison will follow with my next run on a current build; I can also run any configuration you want checked before a merge.r-devulap commentedon Nov 2, 2026
@tae898 thanks a lot for your numbers! Quick question: the current INT8 implementation in #709 does not enable any compression to be used with native INT8 vectors. So I am curious to know what you mean by
PQ, BQ and NVQ comparison.tae898 commentedon Nov 2, 2026
@r-devulap good question, the wording was ambiguous. I did not mean compressing the INT8 vectors. I meant comparing native INT8 against the compressed paths jvector already has for FP32 vectors, as alternative ways to cut memory and latency for the same data:
on the same datasets as the numbers above (SIFT 128-d, DBpedia 768-d and 1536-d), at matched recall@10 targets, reporting p50 and p99, build time, stored bytes and index file size, on one jvector head. The question it answers for a user like ArcadeDB is which of these to pick for a given memory budget, and where native INT8 sits among them.
If a different comparison would be more useful for reviewing #709 (for example INT8 against FP32 at more recall targets, or other dimensions), tell me and I will run that first.
r-devulap commentedon Nov 2, 2026
ah that makes sense. thanks for clarifying.