What happened?
The write-path dedup in add_hsg_memory treats a simhash table hit as proof of a duplicate. It returns the existing memory's id with deduplicated: true and never stores the new content. The simhash carries far less entropy than its nominal 64 bits, so unrelated memories share a hash routinely and are silently dropped. I expected two unrelated sentences to produce two memories but what actually happens is that the second one is discarded, and the caller receives an id and no error.
I found that there are three compounding causes.
1. The 64-bit simhash is at most 32 bits. In compute_simhash, each token hash h is a 32-bit integer, but the bit loop runs i from 0 to 63 testing h & (1 << i). JavaScript takes the shift count mod 32, so (1 << 35) === (1 << 3) is true. As a result, vec[32..63] exactly mirrors vec[0..31], and every hash the function emits has identical 16-character halves.
2. The surviving 32 bits are mostly dead. Token hashes are h * 31 + c over short stemmed tokens, so the high bits are rarely reached, and the sign-threshold nibble packing (vec[i] > 0) zeroes most of what remains. Real hashes look like c0600000c0600000. Feeding 4,000 distinct generated sentences through compute_simhash yields about 2,300 distinct hashes, roughly 11 bits of observed width. I can share the measurement script if useful.
3. The dedup check treats a hash match as identity, and its collision test is dead code. The lookup is exact:
select * from memories where simhash = ? order by salience desc limit 1
After an exact match, hamming_dist(simhash, existing.simhash) <= 3 is necessarily 0 and can never reject. Nothing compares actual content before the new memory is discarded.
Impact:
This is silent, unrecoverable data loss that worsens as the store grows, because the drop probability scales with the number of stored memories divided by the number of usable hash values. On my personal store of 4,841 memories, I measured 11.7% of new distinct memories landing on an already-occupied hash. The call returns an id and no error, so users have no signal that it is happening.
Where:
main @ 0761b62, packages/openmemory-js/src/memory/hsg.ts, dedup check at line 1141. The Python SDK carries it too, at packages/openmemory-py/src/openmemory/memory/hsg.py line 391, and its compute_simhash reproduces the 32-bit wrap on purpose (1 << (i % 32) at line 187) to keep hashes identical to the JS side. Both SDKs therefore depend on producing the same narrow hash, and any hash-width fix has to change both of them together.
Prior sightings. Open issue #173 (Cyrillic text, only the first record saved) is this bug class manifesting on the v1.2.x line, where Cyrillic text tokenizes to nothing and every such memory got the same all-zero hash. Current main no longer reproduces that literal case because the blake2b fallback for empty token sets now yields distinct hashes, but content that does produce tokens still collides, as the steps below show.
Steps to Reproduce
-
In packages/openmemory-js on main, hash these two unrelated sentences:
npx tsx -e 'import { compute_simhash } from "./src/memory/hsg";
console.log(compute_simhash("The expense report was moved to Thursday."));
console.log(compute_simhash("The holiday calendar was approved this morning."));'
Both print 1e1380001e138000. (Importing the module opens the database, so the process may need Ctrl+C after the output.)
-
Store both sentences through any path that reaches add_hsg_memory: the MCP openmemory_store tool, POST /memory, or a direct call.
-
List memories. Only the first sentence was stored. The second call returned the first memory's id with deduplicated: true and wrote nothing.
Component
Backend (API/Server)
Environment
- Repo: CaviraOSS/OpenMemory @ 0761b62 (main)
- Node.js: v26.5.0
- OS: macOS 15
- Backend: OM_METADATA_BACKEND=sqlite, OM_VECTOR_BACKEND=sqlite
- Embeddings: synthetic (OM_TIER=fast)
Relevant log output
compute_simhash("The expense report was moved to Thursday.") -> 1e1380001e138000
compute_simhash("The holiday calendar was approved this morning.") -> 1e1380001e138000
store A -> id=c0b923db-254c-4027-99e3-2e86c5f8490d deduplicated=false
store B -> id=c0b923db-254c-4027-99e3-2e86c5f8490d deduplicated=true
select content from memories; -> one row, sentence A onlyCode of Conduct
What happened?
The write-path dedup in
add_hsg_memorytreats a simhash table hit as proof of a duplicate. It returns the existing memory's id withdeduplicated: trueand never stores the new content. The simhash carries far less entropy than its nominal 64 bits, so unrelated memories share a hash routinely and are silently dropped. I expected two unrelated sentences to produce two memories but what actually happens is that the second one is discarded, and the caller receives an id and no error.I found that there are three compounding causes.
1. The 64-bit simhash is at most 32 bits. In
compute_simhash, each token hashhis a 32-bit integer, but the bit loop runsifrom 0 to 63 testingh & (1 << i). JavaScript takes the shift count mod 32, so(1 << 35) === (1 << 3)is true. As a result,vec[32..63]exactly mirrorsvec[0..31], and every hash the function emits has identical 16-character halves.2. The surviving 32 bits are mostly dead. Token hashes are
h * 31 + cover short stemmed tokens, so the high bits are rarely reached, and the sign-threshold nibble packing (vec[i] > 0) zeroes most of what remains. Real hashes look likec0600000c0600000. Feeding 4,000 distinct generated sentences throughcompute_simhashyields about 2,300 distinct hashes, roughly 11 bits of observed width. I can share the measurement script if useful.3. The dedup check treats a hash match as identity, and its collision test is dead code. The lookup is exact:
After an exact match,
hamming_dist(simhash, existing.simhash) <= 3is necessarily 0 and can never reject. Nothing compares actual content before the new memory is discarded.Impact:
This is silent, unrecoverable data loss that worsens as the store grows, because the drop probability scales with the number of stored memories divided by the number of usable hash values. On my personal store of 4,841 memories, I measured 11.7% of new distinct memories landing on an already-occupied hash. The call returns an id and no error, so users have no signal that it is happening.
Where:
main@0761b62,packages/openmemory-js/src/memory/hsg.ts, dedup check at line 1141. The Python SDK carries it too, atpackages/openmemory-py/src/openmemory/memory/hsg.pyline 391, and itscompute_simhashreproduces the 32-bit wrap on purpose (1 << (i % 32)at line 187) to keep hashes identical to the JS side. Both SDKs therefore depend on producing the same narrow hash, and any hash-width fix has to change both of them together.Prior sightings. Open issue #173 (Cyrillic text, only the first record saved) is this bug class manifesting on the v1.2.x line, where Cyrillic text tokenizes to nothing and every such memory got the same all-zero hash. Current
mainno longer reproduces that literal case because the blake2b fallback for empty token sets now yields distinct hashes, but content that does produce tokens still collides, as the steps below show.Steps to Reproduce
In
packages/openmemory-jsonmain, hash these two unrelated sentences:Both print
1e1380001e138000. (Importing the module opens the database, so the process may need Ctrl+C after the output.)Store both sentences through any path that reaches
add_hsg_memory: the MCPopenmemory_storetool,POST /memory, or a direct call.List memories. Only the first sentence was stored. The second call returned the first memory's id with
deduplicated: trueand wrote nothing.Component
Backend (API/Server)
Environment
Relevant log output
Code of Conduct