expireover: Add bloom filter for fast history existence checks#339
expireover: Add bloom filter for fast history existence checks#339kev009 wants to merge 1 commit into
Conversation
expireover checks every article in the overview database against the history file to detect orphaned entries. This requires a per-article HISlookup, which does random pread() calls into the DBZ index and history text file. On large spools (1B+ articles), this takes months. Add a bloom filter that is built from a single sequential HISwalk of the history file at startup. The bloom filter acts as a positive-only cache in OVhisthasmsgid: bloom hits skip the slow HISlookup, bloom misses fall through to HISlookup for correctness. False positives are benign (an orphaned overview entry survives one extra cycle). The bloom filter is controlled by the new inn.conf parameter expirebloomfp, which specifies the false positive rate as a reciprocal (default 10000 = 0.01%). Setting it to 0 disables the bloom filter. Memory usage is approximately 20 bits per article (48 MB for 20M articles, 2.4 GB for 1B articles). Changes: - Add lib/bloom.c and include/inn/bloom.h (bloom filter implementation using enhanced double hashing, Kirsch & Mitzenmacher 2006) - Extend HISwalk callback signature to include the message-ID HASH (HISwalk has had zero callers since it was added in 2001) - Set hisv6_walk ignore=true so corrupt lines don't abort the walk - Add OVTOKENCACHE to OVctl for passing the bloom filter to OVhisthasmsgid - Add expirebloomfp to innconf - Add unit tests (lib/bloom-t.c) and integration tests (lib/bloom-hiswalk-t.c)
kev009
force-pushed
the
expireover-token-cache
branch
from
c290835 to
d0c379e
Compare
|
What a nice contribution. Thanks Kevin for your work to improve INN! |
|
@Julien-Elie thanks. Yes I went back and checked all the repo history, this was sketched out in 2001 but I can find no evidence of it being used by any consumers so I think it is safe in principle, but we could also bump to be safe. |
ovsqlite WAL, mmap, direct readers: InterNetNews/inn#337 InterNetNews/inn#338 bloom filter InterNetNews/inn#339
|
Some empirical results from my current tradspool, weeks down to minutes: |
|
Very impressive! |
Julien-Elie
added
enhancement
|
Your patch looks good to me, thanks for it! Just a question: is the Kirsch & Mitzenmacher 2006 algorithm fine to use for us? No copyright or license to mention? I hope other people will test the patch and confirm the time benefits. The news spool of my testing news server uses mostly CNFS storage, and I unfortunately do not see any change in time with or without the bloom filter. expirebloomfp set to expirebloomfp set to expirebloomfp set to |
| spools (1B+ articles). The bloom filter is used as a positive-only | ||
| cache: hits skip the slow history lookup, misses fall through to | ||
| HISlookup for correctness (handles articles added after the walk). */ | ||
| if (innconf->expirebloomfp > 0 && !always_stat) { |
There was a problem hiding this comment.
@Julien-Elie I wonder if we add another chicken bit here and only do it for tradspool/timehash etc?
There was a problem hiding this comment.
But the times with the default fp look reasonable so maybe need to see more data points to see if i.e. CNFS should completely opt out.
There was a problem hiding this comment.
The times are similar so maybe it's not worth adding extra complexity at this point to probe whether articles are in a self-expiring storage method and the -N flag is not used.
FWIW, without the -N flag, the time of run of expireover is still similar in my news spool (about 46 seconds for 416 000 articles).
|
@kev009 There is an issue when INN is built with Seems to occur at the Hmm, maybe this is a pre-existing bug not linked to the new feature? |
2 participants
expireover checks every article in the overview database against the history file to detect orphaned entries. This requires a per-article HISlookup, which does random pread() calls into the DBZ index and history text file. On large spools (1B+ articles), this takes months.
Add a bloom filter that is built from a single sequential HISwalk of the history file at startup. The bloom filter acts as a positive-only cache in OVhisthasmsgid: bloom hits skip the slow HISlookup, bloom misses fall through to HISlookup for correctness. False positives are benign (an orphaned overview entry survives one extra cycle).
The bloom filter is controlled by the new inn.conf parameter expirebloomfp, which specifies the false positive rate as a reciprocal (default 10000 = 0.01%). Setting it to 0 disables the bloom filter. Memory usage is approximately 20 bits per article (48 MB for 20M articles, 2.4 GB for 1B articles).
Changes: