A concurrent, bounded-size Java cache implementing the Window-TinyLFU
admission policy — the same design family as Caffeine.
This is the Java sibling of w-tinylfu-cache,
a from-scratch Rust implementation of the same design; both are learning projects,
built layer by layer to understand why a production cache like Caffeine is
built the way it is, not just to reimplement its surface API.
Requires a JDK; the Gradle wrapper provisions a Java 21 toolchain automatically if one isn't already installed.
./gradlew build # compiles tinylfu-cache and runs its full test suite
./gradlew test # tests only
The cache is built in layers: a small public API on top of one concrete
engine (BoundedLocalCache), which in turn is built from four largely
independent pieces — a concurrent map that is always the synchronous source
of truth, two buffers that decouple reads/writes from policy bookkeeping, an
eviction policy, and (optionally) an expiration timer wheel.
graph TB
Builder["CacheBuilder"]
API["Cache interface"]
Engine["BoundedLocalCache<br/>(the concurrent engine)"]
Map["ConcurrentHashMap<br/>synchronous source of truth<br/>(key to Node)"]
ReadBuf["StripedReadBuffer<br/>hand-rolled lock-free ring buffers, drops on overflow"]
WriteBuf["WriteBuffer<br/>hand-rolled lock-free unbounded queue, never drops"]
Policy["WindowTinyLfuPolicy<br/>admission + SLRU eviction"]
Sketch["FrequencySketch + Doorkeeper<br/>TinyLFU admission signal"]
Wheel["TimerWheel<br/>expire-after-write / -access"]
Builder -->|build| Engine
API -.->|implemented by| Engine
Engine -->|getIfPresent / put / invalidate| Map
Engine -->|buffer a read| ReadBuf
Engine -->|buffer a write| WriteBuf
ReadBuf -->|drain: replay| Policy
WriteBuf -->|drain: replay| Policy
WriteBuf -->|drain: replay| Wheel
Policy --> Sketch
The defining trade-off, borrowed from Caffeine (and shared with this
project's Rust sibling): the map is always right, everything else is
eventually right. getIfPresent/put/invalidate are visible the
instant they return, regardless of whether any bookkeeping has run yet. The
eviction policy's notion of "what's in the window vs. protected vs.
probation" and the timer wheel's notion of "what's about to expire" are both
allowed to lag behind the map by a bounded amount, converging once a drain
runs — the same "soft bound, eventually reclaimed" trade-off maximumSize
itself already makes (estimatedSize() can transiently exceed the
configured maximum between drains).
A two-module Gradle build: tinylfu-cache (the library) and
tinylfu-cache-bench (JMH throughput benchmarks + a hit-rate-vs-plain-LRU
test harness, depending on tinylfu-cache's test fixtures).
w-tinylfu-cache-java/
├── settings.gradle.kts, build.gradle.kts # Java 21 toolchain, JUnit 5
│
├── tinylfu-cache/ # the library
│ ├── src/main/java/io/github/czhaodev/tinylfu/
│ │ ├── Cache.java # the public interface
│ │ ├── CacheBuilder.java
│ │ ├── Ticker.java, SystemTicker.java
│ │ ├── CacheLoader.java
│ │ ├── AsyncLoadingCache.java, LoadingCache.java
│ │ └── internal/
│ │ ├── Node.java # merged "hot"/"cold" entry state
│ │ ├── BoundedLocalCache.java # the concurrent engine
│ │ ├── WindowTinyLfuPolicy.java, FrequencySketch.java, Doorkeeper.java
│ │ ├── AccessOrderDeque.java # intrusive SLRU deque over Node
│ │ ├── StripedReadBuffer.java, WriteBuffer.java, LockFreeQueue.java
│ │ ├── TimerWheel.java, Bucket.java
│ │ ├── DrainStatus.java # the CAS drain state machine
│ │ └── SimpleBoundedCache.java, WindowTinyLfuCache.java # reference impls
│ └── src/test/java/io/github/czhaodev/tinylfu/
│ ├── contract/ # a shared conformance suite run against every concrete Cache impl
│ ├── internal/ # one test class per internal class, same package (package-private access)
│ └── testutil/ # FakeTicker
│
└── tinylfu-cache-bench/
├── src/jmh/java/... # JMH throughput benchmark
└── src/test/java/... # baselines, trace replay, hit-rate comparison
Unlike the Rust sibling, where internal is pub(crate) and genuinely
enforced by the compiler, Java's internal package here is public by
convention only — nothing stops another module from reaching in. This is a
real difference in what the type system can guarantee, not just a naming
choice; see below.
| Type | Role |
|---|---|
Cache<K,V> |
getIfPresent, getWith(key, mappingFunction), put, invalidate/invalidateAll/invalidateEntries, estimatedSize, asMapSnapshot, cleanUp |
Ticker |
the time source for expiration; SystemTicker (System.nanoTime()-backed) by default, swappable for a FakeTicker in tests |
CacheBuilder<K,V> |
fluent configuration → build() / buildLoading(loader, executor) / buildAsync(loader, executor) |
CacheLoader<K,V> |
V load(K key) throws Exception — a plain functional interface; any lambda satisfies it |
AsyncLoadingCache<K,V> |
non-blocking get(key) returning an already-started CompletableFuture<V>, plus a synchronous() view |
LoadingCache<K,V> |
the blocking counterpart, implementing the full Cache-shaped surface |
Cache.getIfPresent/getWith return Optional<V> — Java references are
already freely shareable, so there's no need for the Rust sibling's Arc<V>
wrapper. asMapSnapshot is a point-in-time Map copy, not a live view, for
the same reason the Rust source gives: no single live-map abstraction is
cheaply shared across every backing implementation here (a coarse lock, or a
concurrent map plus buffered bookkeeping).
CacheBuilder.build() always produces a BoundedLocalCache;
buildLoading/buildAsync wrap that same engine (caching a
CompletableFuture<V> instead of V directly) inside a LoadingCache/
AsyncLoadingCache rather than building a second, parallel engine. See
the loading cache section.
BoundedLocalCache
keeps getIfPresent effectively lock-free by never touching the eviction
policy or timer wheel inline. Instead:
- Reads are recorded into a
StripedReadBuffer— several small, fixed-size (16-slot), per-stripe hand-rolled lock-free ring buffers (AtomicReferenceArray+ CAS'd write counter), stripe count =min(availableProcessors(), 64)rounded up to a power of two, so concurrently reading threads rarely contend with each other. A full stripe (or a losing CAS under contention) is only a hint that a drain is due, not a guarantee — the read is simply dropped rather than blocking, which is safe because read-buffer entries are advisory-only bookkeeping, never the source of truth for whether a key is present. - Writes update the backing
ConcurrentHashMapimmediately (so they're visible right away) and enqueue a bookkeeping event onto an unboundedWriteBuffer(a hand-rolled Michael & Scott 1996 lock-free queue). Unlike reads, write events are never dropped. - A single
DrainStatusstate machine (IDLE→REQUIRED→PROCESSING→IDLE), advanced only by compare-and-swap, ensures at most one thread drains at a time; every other caller's buffered event is left safely queued for that drain (or the next one) rather than blocking.
A drain replays the write buffer (always before the read buffer — a buffered
read can only ever reference a node whose ADD event is already enqueued,
so this ordering guarantees every node a buffered read names has already
been linked by the time reads are replayed), then the read buffer, then
sweeps the timer wheel for expired entries, then runs the eviction policy's
own eviction loop.
A cache entry is a single class, unlike the Rust sibling's split
Entry/LinkNode:
Node<K,V>
┌──────────────────────────────┐
│ key │
│ volatile value │
│ volatile expirationTimeNanos │
│ volatile writeExpirationTimeNanos │
│ queueType │ <- window / probation / protected, guarded by the eviction lock
│ prev / next │ <- SLRU deque links
│ prevInTimer / nextInTimer │ <- timer-wheel bucket links
│ timerBucket │ <- direct reference to its Bucket, or null
└──────────────────────────────┘
value/expirationTimeNanos/writeExpirationTimeNanos are the fields every
getIfPresent call touches — reachable without the engine lock, via
volatile. Every other field is mutated only while holding the engine's
ReentrantLock ("eviction lock"). queueType/timerBucket being null is
a normal, expected state during buffered-event replay — a race where an
UPDATE event is drained before its own ADD would be a safe no-op, mirrored
by every method that touches these fields treating null as "not currently
tracked" rather than an error. (In practice this exact race can't happen in
the Java port at all — see below — but
the check is kept anyway, as defense in depth.)
WindowTinyLfuPolicy
implements an admission window (~1% of capacity, plain LRU) feeding a
Segmented LRU (SLRU) main space split into probation (~20% of the remainder)
and protected (~80%) regions.
flowchart LR
NewKey["new key"] --> Window["Admission window<br/>(LRU)"]
Window -->|window overflow| Check{"main space has room,<br/>or candidate frequency<br/>strictly greater than victim's?"}
Check -->|yes: room, or admit| Probation["Probation<br/>(LRU)"]
Check -->|tie, or candidate freq not greater| Evict["evicted"]
Probation -->|hit| Protected["Protected<br/>(LRU)"]
Protected -->|overflow: demote LRU| Probation
Probation -->|LRU, main over capacity| Evict
- New entries always enter the window; a window hit just moves the entry to the window's MRU position.
- When the window overflows, its LRU entry becomes a candidate for the main space. If the main space has room, it's admitted directly into probation. Otherwise it's compared against probation's own LRU entry (the eviction victim): the candidate is admitted, displacing the victim, only if its estimated frequency (from the frequency sketch) is strictly greater. A tie favors the incumbent victim — this is what protects an already-warm entry from being displaced by an equally-frequent newcomer, and is exactly the property that lets the window absorb a large, bursty one-off scan as cheap LRU churn without evicting a proven working set.
- A probation hit promotes the entry to protected (demoting protected's own LRU entry back to probation if protected is now over its ~80% share); a protected hit just moves the entry to protected's MRU position.
Operates directly on Node references — no arena or index-of-node-id
indirection, since Java lets a node hold its neighbors directly.
FrequencySketch
is a 4-bit counting Count-Min Sketch: four independent hash rows ("depth"),
each a byte array with two 4-bit counters packed per byte, saturating at 15.
A key's estimated frequency is the minimum count across all four rows,
each indexed by an independent multiplicative hash.
one row, packed two 4-bit counters per byte:
byte index: 0 1 2 3 ...
bits: [hi|lo] [hi|lo] [hi|lo] [hi|lo]
counter #: 1 0 3 2 5 4 7 6
- A
Doorkeeper— a single-hash Bloom filter — absorbs a key's very first sighting for free; only its second sighting reaches the sketch. Without this, one-hit-wonder keys would pollute frequency estimates as much as genuinely recurring keys. The doorkeeper self-resets once its own fill level crosses 10% of its bit array, independent of the sketch's own aging schedule. - Once
10 × widthincrements have accumulated, every counter is halved (via a right-shift on each 4-bit nibble), not reset to zero — this lets frequency estimates track recent behavior without a separate background aging process, while still preserving relative ordering between keys rather than discarding all history at once. - Key hashing:
key.hashCode()run through a 32-bit avalanche mix (Murmur3'sfmix32finalizer). This is a deliberate departure from the Rust sibling, which mixes a 64-bitHashdigest — see below.
TimerWheel,
used only when expireAfterWrite/expireAfterAccess is configured, is a
4-level hierarchical timer wheel (bucket counts [64, 64, 24, 30]), the same
structure Netty and the Linux kernel use for the same reason: O(1) scheduling
and cancellation without a per-entry timer thread.
flowchart LR
L0["Level 0<br/>64 buckets × 1s ticks<br/>~64s span"]
L1["Level 1<br/>64 buckets<br/>~64s ticks, ~68m span"]
L2["Level 2<br/>24 buckets<br/>~68m ticks, ~27h span"]
L3["Level 3<br/>30 buckets<br/>~27h ticks, ~34d span"]
OF["Overflow list<br/>beyond ~34 days"]
L0 -->|cascade on wraparound| L1
L1 -->|cascade on wraparound| L2
L2 -->|cascade on wraparound| L3
L3 -->|cascade on wraparound| OF
Each level's tick duration equals the previous level's total span, so an
entry scheduled far in the future starts in a coarse bucket and cascades
into progressively finer buckets as time approaches its deadline. advance()
caps how many ticks it will process in one call (100,000, ~27.8 hours) so a
cache that's been idle for a very long time can't stall a caller catching it
up.
A node's Bucket
membership is tracked via a direct object reference (Node.timerBucket),
not a level/index pair — a simplification the Rust sibling's TimerBucketRef
enum can't make, since its arena-indexed nodes have no direct way to address
a bucket object; a Java Bucket is just addressable directly.
AsyncLoadingCache
wraps a Cache<K, CompletableFuture<V>> — reusing BoundedLocalCache's
W-TinyLFU/buffering/expiration machinery wholesale by caching futures
instead of values, the same trick the Rust sibling plays with
Shared<BoxFuture<...>>.
sequenceDiagram
participant A as Caller A
participant B as Caller B
participant C as AsyncLoadingCache
participant E as Executor
participant L as CacheLoader
A->>C: get("k")
C->>C: getWith("k", ...) -- miss, atomically inserts a CompletableFuture
C->>E: execute the load
C-->>A: the (already in-flight) future
B->>C: get("k")
C->>C: getWith("k", ...) -- hit: same future already present
C-->>B: the same future
L-->>E: returns the value (or throws)
E-->>A: resolves
E-->>B: resolves (same value, no second load)
Thundering-herd protection comes from BoundedLocalCache.getWith's existing
atomicity (backed by ConcurrentHashMap.compute's per-key locking): only the
first concurrent caller for a given key constructs the CompletableFuture
and submits the load; every other concurrent caller for that key observes
and shares that exact same future.
get needs none of the Rust sibling's care around eager execution: an
async fn in Rust is lazy (nothing runs until polled), so the Rust get is
deliberately a plain function that starts a tokio::spawned task before
returning, to match Java's CompletableFuture semantics — where
Executor.execute already runs the load independently of whether anyone
ever calls .join()/.get() on the result. Java's model already has the
property Rust had to work to recover.
A failed load invalidates its own key (rather than leaving it cached as a
permanent failure) so a later get() retries instead of staying poisoned —
attached via future.whenComplete(...) only after the triggering
getWith call has returned, never nested inside that key's own compute()
callback (see below for why that
ordering matters). LoadingCache.get blocks the calling thread via
CompletableFuture.join().
A few design points where this port doesn't translate the Rust source line-for-line, and why:
- No arena, no generational node id. The Rust source's
Entry/LinkNodesplit,NodeId, andSlotMaparena all exist because Rust's ownership model can't cheaply hold direct mutable references between intrusively-linked nodes across threads. Java has no such restriction — aNode<K,V>holds direct references to its neighbors, exactly like Caffeine's actual production design. This collapsesTimerBucketRef(Wheel{level,index}vs.Overflow) into a plainNode.timerBucketreference too, since aBucketis directly addressable. - Side effects deferred until after
compute()returns — for a sharper reason than it first appears.BoundedLocalCache.rawPut/getWithboth wait until aConcurrentHashMap.compute()call has returned before triggeringafterAdd/afterUpdate/afterRemove(and the synchronous drain those may trigger). The Rust source does the analogous thing withDashMap::entry()to avoid a reentrant-lock deadlock, sinceparking_lot::Mutexisn't reentrant. Java'scompute()lock is reentrant (it's a plainsynchronizedblock), so that specific deadlock isn't actually a risk here — but a different, more fundamental hazard is:compute()unconditionally reinstalls its callback's return value into the map when the callback exits, regardless of what any nested operation did to that same key in the meantime. Triggering a drain from inside the callback — before that installation happens — lets a nested eviction race against it: a brand-new node can be evicted (unlinked from the policy) beforecompute()has even installed it, after whichcompute()reinstalls it anyway, permanently orphaning it from the policy while it lingers in the map forever. This was not a theoretical concern: it surfaced as a real, reproducible test failure (inflatedestimatedSize(), and — from a related ordering issue — a concurrent test observing a stale value for a key) during development, and is fixed by deferring every side effect untilcompute()has fully returned. - Frequency sketch hashing. The Rust source's
spread()mixes a 64-bitHashdigest with splitmix64-style constants — a deliberate deviation from Caffeine's canonical 32-bithashCode()-basedspread(int), which the Rust project's own README flags as an open question for this port. This port follows Caffeine's original approach instead:hashCode()through a 32-bit Murmur3fmix32-style avalanche mix. Every other constant (depth 4, max counter 15, 4-bit packed counters, doorkeeper gating, ×4 width oversizing, sampling-size aging) is unchanged. - Buffers are hand-rolled, not borrowed.
StripedReadBufferandWriteBufferare built directly onAtomicReferenceArray/AtomicLong/AtomicReferencerather than wrappingjava.util.concurrent's own concurrent collections — a deliberate choice to keep this project's character as a from-scratch concurrency exercise, matching the Rust sibling's use ofcrossbeam's primitives rather than a higher-level off-the-shelf queue. - No
FnLoaderadapter, no boxedLoaderErrortype. Rust needsFnLoaderbecause closures can't directly implement a custom trait, andLoaderError = Box<dyn Error + Send + Sync>becauseAsyncLoadingCachewould otherwise need a third generic parameter purely for error plumbing. Neither problem exists in Java: any lambda satisfies the functionalCacheLoaderinterface directly, andCompletableFuture's exceptional completion already carries aThrowablewithout needing a parallel error type.
import io.github.czhaodev.tinylfu.Cache;
import io.github.czhaodev.tinylfu.CacheBuilder;
import java.time.Duration;
Cache<String, String> cache = new CacheBuilder<String, String>()
.maximumSize(10_000)
.expireAfterWrite(Duration.ofMinutes(5))
.build();
cache.put("k", "v");
assert cache.getIfPresent("k").equals(java.util.Optional.of("v"));A loading cache:
import io.github.czhaodev.tinylfu.CacheBuilder;
import io.github.czhaodev.tinylfu.LoadingCache;
import java.util.concurrent.Executors;
LoadingCache<String, String> cache = new CacheBuilder<String, String>()
.maximumSize(10_000)
.buildLoading(key -> "loaded-" + key, Executors.newVirtualThreadPerTaskExecutor());
String value = cache.get("k"); // blocks; loads on miss
assert value.equals("loaded-k");tinylfu-cache-bench backs up the design claims above with actual numbers:
does W-TinyLFU's admission policy really out-hit plain LRU on the workloads
it's meant for, and does the buffered/lock-free engine really pay off over a
coarse-locked implementation of the identical policy? Raw data:
hit_rates.csv,
jmh_results.json.
Three synthetic traces, each built to isolate one specific failure mode (see
Traces.java):
- Adversarial scan — a small hot key set repeatedly re-accessed, interleaved with a one-time scan of never-repeated keys every cycle. Defeats plain LRU, which evicts the entire hot set on every scan.
- Zipfian (skew 0.99) — a small number of keys accessed disproportionately often with a long tail, the standard model for real-world popularity skew.
- Popularity shift — a hot key set is "burned in" until every key has a high hit count, then access shifts entirely to a disjoint, never-before-seen set. Exposes plain LFU's lack of frequency decay.
CollectResults (./gradlew :tinylfu-cache-bench:run) replays each trace
through LRU, FIFO, LFU, and W-TinyLFU and records the resulting hit rate.
This part is deterministic trace replay with no timing involved, so it's
hardware-independent — it was re-run on the benchmarking VM below purely for
provenance, and produced byte-identical numbers to a local run.
Throughput is a different story: ConcurrentReadThroughputBenchmark
(./gradlew :tinylfu-cache-bench:jmh) measures actual wall-clock
getIfPresent throughput under 8 concurrent reader threads (JMH: 3 warmup +
5 measurement iterations, 1 fork), and that is sensitive to whatever else
is running on the machine — a laptop's thermal throttling and background
processes are enough to swing multi-threaded throughput numbers run to run.
To get a clean, reproducible number, the JMH run was taken on a dedicated GCP
c2-standard-8 VM (compute-optimized, 8 dedicated vCPUs — matching the
benchmark's 8 reader threads with no oversubscription, Ubuntu 24.04, JDK 21),
provisioned fresh, run once, and torn down immediately after.
- Adversarial scan: LRU/FIFO hold at 11.1%, unable to keep the hot set resident across a single scan. LFU and W-TinyLFU both reach 16.4% — the frequency signal (sketch-based for W-TinyLFU, exact counts for this LFU baseline) is what protects the hot set here, so the two tie.
- Zipfian: all four policies are close (59–70%), as expected for a workload with no adversarial structure to exploit; W-TinyLFU edges out every baseline (70.2%), including plain LFU (69.5%).
- Popularity shift: the honest result. Plain LFU collapses to 0% — its counts never decay, so the first-phase keys accumulate counts the second-phase keys can mathematically never catch up to, and every post-shift key loses admission forever. LRU/FIFO hit 98%, but not because they "adapt" — they have no long-term memory at all, so a hard cutover costs them nothing. W-TinyLFU lands in between at 54%: the frequency sketch's periodic halving (see The frequency sketch) does let old counts decay and new keys eventually win admission, just not as instantly as a policy with zero memory. That gap is the real, intentional price of W-TinyLFU's frequency signal — not a bug, and exactly why production caches pair it with a small recency-only admission window rather than relying on frequency alone.
(Log-scaled — the unbounded ConcurrentHashMap ceiling is nearly two orders
of magnitude above everything else and would flatten the rest of the chart on
a linear axis.)
ConcurrentHashMap(688.2M ops/s) is a ceiling, not a competitor: no eviction policy, nothing to keep bounded. It's here to show how much headroom any bounded cache gives up just by existing.- The production engine (
BoundedLocalCache, 28.0M ops/s) is the project's actual point: ~5.4× faster than the coarse-locked reference implementation of the identical W-TinyLFU policy (5.2M ops/s), and roughly 2× every synchronized baseline (FIFO 15.1M, LRU 14.1M, LFU 10.4M ops/s). Same admission/eviction algorithm, same hit rates — the only difference is thatgetIfPresentnever takes a lock, instead recording reads into striped ring buffers that get drained asynchronously (see The concurrent engine). This is the number that validates the whole buffered-reads design: the coarse-lockedWindowTinyLfuCacheexists specifically as a control to measure that design against, and the gap is the payoff. - The synchronized LRU/FIFO/LFU baselines cluster together (10–15M ops/s), all bottlenecked the same way: every reader serializes on one lock. The coarse-locked W-TinyLFU reference falls below even those, since its per-access bookkeeping (sketch updates, SLRU queue movement) all happens inside that same single lock, where the synchronized baselines only guard a plain hash map.
Reproduce locally: ./gradlew test, ./gradlew :tinylfu-cache-bench:run,
./gradlew :tinylfu-cache-bench:jmh, then
python3 tinylfu-cache-bench/scripts/plot_results.py (requires
tinylfu-cache-bench/scripts/requirements.txt). Throughput numbers will vary
with your hardware; hit-rate numbers won't.
Essential W-TinyLFU features only, matching this project's Rust/C++ siblings: no stats/counters, no background maintenance executor, no weight-based eviction, no removal listeners, no refresh-after-write, no bulk get/put.
- Einziger, G., Friedman, R., & Manes, B. "TinyLFU: A Highly Efficient Cache Admission Policy." arXiv:1512.00727
- Caffeine — the Java cache this design (and its admission/eviction policy) is modeled after; see also its design wiki.
- Cormode, G., & Muthukrishnan, S. "An Improved Data Stream Summary: The Count-Min Sketch and its Applications." See also Wikipedia: Count-min sketch.
- Bloom, B. H. "Space/Time Trade-offs in Hash Coding with Allowable Errors." See also Wikipedia: Bloom filter.
- Varghese, G., & Lauck, T. "Hashed and Hierarchical Timing Wheels: Efficient
Data Structures for Implementing a Timer Facility." The design
TimerWheelhere (and Netty'sHashedWheelTimer) are both based on. - Michael, M. M., & Scott, M. L. "Simple, Fast, and Practical Non-Blocking
and Blocking Concurrent Queue Algorithms" (1996) — the algorithm behind
LockFreeQueue. w-tinylfu-cache— the Rust sibling this project is ported from.
This project is licensed under the MIT License. See LICENSE for details.

