[None][feat] Attribute KV-cache hits to storage tiers in metrics (gpu/host/disk/remote)

[zheyuf]

[None][feat] Attribute KV-cache hits to storage tiers in metrics (gpu / host / disk / remote)

Description

Problem. With host offloading (host_cache_size) a prefix-cache hit can be served from GPU
memory or copied back from host memory (or disk / an external store), but kv_cache_hit_rate,
kv_cache_iter_reused_blocks_total and prompt_cached_tokens_total report one aggregate.
kv_cache_host_utilization says how full the host pool is, not whether it served anything, and
kv_cache_onboard_bytes_total cannot be used to reconstruct hits because blocks also move on
resume after preemption and on prefetch. Operators cannot answer "is CPU offloading paying off",
and benchmark harnesses that price cached vs. uncached tokens get no tier information.

Solution. The KV cache manager attributes every reused block to the tier it was served from
at the moment the block is matched, before any onboard copy, and the split is exported end to end:

• Schema: KvCacheStats::reusedBlocks{Gpu,Host,Disk,Remote},
  KvCacheIterationStats::iterReusedBlocks{Gpu,Host,Disk,Remote}, the same four fields on
  executor::KvCacheStats (JSON + binary serialization, nanobind), KvCacheTier enum shared
  with Python (tensorrt_llm/metrics/enums.py::CACHE_TIER_LABELS).
• KVCacheManager (C++): tier decided in onboardAndAllocateBlocks before onboardBlock()
  (isPrimary() on the matched block / partial-copy source; secondary is host for transfer mode
  DRAM and disk for GDS/POSIX). Connector-matched whole blocks move from missed to
  reused(remote) so aggregate hit rate and tier split agree. Per request the manager records
  ordered (tier, tokens) runs for the window that bounds the prepopulated length
  (LlmRequest.reuse_tier_segments).
• KVCacheManagerV2: per attention life cycle, reused blocks are split by page.cache_level; the
  request segments use the worst level among the life cycles that load a block, and none for
  blocks outside every sliding window (skipped by reuse, nothing loaded).
• Python: LlmRequest.cached_tokens_by_tier derives the request-level split from the segments
  and the latched cached_tokens, so the two request counters agree by construction. The split
  travels with the response (LlmResult.cached_tokens_by_tier) into
  usage.prompt_tokens_details.cached_tokens_details, including through disaggregated serving
  (ctx_usage and the KV-transfer aux buffer).
• Prometheus (MetricsCollector):
  • trtllm_kv_cache_reused_blocks_by_tier_total{cache_tier} — sums to
    trtllm_kv_cache_iter_reused_blocks_total
  • trtllm_prompt_cached_tokens_by_tier_total{cache_tier} — sums to
    trtllm_prompt_cached_tokens_total for every request that carried an attribution
  • all cache_tier values pre-seeded to 0; no existing metric changes shape.
• Warmup: per-iteration KV deltas accumulated during warmup are drained after warmup, so the
  iteration counters start from the same point as the already-baselined cumulative counters.
• Docs: docs/source/features/kvcache.md, "Cache Hit Metrics by Storage Tier" (semantics,
  invariants, warmup, disagg, per-manager tier meaning).

Behavior changes to note: (1) with a KV cache connector, whole connector-filled blocks now count
as reuse (remote) instead of misses, so kv_cache_hit_rate reflects avoided recomputation;
(2) kv_cache_iter_* counters no longer include warmup dummy requests.

Test Coverage

• tests/unittest/metrics/test_collector.py::TestCacheTierMetrics — pre-seeding, request split
  sums to aggregate, no guessing without attribution, none tier, V1 window and V2 lifecycle
  block aggregation.
• tests/unittest/_torch/executor/test_kv_cache_tier_split.py — segment-to-dict split, prefix
  trimming, sum invariant.
• tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_tier_attribution.py — V2 gpu / host /
  none attribution against the runtime manager (host hits cross-checked with onboard traffic).
• cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp —
  KVCacheManagerReuseTierAttributionGpuTest, KVCacheManagerReuseTierAttributionHostTest.
• End-to-end (manual, script in the PR discussion): trtllm-serve with a 4096-token GPU pool and a
  host cache, prompt A -> fillers -> prompt A again shows cache_tier="host" hits for both
  managers and matching cached_tokens_details in the response usage.

PR Checklist

• PR description clearly explains what and why.
• Follows TRT-LLM CODING GUIDELINES.
• Test cases provided for new code paths.
• Documentation updated.

🤖 Generated with Claude Code