Add usage.prompt_tokens_details.cached_tokens for prefix caching

[lvhan028]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[Copilot]

Pull request overview

This PR adds end-to-end reporting for prefix-cache hit tokens (“cached tokens”) so that OpenAI-compatible responses can expose usage.prompt_tokens_details.cached_tokens, and internal metrics/benchmarks can track prefix caching effectiveness.

Changes:

• Add PromptTokensDetails(cached_tokens) to the OpenAI protocol and centralize usage construction via build_usage_info(...).
• Plumb cached_tokens from prefix-cache matching (BlockTrie.match) through PyTorch engine messaging/metrics to server responses (GenOut.cached_tokens).
• Add Prometheus metrics + benchmark support for cached token counts / cache hit ratio, and add unit tests for the new behavior.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
lmdeploy/serve/openai/protocol.py	Adds PromptTokensDetails and build_usage_info to include cached token details in UsageInfo.
lmdeploy/serve/openai/api_server.py	Switches usage construction to build_usage_info and forwards res.cached_tokens into OpenAI responses.
lmdeploy/serve/core/async_engine.py	Adds cached_tokens to GenOut and propagates it from engine outputs into responses.
lmdeploy/pytorch/paging/scheduler.py	Fixes migration scheduling to call block_trie.match(seq) correctly.
lmdeploy/pytorch/paging/block_trie.py	Records seq.prefix_cache_hit_tokens (and resets to 0 when prefix caching is disabled).
lmdeploy/pytorch/messages.py	Adds prefix_cache_hit_tokens to SchedulerSequence for prefix caching accounting.
lmdeploy/pytorch/engine/engine_loop.py	Emits cached_tokens in RequestMetrics based on msg.prefix_cache_hit_tokens.
lmdeploy/pytorch/engine/engine_instance.py	Extracts cached_tokens from req_metrics and forwards it via EngineOutput.
lmdeploy/messages.py	Adds cached_tokens fields to Response, RequestMetrics, and EngineOutput dataclasses.
lmdeploy/metrics/stats.py	Adds cached_tokens to per-request stats.
lmdeploy/metrics/metrics_processor.py	Copies outputs.cached_tokens into RequestStats.cached_tokens.
lmdeploy/metrics/loggers.py	Adds Prometheus histograms/counter for cached tokens and cache-hit ratio.
benchmark/benchmark_chat_completion.py	Forwards per-row tools/tool_choice and extracts cached_tokens from streamed usage payloads for reporting.
tests/test_lmdeploy/test_prefix_cache_hit_tokens.py	Adds a unit test ensuring disabled prefix caching sets hit tokens to 0.
tests/test_lmdeploy/serve/openai/test_usage_info.py	Adds unit tests verifying prompt_tokens_details.cached_tokens appears in built usage and dumps correctly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.