LCORE-3910: count, charge and report the summarization calls of compaction

docs/design/conversation-compaction/conversation-compaction.md

@@ -311,7 +311,7 @@ Add `compaction` field to the root `Configuration` class.
 |----------------------------------------|----------------------------------------------------------------------------|
 | `pyproject.toml`                       | Add `tiktoken` dependency                                                  |
 | `src/utils/token_estimator.py`         | New module: `estimate_tokens()`, `estimate_conversation_tokens()`          |
-| `src/utils/compaction.py`              | New module: summarization logic, partitioning, additive summary management |
+| `src/utils/compaction.py`              | New module: summarization logic, partitioning, additive summary management; reports the usage of each LLM call it makes — LCORE-3910 |
 | `src/models/config.py`                 | Add `CompactionConfiguration` (near `ConversationHistoryConfiguration`)    |
 | `src/configuration.py`                 | Add `compaction_configuration` property to `AppConfig` singleton           |
 | `src/utils/conversation_compaction.py` | New module: `apply_compaction()` / `apply_compaction_blocking()`, `needs_compaction_path()`, marker helpers, per-conversation lock |
@@ -333,9 +333,11 @@ reusable unit in `src/utils/conversation_compaction.py` that each endpoint
 calls after its params are prepared:
 
 - `apply_compaction_blocking(client, params, inference_config, compaction_config)`
-  returns a `CompactionResult` (possibly-rewritten params, a `summarized`
-  flag, and the `original_input`). Non-streaming `/v1/query`, A2A, and
-  `/v1/responses` use this.
+  returns a `CompactionResult` (possibly-rewritten params, a `compacted`
+  flag, the `original_input`, and the `summarization_usage`). Non-streaming
+  `/v1/query`, A2A, and `/v1/responses` use this. The endpoints also pass
+  `endpoint_path` and, where there is a quota, `charge`; see
+  [Who pays for the summarization calls](#who-pays-for-the-summarization-calls).
 - `apply_compaction(..., emit_events=True)` is the async-generator variant that
   yields a `CompactionStartedEvent` before the summarization LLM call; the
   native `/v1/streaming_query` SSE path uses it to satisfy R12.
@@ -441,6 +443,51 @@ Example config files go in `examples/`.
 
 Compaction adds latency only on the trigger turn. In PoC testing, compaction turns took 14-40 seconds vs 9-20 seconds for normal turns (gpt-5.6-luna).
 
+## Who pays for the summarization calls
+
+The provider bills the summarization call and the fold call like any other, so
+lightspeed-stack counts them (LCORE-3910). Before that, their usage was
+discarded: a user could go over the quota without it ever showing, and by more
+the longer the conversation was.
+
+`summarize_chunk()` and `recursively_resummarize()` report each call they make
+to a `count_call` callback, with the model and the usage the provider reported.
+They do it as soon as the response arrived and before they look at it, so a
+call that returned no text is reported too. `apply_compaction()` takes the
+path of the endpoint and a `charge` callback, and counts the calls with
+`SummarizationCalls` (`src/utils/compaction_usage.py`): for each call it
+records the token and call metrics under that endpoint, calls `charge`, and
+adds the usage up. The sum is handed to the endpoint as
+`CompactionResult.summarization_usage`.
+
+| Endpoint               | Quota   | Counts the client sees                                   |
+|------------------------|---------|----------------------------------------------------------|
+| `/v1/query`            | charged | `input_tokens` / `output_tokens` include summarization   |
+| `/v1/streaming_query`  | charged | the same, in the `end` event                             |
+| `/v1/responses`        | charged | `usage` as the provider reported it, the answer alone    |
+| `/a2a`                 | none    | none; the endpoint has no quota, the metrics are recorded |
+
+The endpoints with a quota pass `consume_summarization_tokens()`
+(`src/utils/query.py`), bound to the user, as `charge`. A call is therefore
+charged when it returned, and not together with the turn: the summary is
+written before the model is asked for the answer and is kept whatever becomes
+of the turn, so the call is a cost of the conversation. It is charged also
+when the turn is blocked, fails or is interrupted, and when compaction itself
+fails after the call (the marker cannot be written, the fold fails). `/a2a`
+passes no `charge`. The quota is checked once, before compaction: a request
+that started with quota left runs to its end, as before.
+
+`/v1/query` and `/v1/streaming_query` add `summarization_usage` to the usage of
+the turn in what they report to the client. The `llm.inference` span keeps the
+usage of the answer alone. The sum is not stored with the turn; the token
+usage history receives the charges one by one.
+`/v1/responses` passes the `usage` object of the response through unchanged, to
+stay a drop-in for clients of the OpenAI Responses API.
+
+Two limits. A call that fails, or is cancelled before its response arrived,
+has no usage to count, whatever the provider bills for it. And a call the
+provider reports no usage for is counted as a call and charges nothing.
+
 # Open Questions for Future Work
 
 - **Compaction-proof instructions**: Allow "pinned" messages that always survive compaction (inspired by Claude Code's CLAUDE.md pattern). Not needed for v1.

docs/design/prompt-guardrails/prompt-guardrails.md

@@ -442,7 +442,8 @@ product need justifies it.
   version-specific prompt.
 - **Guardian token usage:** whether guardian calls should count against user
   quota or be tracked as service overhead. Compaction's summarization calls
-  raise the same question.
+  are charged to the user's quota (LCORE-3910); the same choice is open for
+  guardian calls.
 - **Streaming checkpoint sizing:** defaults for LCORE-3391 (spike Decision
   T4, 70% confidence); tune with real latency data.
 - **Cheap classifier tier for `tool`:** Prompt Guard 2-class, and its

docs/devel_doc/ARCHITECTURE.md

@@ -274,9 +274,9 @@ The system defines 30+ actions that can be authorized. Examples (see `docs/user_
    - Update quota counters
 
 3. **On Error:**
-   - If LLM call fails, no tokens are consumed
-   - Quota remains unchanged
-   - User can retry the request
+   - If LLM call fails, no tokens are consumed for that call
+   - Quota remains unchanged, with one exception: the calls conversation compaction made to summarize older turns for the request were charged when they were made
+   - User can retry the request; the summary is kept, so the retry does not pay for it again
 
 ---
 

docs/devel_doc/query_endpoint.md

@@ -308,7 +308,7 @@ Cancels an in-progress streaming query.
 | `interrupted` | boolean | Whether an active stream was interrupted (`false` if already completed) |
 | `message` | string | Human-readable status message |
 
-When a stream is interrupted, any partial response is persisted to conversation history and token consumption is skipped.
+When a stream is interrupted, any partial response is persisted to conversation history and token consumption for the answer is skipped. The summarization calls conversation compaction made for the request were charged when they were made (see [Quota and Token Counting](#quota-and-token-counting)).
 
 ---
 
@@ -360,9 +360,10 @@ If the server configuration sets `disable_query_system_prompt` to `true`, reques
 ## Quota and Token Counting
 
 - **Pre-request:** `check_tokens_available()` verifies the user/cluster has available quota (429 if not)
-- **Post-response:** `consume_query_tokens()` deducts `input_tokens` and `output_tokens` from configured quota limiters
+- **Post-response:** `consume_query_tokens()` deducts the input and output tokens of the answer from configured quota limiters
+- **Compaction:** each LLM call that summarizes older turns is charged by `consume_summarization_tokens()` as soon as it returned, so also when the turn is then blocked, fails or is interrupted. The `input_tokens` and `output_tokens` the response reports include these calls
 - **Available quotas:** Remaining balances per limiter are included in the response (`available_quotas` field in sync, `end` event in streaming)
-- **Stream interruption:** Token consumption is skipped for interrupted streams
+- **Stream interruption:** Token consumption is skipped for interrupted streams, except for the summarization calls, which were charged before the answer started
 
 ---
 

docs/user_doc/conversation_compaction.md

@@ -187,6 +187,37 @@ Compaction acquires a per-conversation lock to prevent concurrent requests on th
 
 Over very long conversations, multiple compaction summaries may accumulate. When the total size of cached summaries approaches the context window threshold, they are recursively folded into a single summary using a dedicated re-summarization prompt. This prevents summaries from themselves exceeding the context window.
 
+### Token usage and quota
+
+Summarizing older turns takes an LLM call of its own, and so does folding the
+summaries. The provider bills these calls, so they are counted:
+
+| Endpoint | Quota | Reported token counts | Metrics |
+|---|---|---|---|
+| `POST /v1/query` | Charged to the user. | `input_tokens` and `output_tokens` include the summarization calls. | Counted under `/v1/query`. |
+| `POST /v1/streaming_query` | Charged to the user. | `input_tokens` and `output_tokens` of the `end` event include the summarization calls. | Counted under `/v1/streaming_query`. |
+| `POST /v1/responses` | Charged to the user. | `usage` is what the provider reported for the response, so it covers the answer alone. `available_quotas` shows the quota left after both. | Counted under `/v1/responses`. |
+| `POST /a2a` | Not charged. The endpoint does not use quotas. | None. | The summarization calls are counted under `/a2a`. The call that answers the request is not recorded in these metrics on this endpoint. |
+
+The metrics are `ls_llm_calls_total`, `ls_llm_token_sent_total` and
+`ls_llm_token_received_total`, labelled with the endpoint.
+
+Only a request that makes a summarization call pays for it: one whose
+estimated input crosses the threshold. The other requests are served from the
+stored summary and cost what they cost without compaction. With the default
+settings a summarization happens once every several turns; with a small
+context window or a low `threshold_ratio` it can happen on every request.
+
+Each summarization call is charged as soon as it returned, before the model is
+asked for the answer. It is therefore charged also when the request is blocked
+by a shield, fails afterward, or is interrupted by the client. The summary is
+kept in these cases, so the next request on the conversation does not pay for
+it again. A summarization call that itself fails, or is cut off before it
+returned, reports no usage and is not charged.
+
+On `/v1/responses`, the difference between the quota consumed and the `usage`
+of the response is the summarization.
+
 ## When compaction is disabled
 
 When compaction is disabled (the default), requests that cause the conversation history to exceed the model's context window will fail with HTTP 413 (Prompt Too Long). Clients must manage conversation length themselves, for example by starting new conversations or deleting old ones.
@@ -201,7 +232,11 @@ Compaction summarizes older turns, so fine-grained details from early in the con
 
 **Does compaction use extra tokens?**
 
-Yes. The summarization step requires an additional LLM call, which consumes tokens. These tokens are counted against the user's quota. The trade-off is that the conversation can continue instead of failing with HTTP 413.
+Yes. The summarization step requires an additional LLM call, which consumes tokens. On `/v1/query` and `/v1/streaming_query` these tokens are counted against the user's quota and are included in the `input_tokens` and `output_tokens` of the request that triggered the summarization. `/v1/responses` charges the quota and leaves `usage` as the provider reported it; `/a2a` has no quota. See [Token usage and quota](#token-usage-and-quota). The trade-off is that the conversation can continue instead of failing with HTTP 413.
+
+**Why did one request use many more tokens than the ones before it?**
+
+On `/v1/query` and `/v1/streaming_query`, that request triggered a summarization. Its `input_tokens` include the older turns that were sent to the LLM to be summarized, and its `output_tokens` include the summary. `context_status` is `"summarized"` on that request, and also on the requests after it that are served from the stored summary and do not pay for it again.
 
 **Can I use compaction with all LLM providers?**
 

src/app/endpoints/a2a.py

@@ -57,7 +57,7 @@
 from authorization.middleware import authorize
 from client.ogx import AsyncOgxClientHolder
 from configuration import configuration
-from constants import MEDIA_TYPE_EVENT_STREAM
+from constants import ENDPOINT_PATH_A2A, MEDIA_TYPE_EVENT_STREAM
 from log import get_logger
 from models.api.requests import QueryRequest
 from models.common.responses.responses_api_params import ResponsesApiParams
@@ -241,6 +241,7 @@ async def _compact_a2a_request(
         responses_params,
         configuration.inference,
         configuration.compaction,
+        endpoint_path=ENDPOINT_PATH_A2A,
     )
     return compaction.params, PendingTurn.for_request(
         client, compaction.params, compaction.original_input

src/app/endpoints/query.py

@@ -1,6 +1,7 @@
 """Handler for REST API call to provide answer to query using Response API."""
 
 import datetime
+from functools import partial
 from typing import Annotated
 
 from fastapi import APIRouter, Depends, Request
@@ -50,6 +51,7 @@
 from utils.pending_turn import pending_turn
 from utils.query import (
     consume_query_tokens,
+    consume_summarization_tokens,
     store_query_results,
     validate_attachments_metadata,
     validate_model_provider_override,
@@ -242,6 +244,8 @@ async def _handle_query_with_tracing(
         cache=configured_conversation_cache(),
         user_id=user_id,
         skip_user_id_check=_skip_userid_check,
+        endpoint_path=endpoint_path,
+        charge=partial(consume_summarization_tokens, user_id),
     )
     responses_params = compaction.params
 
@@ -306,6 +310,9 @@ async def _handle_query_with_tracing(
         model_id=responses_params.model,
         token_usage=turn_summary.token_usage,
     )
+    # The turn reports the summarization calls made for it as part of its
+    # usage. They were charged when they were made.
+    reported_usage = turn_summary.token_usage + compaction.summarization_usage
 
     logger.info("Getting available quotas")
     available_quotas = get_available_quotas(
@@ -351,7 +358,7 @@ async def _handle_query_with_tracing(
         referenced_documents=turn_summary.referenced_documents,
         truncated=False,
         context_status=compaction.context_status,
-        input_tokens=turn_summary.token_usage.input_tokens,
-        output_tokens=turn_summary.token_usage.output_tokens,
+        input_tokens=reported_usage.input_tokens,
+        output_tokens=reported_usage.output_tokens,
         available_quotas=available_quotas,
     )

src/app/endpoints/responses.py

@@ -6,6 +6,7 @@
 import time
 from collections.abc import AsyncIterator, Sequence
 from datetime import UTC, datetime
+from functools import partial
 from typing import TYPE_CHECKING, Annotated, Any, Final, NoReturn, Optional, cast
 
 from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
@@ -90,6 +91,7 @@
 from utils.prompts import get_system_prompt
 from utils.query import (
     consume_query_tokens,
+    consume_summarization_tokens,
     extract_provider_and_model_from_model_id,
     handle_known_apistatus_errors,
     is_context_length_error,
@@ -737,6 +739,11 @@ async def handle_responses_with_tracing(  # pylint: disable=too-many-locals
             cache=configured_conversation_cache(),
             user_id=user_id,
             skip_user_id_check=skip_userid_check,
+            endpoint_path=endpoint_path,
+            # The summarization calls are charged to the user. The usage
+            # object of the response stays as OGX reported it, so it covers
+            # the answer alone.
+            charge=partial(consume_summarization_tokens, user_id),
         )
         api_params = compaction.params
         if compaction.compacted:

src/app/endpoints/streaming_query.py

@@ -3,6 +3,7 @@
 import asyncio
 import datetime
 from collections.abc import AsyncIterator
+from functools import partial
 from typing import TYPE_CHECKING, Annotated, Optional
 
 from fastapi import APIRouter, Depends, HTTPException, Request
@@ -69,6 +70,7 @@
 )
 from utils.pending_turn import PendingTurn
 from utils.query import (
+    consume_summarization_tokens,
     extract_provider_and_model_from_model_id,
     handle_known_apistatus_errors,
     is_context_length_error,
@@ -89,6 +91,7 @@
     stream_start_event,
 )
 from utils.suid import get_suid, normalize_conversation_id
+from utils.token_counter import TokenCounter
 from utils.types import Responses
 from utils.vector_search import build_rag_context
 
@@ -424,6 +427,7 @@ async def generate_response_with_compaction(
 
         turn: Optional[PendingTurn] = None
         context_status: ContextStatus = "full"
+        summarization_usage = TokenCounter()
         try:
             async for item in apply_compaction(
                 context.client,
@@ -434,6 +438,8 @@ async def generate_response_with_compaction(
                 cache=configured_conversation_cache(),
                 user_id=context.user_id,
                 skip_user_id_check=context.skip_userid_check,
+                endpoint_path=endpoint_path,
+                charge=partial(consume_summarization_tokens, context.user_id),
             ):
                 if isinstance(item, CompactionStartedEvent):
                     yield stream_compaction_event(context.conversation_id)
@@ -443,6 +449,7 @@ async def generate_response_with_compaction(
                         context.client, item.params, item.original_input
                     )
                     context_status = item.context_status
+                    summarization_usage = item.summarization_usage
 
             generator, turn_summary = await retrieve_agent_response_generator(
                 responses_params=responses_params,
@@ -497,6 +504,7 @@ async def generate_response_with_compaction(
             emit_start=False,
             turn=turn,
             context_status=context_status,
+            summarization_usage=summarization_usage,
         ):
             yield event
     finally:

src/constants.py

@@ -344,6 +344,7 @@
 ENDPOINT_PATH_QUERY: Final[str] = "/v1/query"
 ENDPOINT_PATH_STREAMING_QUERY: Final[str] = "/v1/streaming_query"
 ENDPOINT_PATH_RESPONSES: Final[str] = "/v1/responses"
+ENDPOINT_PATH_A2A: Final[str] = "/a2a"
 
 # Context window (tokens) conversation compaction assumes for a model when
 # neither its catalog metadata nor automatic resolution gives one.