[agentserver] Defer terminal span/storage write off streaming last-byte path (warm-path latency) by harsheet-shah · Pull Request #49040 · Azure/azure-sdk-for-python

Harsheet Shah (harsheet-shah) · 2026-09-16T11:47:14Z

Summary

For the in-process (non-resilient) streaming path, the Responses endpoint performs the terminal storage write synchronously before emitting the final SSE bytes (response.completed / terminal event). The last byte the client receives is therefore gated on a storage HTTPS round-trip, adding that write's latency to every streamed response's tail even though the response content is already fully computed.

Changes

azure-ai-agentserver-responses — terminal resolution is split into a no-I/O part and a deferred-I/O part so the terminal event is emitted first and the storage write happens after the wire is closed:

The canonical record is stamped terminal and the final event is emitted to the stream immediately (no storage I/O on the last-byte path).
The terminal persist is deferred and executed in a finally after the stream closes; eviction is gated so a GET during the deferral window still serves the in-memory record (no 404 / stale read).
A rare terminal-write failure is stamped on the canonical runtime-state record and surfaces via a later GET (failed), not on the already-closed stream.
Scope is limited to the in-process / non-resilient streaming path; the resilient path is unchanged.
Shutdown-drain now waits on any live execution task so a deferred write isn't lost on graceful shutdown.

Backwards compatibility

Default-on for the in-process streaming path only. Response content and event ordering are unchanged; the only difference is the terminal storage write no longer blocks the final byte. The resilient path is untouched. A terminal-write failure that previously surfaced inline now surfaces via a subsequent GET.

Tests

New contract tests (test_async_terminal_persist.py): terminal-emitted-before-slow-write, GET-during-deferral serves the in-memory record, and deferred-write failure surfaces via GET. Full responses suite: 1462 passed, 85 skipped.

…ath latency) For in-process (non-resilient) store=true streaming responses, emit the terminal response.completed/failed event and close the wire stream BEFORE performing the terminal provider write. This moves the terminal storage round-trip off the client's last-byte (TTLB) path. - Split terminal resolution into a no-I/O part (snapshot/transition/emit) and a deferred I/O part run after the wire stream closes. - GET during the deferral window serves the completed snapshot from the in-memory runtime state (record retained until the deferred write completes). - A rare terminal-write failure now surfaces on a later GET (record stamped storage_error) instead of on the SSE stream; the client always sees response.completed on success. - The deferred persist targets the canonical runtime_state record so a persistence failure is correctly reflected on GET. - Shutdown drain waits on any in-flight execution task so deferred persists complete on graceful shutdown. The resilient path (buffer-then-persist-then-yield) is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d900395-13d0-4698-bf9d-f6670f9e545c

azure-pipelines · 2026-09-16T11:47:50Z

Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

🟡 Changes recommended

Empty-handler streams can return 404 during deferral, and shutdown can still miss the deferred write task.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Defers terminal persistence until after in-process streams close, reducing tail latency.

Changes:

Splits terminal resolution from persistence I/O.
Retains runtime records during deferred persistence.
Adds contract tests and shutdown-drain handling.

File summaries

File	Description
`test_persistence_failure.py`	Updates deferred-failure expectations.
`test_async_terminal_persist.py`	Adds deferred-persistence contract tests.
`CHANGELOG.md`	Documents behavior change.
`_orchestrator.py`	Implements deferred terminal persistence.
`_endpoint_handler.py`	Expands shutdown task draining.

Review details

Files reviewed: 5/5 changed files
Comments generated: 3
Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

… GET & shutdown In-process store=True streaming fallback (deferred terminal write): - Register the synthesized record before emitting the terminal. Previously, when no canonical record existed (handler produced a terminal without a create event), the fallback used _make_ephemeral_record but never added it to runtime_state, so a GET during the deferred-persist window returned 404 and a stamped persistence failure was unreachable. Now add() it up front. - Track the draining task on _PipelineState and attach it to the canonical record at registration (_register_bg_execution), not after the handler drains. handle_shutdown drains records whose execution_task is live; attaching it up front prevents the shutdown wait loop from returning before the deferred terminal write completes (avoiding loop-teardown cancellation of the write). - Add contract test test_deferred_write_reaches_provider_after_release: releases the gate and asserts the deferred write resumes AND the terminal is durably persisted in the backing provider (eventual-persistence guarantee), which the prior gate-release-only assertion did not verify. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d900395-13d0-4698-bf9d-f6670f9e545c

Copilot

🟡 Changes recommended

Deferred persistence can be lost during foreground shutdown and can race with deletion to recreate stored responses.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_orchestrator.py:3182

The deferred write can race a successful DELETE after the terminal closes. In the no-first-event fallback provider_created is false, so DELETE removes/tombstones the runtime record and finds no provider row, then this callback executes create_response and recreates the deleted response in storage; the in-process tombstone only hides it until restart. Coordinate DELETE with the live execution task (or otherwise serialize deletion and deferred persistence) so deletion always wins.

                        await state.deferred_terminal_persist()

Files reviewed: 5/5 changed files
Comments generated: 1
Review effort level: Balanced

…t record _finalize_stream Path B builds a fresh ResponseExecution that overwrites the runtime_state record carrying state.execution_task (foreground store=True, and the in-process fallback whose finally funnels here). The replacement dropped the task, so handle_shutdown saw execution_task is None and could complete shutdown while the deferred terminal provider write was still in flight. Copy state.execution_task onto the replacement before add(); add a regression test asserting the live record retains a non-done execution_task during the deferral window. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d900395-13d0-4698-bf9d-f6670f9e545c

Copilot

🟡 Changes recommended

A shutdown race can still cancel fallback execution before terminal persistence.

Get a fresh assessment by requesting another Copilot review.

Review details

Files reviewed: 5/5 changed files
Comments generated: 1
Review effort level: Balanced

+                # so attaching it up front — not after the handler drains —
+                # prevents the shutdown wait loop from returning before the
+                # deferred terminal write below completes.
+                state.execution_task = asyncio.current_task()


github-actions · 2026-09-17T05:35:04Z

[Pilot] PR Pipeline Failure Analysis

What failed

Azure Pipelines build 6847244 (python - pullrequest) failed on two independent checks against azure-ai-agentserver-responses:

CSpell (Validate/BuildAnalyze step) — unknown word in a new test file.
Pylint (Analyze step) — 4 style/quality violations in _orchestrator.py, including one real logic warning (not-callable).

Relevant pipeline output

sdk/agentserver/azure-ai-agentserver-responses/tests/contract/test_async_terminal_persist.py:8:40 - Unknown word (TTLB)
Spelling errors detected. To correct false positives or learn about spell checking see: https://aka.ms/azsdk/engsys/spellcheck
PowerShell exited with code '1'.

Module azure.ai.agentserver.responses.hosting._orchestrator
azure/ai/agentserver/responses/hosting/_orchestrator.py:3194: [C0301(line-too-long)] Line too long (130/120)
azure/ai/agentserver/responses/hosting/_orchestrator.py:1304: [R0902(too-many-instance-attributes), _PipelineState] Too many instance attributes (13/10)
azure/ai/agentserver/responses/hosting/_orchestrator.py:3190: [E1102(not-callable), _ResponseOrchestrator._live_stream._resilient_stream_fallback] state.deferred_terminal_persist is not callable
azure/ai/agentserver/responses/hosting/_orchestrator.py:3083: [R0915(too-many-statements), _ResponseOrchestrator._live_stream] Too many statements (56/50)
azure-ai-agentserver-responses main package exited with linting error 26.

Recommended next steps

CSpell: TTLB (likely used in a docstring/comment for "time to last byte", matching the PR's warm-path latency goal) is not in the project dictionary. Either rename to an already-accepted term or add TTLB to .vscode/cspell.json (words list).
Pylint E1102(not-callable): at _orchestrator.py:3190, state.deferred_terminal_persist is invoked but pylint infers it may not be callable (e.g. typed as Optional[Callable] or assigned None on some paths). Verify the attribute's type/initialization on _PipelineState and either add a type-safe callable default/guard or an Optional narrowing check before calling it — this is directly relevant to the new deferred-persist logic this PR introduces.
Pylint C0301: shorten the long line at _orchestrator.py:3194 to ≤120 chars.
Pylint R0902/R0915: _PipelineState (13/10 attributes) and _ResponseOrchestrator._live_stream (56/50 statements) exceed complexity thresholds. Split _live_stream into helper methods (e.g. extract the deferred-persist/finally block into its own method) and/or group related _PipelineState fields into a small dataclass, or add a targeted # pylint: disable=too-many-instance-attributes / too-many-statements justification if refactor isn't feasible.
See https://aka.ms/ci-fix

Automated fix: Requested

Generated by Pipeline Analysis Next Steps · auto · 124.7 AIC · ⌖ 1.96 AIC · ⊞ 9.2K · ◷

Copilot AI balanced review requested due to automatic review settings September 16, 2026 11:47

Harsheet Shah (harsheet-shah) requested review from Ravi Pidaparthi (RaviPidaparthi), Shiva S (Shivakishore14), Ankit Sinha (ankitbko) and Pranav Pandit (vangarp) as code owners September 16, 2026 11:47

github-actions Bot added the Hosted Agents sdk/agentserver/* label Sep 16, 2026

Copilot started reviewing on behalf of Harsheet Shah (harsheet-shah) September 16, 2026 11:47 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Copilot AI review requested due to automatic review settings September 17, 2026 04:33

Copilot started reviewing on behalf of Harsheet Shah (harsheet-shah) September 17, 2026 04:34 View session

Copilot AI reviewed Sep 17, 2026

View reviewed changes

Comment thread ...erver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_orchestrator.py

Copilot AI review requested due to automatic review settings September 17, 2026 04:56

Copilot started reviewing on behalf of Harsheet Shah (harsheet-shah) September 17, 2026 04:57 View session

Copilot AI reviewed Sep 17, 2026

View reviewed changes

Navigation Menu

Uh oh!

[agentserver] Defer terminal span/storage write off streaming last-byte path (warm-path latency) - #49040

[agentserver] Defer terminal span/storage write off streaming last-byte path (warm-path latency)#49040
Harsheet Shah (harsheet-shah) wants to merge 3 commits into
mainfrom
user/harsheetshah/warm-path-responses-improvements

Harsheet Shah (harsheet-shah) commented Sep 16, 2026

Uh oh!

azure-pipelines Bot commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Sep 17, 2026

What failed

Recommended next steps

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Harsheet Shah (harsheet-shah) commented Sep 16, 2026

Summary

Changes

Backwards compatibility

Tests

Uh oh!

azure-pipelines Bot commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

github-actions Bot commented Sep 17, 2026

What failed

Recommended next steps

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants