Skip to content

feat: make TAC safe to run as N replicas with no shared datastore - #123

Closed
ryanrouleau wants to merge 1 commit into
mainfrom
feat/horizontal-scaling
Closed

feat: make TAC safe to run as N replicas with no shared datastore#123
ryanrouleau wants to merge 1 commit into
mainfrom
feat/horizontal-scaling

Conversation

@ryanrouleau

@ryanrouleau ryanrouleau commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Makes the SDK safe behind a load balancer with no Redis, no database, and no sticky sessions for messaging. Implements TAC Horizontal Scaling Plan in four phases (P0a/P0b/P1/P2 + docs).

Two problems, two answers:

Messaging is now stateless. MessagingChannel has no session store. Each webhook derives a ConversationSession from the payload, config, and one list_participants call it already needed for the self-message check and reconciliation — so any replica serves any webhook.

Voice stays instance-local, correctly — a WebSocket pins a call to one process — so the fix is guaranteed teardown plus a way for the call's other traffic to find it.

Bugs this fixes

Before After
CONVERSATION_UPDATED/CLOSED filtered on locally-tracked state, so with N instances on_conversation_ended usually never fired and the session leaked for the process lifetime rebuilt from Conversation Orchestrator; fires once, on whichever replica Twilio picked
Voice teardown in orchestrated mode _cleanup_connection deliberately skipped _end_conversation, waiting for a CLOSED webhook that lands on a random instance — the socket closed, the session stayed always releases; new on_call_ended hook fires with the live session
Media Streams _call_session_configs written during the TwiML request, read when the WebSocket connects — two requests that can hit different instances — and never evicted for calls that don't connect rides the TwiML as a <Stream> custom parameter (as outbound already did); the remaining bridge is bounded + expiring
Call events (status/AMD/recording) carry only a CallSid, resolve to None on any instance that didn't hold the call TACConfig.instance_public_domain mints every callback URL against this process
Scale-in / redeploy live calls dropped with no callback at all VoiceChannel.aclose() drains, wired to the FastAPI shutdown event

API-call budget

Statelessness is not paid for with extra calls. Per inbound message, pinned by a regression test:

memory_mode Before After
"never" (default) 2 2
"always" 3 4 (get_profile)
"always" + fetch_profile_traits=False — 3

profile_id now comes off the participant list we already fetch — no lookup_profile, and more reliable than the address-based fallback, which infers the identifier type and misses on CHAT and RCS. A conversation's first message is one call cheaper than before (the double list_participants is deduplicated).

New API

  • VoiceChannel.on_call_ended(handler) — fires at WebSocket teardown with the live session. The only place the transcript is still reachable.
  • VoiceChannel.aclose(grace_period=...) / TACFastAPIServer.aclose()
  • TACConfig.instance_public_domain, TwilioMemoryConfig.fetch_profile_traits, TACServerConfig.shutdown_grace_period

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring
  • Release / version bump

Breaking changes

  1. MessagingChannel.get_agent_address(session) — was get_agent_address(conversation_id). The channel no longer holds a session to look up. Affects anyone subclassing MessagingChannel directly.
  2. MessagingChannel.send_response(conversation, ...) — first parameter was named conversation_id and now accepts a ConversationSession too. Positional calls are unaffected; keyword calls need updating. Pass the session and the send costs no extra API call.
  3. Messaging channels reject memory_mode="once" at construction. It caches a recall on a long-lived session, which messaging no longer has; statelessly it would cost the same as "always" with strictly worse relevance, so it fails loudly rather than degrading silently. "always" is the drop-in replacement. Unchanged on Voice, where it's the mode that matters most.

Also worth flagging (not breaking, but a behavior change): in orchestrated voice mode on_conversation_ended now receives a session rebuilt from Conversation Orchestrator, so in-memory state (transcript, application metadata) is gone by then. Use on_call_ended for that.

Checklist

  • Tests added/updated — 1004 passing (was 946). New: tests/test_voice_scaling.py, tests/test_messaging_scaling.py, tests/test_expiring_dict.py, and tests/voice_invariants.py (the assert_no_residual_state teardown invariant, asserted by every voice test that opens a socket). tests/test_memory_mode_once.py moved from SMS to Voice.
  • Documentation updated — new docs/deployment.md; the CLAUDE.md "horizontal scaling limitation" caveat is now a description of the model.
  • Tested E2E — not done, see below.

Not verified against a live account

Two items need a real Twilio account and are the main thing to check before merging:

  1. Address-mode send. The plan called for from = {address, channel} (Actions API mode 3), gated on a live test. I couldn't run it, so send_response prefers the participant id when it has one — which reconciliation always establishes on the inbound path — and falls back to address mode only when there isn't one. That keeps today's verified behavior on the hot path. If the live test passes, this can be simplified to address-only; if it fails, nothing to undo.
  2. What v2-native capture assigns as a participant type. _reconcile_participants' docstring claims it's a v1-bridge-only repair pass; nothing in the repo establishes that. Reconcile is now unconditional (and free — it reuses the fetched participant list and writes nothing on the happy path), so the answer doesn't change behavior, but it would settle a standing unknown in the code's own documentation.

SDK Parity

  • Change is Python-specific (no TypeScript update needed)
  • TypeScript SDK PR created:

Not Python-specific — the TypeScript SDK has the same architecture and the same leak. A parity PR should follow; not created yet.

🤖 Generated with Claude Code

@ryanrouleau @claude
feat: make TAC safe to run as N replicas with no shared datastore
3a2c450 
Messaging and voice needed different answers.

Messaging is now stateless. `MessagingChannel` has no session store at all:
each webhook derives a `ConversationSession` from the payload, config, and one
`list_participants` call it already needed for the self-message check and
reconciliation. Any replica serves any webhook. Previously `CONVERSATION_UPDATED`
was filtered on locally-tracked state, so with N instances `on_conversation_ended`
usually never fired and the session leaked for the process lifetime; it now
rebuilds the session from Conversation Orchestrator and fires on whichever
replica Twilio picked.

Voice state is legitimately instance-local — a WebSocket pins a call to one
process — so the fix is guaranteed teardown instead. `_cleanup_connection` no
longer skips cleanup in orchestrated mode waiting for a CLOSED webhook that may
land elsewhere (the largest leak); it always releases, and a new `on_call_ended`
hook carries the live session, which is the only place the transcript is still
reachable. `VoiceChannel.aclose()` drains live calls at shutdown, wired to the
FastAPI shutdown event.

For a call's out-of-band webhooks (status/AMD/recording, `<Connect action>`),
`TACConfig.instance_public_domain` makes TAC mint every callback URL against
this process, so they return to the replica holding the socket.

No Redis, no database, and no extra API calls in the default configuration —
`memory_mode="never"` still costs 2 calls per message, and the budget is pinned
by a regression test.

Breaking changes:
- `MessagingChannel.get_agent_address(session)` was `get_agent_address(conversation_id)`.
- `MessagingChannel.send_response` takes `conversation` (a session or an id) as
  its first parameter, was `conversation_id`. Positional calls are unaffected.
- Messaging channels reject `memory_mode="once"` at construction. It caches a
  recall on a long-lived session, which messaging no longer has; statelessly it
  would cost the same as `"always"` with worse relevance. Use `"always"`.
  `"once"` is unchanged on Voice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 1, 2026 18:08
Copilot AI reviewed Sep 1, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There is a teardown correctness risk in VoiceChannel._handle_conversation_closed() popping live sessions (potentially preventing on_call_ended on later socket teardown), and the FastAPI server’s voice-domain validation error message is misleading vs the new voice_callback_domain behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates Twilio Agent Connect (TAC) to support horizontal scaling behind a load balancer without sticky sessions or a shared datastore by making messaging channels stateless per webhook and tightening voice lifecycle/teardown semantics (including instance-affinity URL generation and shutdown draining).

Changes:

  • Refactors MessagingChannel to derive ConversationSession per webhook (no in-process session store) and rebuild sessions from Conversation Orchestrator when needed.
  • Adds voice lifecycle improvements: on_call_ended, guaranteed teardown via _release_session, instance-affinity via instance_public_domain, and shutdown draining (VoiceChannel.aclose wired into TACFastAPIServer).
  • Introduces ExpiringDict for bounded, expiring cross-request state and adds extensive tests + deployment documentation.
File summaries
File Description
tests/voice_invariants.py Adds a reusable invariant asserting voice teardown leaves no residual local state.
tests/test_whatsapp_channel.py Updates WhatsApp tests to reflect stateless messaging and CO-rebuilt on_conversation_ended.
tests/test_voice_scaling.py New end-to-end-ish scaling contract tests for voice teardown, CLOSED handling, draining, and instance-affinity.
tests/test_voice_channel.py Aligns existing voice tests with unified teardown + on_call_ended behavior.
tests/test_sms_channel.py Updates SMS tests for stateless sessions, CO-rebuilt CLOSED handling, and session-based send_response.
tests/test_rcs_channel.py Updates RCS tests to match stateless messaging and CO-rebuilt on_conversation_ended.
tests/test_profile_retrieval.py Adjusts expectations for profile trait fetch behavior under stateless messaging sessions.
tests/test_participant_reconciliation.py Refactors reconciliation tests to pass already-fetched participants and a session object.
tests/test_outbound.py Updates outbound tests for “return session to reply with” pattern and new self-message logic.
tests/test_openai_realtime_provider.py Updates Media Streams provider tests to use token-based per-call config propagation and cleanup.
tests/test_messaging_scaling.py New scaling contract tests for stateless messaging + call budget invariants.
tests/test_memory_mode_once.py Moves/reshapes "once" memory-mode coverage to voice-only and enforces rejection on messaging channels.
tests/test_integration.py Updates integration expectations for CLOSED handling now rebuilt from CO rather than local session cleanup.
tests/test_expiring_dict.py Adds tests for bounded, expiring mapping used for cross-request correlation.
tests/test_chat_channel.py Updates chat tests for stateless sessions, participant-based rebuilds, and session-based send_response.
src/tac/utils/expiring_dict.py Adds a bounded, TTL-based mapping for short-lived cross-request state (token correlation, etc.).
src/tac/server/fastapi_server.py Wires TACFastAPIServer.aclose() to FastAPI shutdown; updates voice domain validation to use callback domain.
src/tac/server/config.py Adds shutdown_grace_period to configure voice drain behavior on shutdown.
src/tac/core/tac.py Adds _has_conversation_ended_callback() and gates profile trait fetch behind fetch_profile_traits.
src/tac/core/config.py Adds fetch_profile_traits, instance_public_domain, and voice_callback_domain to support scaling semantics.
src/tac/channels/whatsapp.py Updates get_agent_address signature to accept ConversationSession.
src/tac/channels/voice/twiml.py Uses voice_callback_domain when deriving default WebSocket URL (supports instance pinning).
src/tac/channels/voice/provider.py Introduces provider capability flag for whether CO CLOSED owns on_conversation_ended.
src/tac/channels/voice/media_streams/openai_realtime/provider.py Replaces unbounded per-call config stash with expiring dict + token propagation on TwiML; ensures cleanup uses _release_session.
src/tac/channels/voice/conversation_relay/twiml.py Uses voice_callback_domain for action URL resolution.
src/tac/channels/voice/conversation_relay/provider.py Ensures teardown always releases sessions and properly models CO-owned conversation closure.
src/tac/channels/voice/channel.py Adds on_call_ended, _release_session, CLOSED rebuild path, aclose() drain, and instance-affinity behaviors.
src/tac/channels/voice/init.py Exports new CallEndedHandler.
src/tac/channels/sms.py Updates get_agent_address signature to accept ConversationSession.
src/tac/channels/rcs.py Updates get_agent_address signature to accept ConversationSession.
src/tac/channels/messaging.py Core refactor: stateless session derivation, single participant fetch per message, session-based send_response, and CO-rebuilt CLOSED handling.
src/tac/channels/chat.py Makes agent address derivation session-driven (for per-conversation channelId).
src/tac/channels/base.py Removes shared session storage and centralizes safe on_conversation_ended triggering behavior.
mkdocs.yml Adds the new deployment/scaling document to site navigation.
getting_started/examples/features/dashboard/dashboard.py Updates dashboard mounting to accept app-owned messaging session tracking (channels no longer store messaging sessions).
getting_started/examples/features/dashboard/app.py Implements app-level messaging session tracking and clears it on conversation end for dashboard display.
getting_started/examples/features/chat/app.py Updates example to reply with the callback session to avoid extra lookups.
docs/deployment.md New documentation describing the horizontal scaling model, budgets, hooks, and shutdown drain behavior.
CLAUDE.md Updates repo-level architecture notes to reflect stateless messaging + instance-local voice model.
Review details
  • Files reviewed: 39/39 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/tac/channels/voice/channel.py
Comment on lines +457 to +464
session = self._conversations.pop(conv_id, None)
if session is None:
if not self.tac._has_conversation_ended_callback():
return
session = await self._rebuild_session(conv_id, call_sid)
if session is None:
return
await self._trigger_conversation_ended(session)
Comment thread src/tac/server/fastapi_server.py
Comment on lines +162 to 168
if not self.tac.config.voice_callback_domain:
raise ValueError(
"Voice channel is configured but TACConfig.voice_public_domain is "
"not set. Set it directly or via the TWILIO_VOICE_PUBLIC_DOMAIN env var."
"not set. Set it directly or via the TWILIO_VOICE_PUBLIC_DOMAIN env var. "
"(TACConfig.instance_public_domain also satisfies this, if each replica "
"is directly addressable.)"
)
@wenzhu1587
wenzhu1587 force-pushed the feat/horizontal-scaling branch 2 times, most recently from 0829cda to 3a2c450 Compare October 1, 2026 02:38
This was referenced Oct 1, 2026
Closed
Merged
@wenzhu1587

wenzhu1587 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Heads-up: #132 carries this PR's commit, rebased onto current main (authorship preserved), plus fixes for the gaps listed in §11 of the plan doc in #131. Those fixes are G1, G3, G4, G6, G8, G9 and G10, with G2 and G5 documented. #132 also merges with #128 (multi-sender). It supersedes this PR's diff. If that works for you, review there and close this one.

🤖 Generated with Claude Code

wenzhu1587 added a commit that referenced this pull request Oct 19, 2026
@wenzhu1587 @ryanrouleau @claude
feat: run TAC as N replicas with no shared datastore (supersedes #123) (
2fc69c0 
#132)

* feat: make TAC safe to run as N replicas with no shared datastore

Messaging and voice needed different answers.

Messaging is now stateless. `MessagingChannel` has no session store at all:
each webhook derives a `ConversationSession` from the payload, config, and one
`list_participants` call it already needed for the self-message check and
reconciliation. Any replica serves any webhook. Previously `CONVERSATION_UPDATED`
was filtered on locally-tracked state, so with N instances `on_conversation_ended`
usually never fired and the session leaked for the process lifetime; it now
rebuilds the session from Conversation Orchestrator and fires on whichever
replica Twilio picked.

Voice state is legitimately instance-local — a WebSocket pins a call to one
process — so the fix is guaranteed teardown instead. `_cleanup_connection` no
longer skips cleanup in orchestrated mode waiting for a CLOSED webhook that may
land elsewhere (the largest leak); it always releases, and a new `on_call_ended`
hook carries the live session, which is the only place the transcript is still
reachable. `VoiceChannel.aclose()` drains live calls at shutdown, wired to the
FastAPI shutdown event.

For a call's out-of-band webhooks (status/AMD/recording, `<Connect action>`),
`TACConfig.instance_public_domain` makes TAC mint every callback URL against
this process, so they return to the replica holding the socket.

No Redis, no database, and no extra API calls in the default configuration —
`memory_mode="never"` still costs 2 calls per message, and the budget is pinned
by a regression test.

Breaking changes:
- `MessagingChannel.get_agent_address(session)` was `get_agent_address(conversation_id)`.
- `MessagingChannel.send_response` takes `conversation` (a session or an id) as
  its first parameter, was `conversation_id`. Positional calls are unaffected.
- Messaging channels reject `memory_mode="once"` at construction. It caches a
  recall on a long-lived session, which messaging no longer has; statelessly it
  would cost the same as `"always"` with worse relevance. Use `"always"`.
  `"once"` is unchanged on Voice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(voice): take a rebuilt session's call_sid from the VOICE participant

CO's CONVERSATION_UPDATED payload has no top-level channelId, so rebuilt
sessions always got call_sid=None. Read it from the VOICE participant
address's channelId instead, and make the fixture match the documented
payload. (G10)

Includes review fixes:
- fix(voice): prefer the customer's VOICE channelId for a rebuilt call_sid

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(voice): keep a live call's session when CO closes it mid-call

A CLOSED webhook for a call still live on this instance popped its
session, so on_call_ended never fired and the call's later prompts were
dropped. Fire on_conversation_ended now from a snapshot and leave the
session for teardown to release. (G1)

Updates two pre-existing tests in test_voice_channel.py that asserted
the old behavior (session removed on CLOSED while the call was still
live), which this change intentionally replaces.

Includes review fixes:
- fix(voice): document mid-call CLOSED on process_webhook; test end_call and raising callback

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(voice): resolve a call's session by CallSid on any instance

Add VoiceChannel.resolve_conversation_session_by_call_sid: the live
session when held locally, otherwise (orchestrated ConversationRelay) an
identity-only session rebuilt from Conversation Orchestrator, preferring
the active conversation. Lets CallSid-only call events find the
conversation without per-pod routing. (G4)

Includes review fixes:
- fix(voice): document rebuilt-session limits and cost; parse created_at as datetime

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* refactor(voice): drop GPT-Live's per-token expiry timers

The shared Media Streams base now stashes session configs in a bounded
ExpiringDict, so GPT-Live's asyncio sleep task per outbound token was a
second TTL for the same entries. Unclaimed entries now expire on the
shared default, same as OpenAI Realtime. (G6)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs: document mid-call CLOSED behavior and the CallSid fallback

Also tidies a test docstring.

Includes review fixes:
- docs(voice): note the per-replica conversation id after a mid-call close

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(messaging): deprecate memory_mode="once" instead of rejecting it

Messaging channels accepted "once" on main, so narrowing the config to
never/always and raising in the constructor broke existing apps at
startup. Accept it again, emit a DeprecationWarning plus a logged
warning, and run it as "always" (memory stays on, with each message's
text as the query). Removal is planned for 3.0; Voice is unchanged.
(G9)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: add shared ISO 8601 helpers for CO timestamps

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat: share one participant lookup across channels per CO webhook

Adds TAC._list_participants_shared (in-flight + 60 s cache, failures not
cached) and analytics.analytics_enabled(), for CO-driven lifecycle
analytics.

Includes review fixes:
- fix: evict failed shared participant lookups via a done-callback

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat(analytics): report messaging lifecycle from CO webhooks

Conversation Started comes from PARTICIPANT_ADDED for the customer and
Conversation Ended from CONVERSATION_UPDATED/CLOSED (duration = CO's
updatedAt - createdAt), each channel reporting its own channel type. A
CLOSED webhook's participant lookup is shared across channels. (G3)

Includes review fixes:
- fix(analytics): log unparseable PARTICIPANT_ADDED; document CLOSED reporting

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat(analytics): report orchestrated voice lifecycle from CO webhooks

With a CO conversation behind the call (orchestrated ConversationRelay),
Conversation Started/Ended now come from PARTICIPANT_ADDED and CLOSED,
like messaging, with CO's duration; the call's session start and
teardown report them only in relay-only and Media Streams modes. The
CLOSED rebuild shares the cross-channel participant lookup. (G3)

Includes review fixes:
- fix(voice): keep G4's CallSid lookup fresh; tidy lifecycle comments
- test(analytics): pin voice sharing the CLOSED lookup; fix rebuild log text

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* docs: describe CO-driven lifecycle analytics

Includes review fixes:
- docs: note telemetry lookup cost and at-least-once lifecycle reporting

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(voice): require instance_public_domain for per-call Media Streams configs

Punts the G2 fix. A per-call session_config lives on the instance that
served the TwiML or placed the call, so with several instances the
stream can land elsewhere and fall back to default_session_config.
Both providers now claim the stashed config through one shared helper
that logs a warning naming instance_public_domain when the token isn't
held locally. docs/deployment.md lists what needs instance affinity and
corrects the per-pod guidance (a headless Service isn't reachable from
Twilio).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: support conversation metadata in the CO client

Adds metadata to ConversationRequest/ConversationResponse and to
create_conversation / create_or_reuse_conversation, plus get_conversation
and patch_conversation_metadata (PATCH merge). (G8)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat: fit app metadata to CO conversation-metadata limits

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat: add ConversationSession.conversation_metadata()

Reads a conversation's Conversation Orchestrator metadata: answered from
what the session already knows, otherwise via a channel-supplied loader
whose result is kept for the turn; failures return {} and retry. (G8)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat(messaging): carry outbound metadata across turns

Outbound initiation writes the metadata that fits CO's limits onto the
conversation (at creation; PATCH merge on reuse). A bounded, per-instance
cache shares each conversation's session.metadata across its turns, as on
main, and session.conversation_metadata() reads the CO-stored metadata on
any instance, fetching only when it isn't known locally. CLOSED carries
the payload metadata and evicts the cache entry. (G8)

Includes review fixes:
- fix(messaging): keep fetched conversation metadata per turn; fetch on bare CLOSED

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* docs: describe conversation metadata across turns

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix: tolerate non-string CO metadata values when parsing conversations

ConversationResponse.metadata is typed dict[str, str] | None, which is strict in Pydantic v2. A conversation whose metadata another tool wrote with a non-string or null value (e.g. {"n": 1, "x": None}) raised ValidationError on every parse: the CLOSED webhook (losing on_conversation_ended and analytics), list_conversations (voice resolves by CallSid with it), update_conversation and get_conversation. The field did not exist before G8, so this was a regression from main.

Add a before-validator that keeps only str/str entries and turns a non-dict value into None. ConversationRequest stays strict because TAC builds it itself.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(messaging): consistent CLOSED metadata; document accessor freshness

A CLOSED payload without metadata now uses the cache entry's co_metadata when this instance wrote it, instead of wiring a fetch. session.metadata for on_conversation_ended no longer depends on the replica: payload metadata, then local entries over it, then the rebuilt session's fields. Document that conversation_metadata() answered from local writes does not see later changes made elsewhere, and note the outbound reuse PATCH sets direction.

Includes review fixes:
- docs(messaging): don't overclaim CLOSED metadata consistency

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(messaging): pin metadata call budgets; docs and log tidy

Pin exact call lists for outbound create, outbound reuse and an inbound turn that reads conversation_metadata() on a non-initiating instance. Correct the docs cost sentence, and stop logging the CO response body from create_conversation when metadata was sent (CO may echo the value); log the status code instead.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* docs: describe sender selection with multiple senders per channel

After rebasing onto #128: a reply goes out from the sender the customer
messaged (webhook recipient, else the one the conversation's participants
hold), now that each message builds its own session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): detach the mid-call CLOSED snapshot from the live session

The snapshot passed to on_conversation_ended for a call still live on
this instance only copied metadata's top level, so nested values and the
session's mutable models were shared: clearing metadata["transcript"] in
the callback cleared the live transcript, and the live call's later
appends changed a snapshot the callback kept. Deep-copy the
callback-visible data (metadata, profile, author/agent info, pending
handoff, cached memory) and give the snapshot its own cache_lock instead
of the live call's. A metadata value that can't be copied is shared
rather than failing the hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): drain waits for calls in setup and for end-of-call hooks

aclose() treated an empty session store as a finished drain. But an
admitted WebSocket can still be in setup (e.g. the background CO lookup)
with no session yet, and create one after aclose() returns; and
_release_session pops the session before awaiting on_call_ended, so the
drain could return while that callback was still archiving.

Track admitted WebSocket handlers from admission to completion and count
in-flight releases, and wait on both. After the grace period, cancel
handlers still running and give their teardown up to 5 s to fire the
end-of-call hooks, then force-release whatever sessions remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): force-close leftover calls through the provider at shutdown

After the drain's grace period, aclose() released leftover sessions at
the channel only: on_call_ended fired, but the WebSocket stayed open and
the provider's socket registry, session-manager state, stream task,
model connection and call state survived.

Add VoiceProvider._force_close_call(): ConversationRelay closes the
WebSocket and runs _cleanup_connection; the Media Streams providers close
the Twilio socket and run _cleanup_call (model socket, call state,
stashed config). The base version only releases the channel session, for
providers with no transport state. Both cleanups are now idempotent, so a
stuck handler finishing after the force-close doesn't fire the hooks or
"Websocket Disconnected" twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): take the VOICE customer when a conversation spans channels

A conversation grouped across channels can list an SMS- or CHAT-only
CUSTOMER before the caller. The CLOSED rebuild and CallSid lookup then took
that customer's profile, dropped author_info and preferred the agent's
CallSid; ConversationRelay setup took the wrong profile_id. Select the
CUSTOMER that has a VOICE address, as the messaging channels already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): recognize every configured number as TAC's own address

_is_own_co_address checked only the default phone_number, so an UNKNOWN
VOICE participant at another configured sender counted as a customer and
reported an extra Conversation Started. Check the phone_numbers allowlist,
as the messaging channels already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): keep inherited stream parameters when adding the session-config token

The token was added to the per-call TwiML options' custom_parameters alone.
The builder replaces whole fields, so that per-call dictionary dropped any
parameters inherited from default_twiml_options or host_twiml_options. Add
the token to the parameters the builder would resolve instead, through one
helper shared by inbound and both providers' outbound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: rename the scaling guide to horizontal-scaling.md

The page covers running TAC as several instances, not deployment in
general. Rename it before its first publish, while no links depend on it,
and use the same name for its heading and site navigation entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: correct what instance_public_domain pins and the mid-call close

instance_public_domain pins status, AMD and recording callbacks only on
calls TAC places; inbound call events come from the number's own
configuration and still reach the load balancer. A mid-call CLOSED fires
from a live snapshot only on the replica holding the call, and other
replicas resolve the call to the new conversation only once CO has
started one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: tighten the scaling guide's remaining imprecise claims

- The call budget counts turns that get a reply; one without makes no
  POST /Actions.
- profile_id comes off the participant list once CO has resolved it;
  until then a memory fetch falls back to the address lookup.
- Add the 128-character metadata key limit.
- Only Media Streams records a transcript; on_conversation_ended fires on
  the instance holding the call outside orchestrated mode; only
  ConversationRelay has a text send_response.
- VoiceChannel.initiate_outbound_conversation: both Media Streams
  providers place outbound calls too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(messaging): keep get_agent_address(conversation_id) backward compatible

The stateless rewrite changed MessagingChannel.get_agent_address to take a
session, which broke callers passing an id and subclasses overriding the old
signature. Restore main's get_agent_address(conversation_id). TAC's own paths
now call a private _agent_address(session) that defaults to
get_agent_address(session.conversation_id), so an override of the old
signature keeps receiving ids; the built-in chat channel overrides it to read
channelId off the session, and answers get_agent_address by id from what this
instance cached, as on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(messaging): keep send_response's conversation_id parameter name

The first parameter was renamed to conversation when send_response started
accepting a session, which broke keyword calls passing conversation_id=.
Name it conversation_id again, typed str | ConversationSession, so both
positional and keyword calls from main keep working and a session can still
be passed to skip the participant lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): give orchestrated on_conversation_ended main's session again

On main, an orchestrated ConversationRelay session stayed live until CO's
CLOSED, so on_conversation_ended got the full session, metadata included,
and end_call() fired it straight away. The stateless rewrite released the
session at hangup and rebuilt an identity-only one on CLOSED, and deferred
end_call() to CLOSED.

Keep each ended orchestrated session in a bounded per-instance store
(24 h, 10k) until its CLOSED, which then fires with that session on the
instance that held the call; other instances, and expired entries, still
rebuild from CO. end_call() fires the hook now and marks the kept session
so that instance's CLOSED doesn't refire. A conversation CO closed mid-call
is neither kept nor refired at teardown. The bound replaces main's leak of
sessions whose CLOSED never came.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: describe messaging on_conversation_ended under several replicas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(messaging): answer later turns from cached participants when the lookup fails

On main a later turn reused its in-memory session, so a failed participant
lookup didn't stop it being answered. The stateless rewrite fetched
participants every turn and dropped the message whenever that failed.

Keep the last reconciled participants (customer, agent, profile_id) in the
per-instance conversation cache, and when the lookup fails answer from them
with a warning. With nothing cached, or when the webhook names a different
sender than the cached one, the message is still dropped, as main dropped a
first message. The happy path is unchanged and costs no extra calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(messaging): warn when CO rejects a conversation created with metadata

Outbound now sends conversation metadata at creation, which main never did,
and the client logs no response body when it does. On a 4xx at creation,
log a warning naming the metadata keys (not values) as a possible cause;
the error still propagates as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(config): say which callbacks instance_public_domain pins

Status, AMD and recording callbacks are pinned only on calls TAC places;
an inbound call's come from the number's own configuration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(messaging): stop storing direction in conversation metadata

TAC never reads `direction`; storing it on the CO conversation cost apps
one of CO's 8 metadata keys and, on reuse, overwrote conversations that
began inbound. Only app metadata is written now. session.metadata still
carries `direction: outbound`, as on main.

With no app metadata, creation sends none (as main did), a reused
conversation isn't patched, and a rejected creation no longer names
metadata as a possible cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(messaging): keep nothing between turns

TAC shouldn't hold app state across turns, and a per-instance cache made
session.metadata carry over on one replica but not another. On messaging a
session now lasts one turn; apps keep cross-turn state themselves, keyed by
conversation_id. Outbound metadata stays readable on any replica through
conversation_metadata().

Removed with the cache:
- the last-known-participants fallback: a failed list_participants now
  drops the turn (on_error, dropped_inbound) on every replica alike
- CO metadata copied into session.metadata on CLOSED; read it through
  conversation_metadata(), answered from the payload when it has it
- chat's get_agent_address(conversation_id) channelId, which came from the
  cache; TAC's own paths read it off the turn's session

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(voice): leave shutdown draining to the platform

On SIGTERM, uvicorn closes every open WebSocket with 1012 before the app's
shutdown event runs, so VoiceChannel.aclose()'s grace period never kept a
call alive; the handler's guaranteed teardown already fires on_call_ended
on that disconnect. Keeping calls through a rollout needs SIGTERM delayed,
which only the platform can do.

Removes VoiceChannel.aclose(), TACFastAPIServer.aclose() and its shutdown
hook, TACServerConfig.shutdown_grace_period, the providers'
_force_close_call, and the admission/handler-task tracking only the drain
used. None of it shipped on main. The scaling guide now gives platform
guidance instead: fail readiness, delay SIGTERM with preStop, size the
termination grace period.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(chat): recover channelId when replying by conversation id

send_response(conv_id) rebuilds the session from list_participants, and
Chat's required channelId only arrived on the inbound webhook, so a reply
by id raised on Chat where it worked on main. The rebuilt session now takes
channelId from the customer's or agent's CHAT participant address. Without
one, the error says to pass the ConversationSession.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: document TWILIO_INSTANCE_PUBLIC_DOMAIN

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(voice): keep nothing after a call's teardown

Drop VoiceChannel._ended_sessions. It only restored the full session to
on_conversation_ended when CLOSED reached the replica that held the call;
everywhere else the hook already got a session rebuilt from CO.
on_call_ended is the hook that sees the live session.

end_call() now releases the call like any hangup: on_call_ended fires
right away and, when orchestrated, on_conversation_ended fires once on
the CLOSED that follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(readme): drop the end-of-conversation lookup note from Telemetry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: turn the horizontal scaling page into a getting-started guide

Move docs/horizontal-scaling.md to getting_started/horizontal_scaling.md,
where the other user guides live, so docs/ stays the API reference.
Rewrite it as task-ordered steps for running TAC on several replicas,
leaving the design rationale to CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: describe webhook retries instead of requiring idempotent handlers

TACFastAPIServer acks webhooks before the handler runs, so Twilio only
retries on failures and a cross-replica duplicate reply is rare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: remove the multi-replica guide for now

Drop the links to it; TWILIO_INSTANCE_PUBLIC_DOMAIN stays documented in
the README, the getting-started env var list and .env.example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(chat): send channelId when lazily adding the AI_AGENT participant

Reconciliation rebuilt the agent address from session.ai_agent_info, which
drops the channelId that ChatChannel._agent_address supplies. Conversation
Orchestrator requires channelId on a CHAT address, so the POST failed and the
message was dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Ryan Rouleau <ryan@ryanrouleau.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ryanrishi ryanrishi Awaiting requested review from ryanrishi ryanrishi is a code owner
@xinghaohuang91 xinghaohuang91 Awaiting requested review from xinghaohuang91 xinghaohuang91 is a code owner
@wenzhu1587 wenzhu1587 Awaiting requested review from wenzhu1587 wenzhu1587 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ryanrouleau @wenzhu1587