Skip to content

Sessions: attempts on the explicit recovery VM, with one cancellation rule - #316

Merged
igalshilman merged 11 commits into
mainfrom
claude/sessions-cancellation-once
Nov 17, 2026
Merged

igalshilman merged 11 commits into
mainfrom
claude/sessions-cancellation-once

Conversation

@igalshilman

@igalshilman igalshilman commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Self-contained on main: this carries the commit of #314 (c5e1f7d, "Sessions: attempts on the explicit recovery VM, without the selector", unchanged and with its author) and ten commits on top of it. Merging this one brings in both; #314 can then be closed. Nothing here needs #314 merged first.

Read this first: two regressions from main that come with the VM

1. Not fixed here: a cancellation does not reach an invocation that is suspended on a call

Restate shuts down an attempt that waits for a call, and starts the next one when the call answers. On the explicit recovery VM the suspension names the calls alone, so cancelling the invocation meanwhile does nothing until the call answers, and nothing at all if it never does.

Run against a real server, on main and here:

  • A session object's invocation, shut down while it waits for a call, then cancelled: on main it ends at once; here it stays suspended, with or without explicit cancellation.
  • An agent's turn that waits for an approval, shut down, then stop: on main the turn ends; here stop answers 200 and the turn stays suspended.

It is in shared-core 8.7.0 (TODO the cancellation is ignored for now), so #314 has it too, and so does @restatedev/sessions. The fix is on shared-core's branch claude/cancel-wakes-a-shut-down-attempt: the suspension names the cancellation, unless the invocation was cancelled already. Built against that branch, the four tests below pass, with the rest of the suite. It needs a shared-core release and a bump here.

The tests are in this PR as strict expected failures (tests/sessions_suspended_server.py, tests/agents_suspended_stop_server.py): they start failing, and lose their marker, with the shared-core that fixes them.

2. Fixed here, in 14a0604: an agent's turn longer than the inactivity timeout restarted forever

On main a turn's attempt is kept alive by its working run, which #314 removes with the selector. #314 heartbeats only while an annotation is open, and only the OpenAI agents open one. So anything else that works outside Restate for longer than the inactivity timeout (a minute, by default) was shut down and restarted, over and over: a model request of Pydantic AI, ADK or LangGraph, a tool, a summarizer.

One test on both (tests/agents_long_work_server.py: a model that takes 3 s, an inactivity timeout of 1 s): on main the model is asked once and the turn answers; on #314 the turn never answers, and the model was asked 20 times in 20 seconds.

An attempt now heartbeats for as long as its handler runs, when either holds: nothing of Restate's is outstanding for the handler (the rule of @restatedev/sessions), or an annotation is open (as #314 has it, for work beside a call). With an answer of Restate's outstanding and no annotation it stays silent, so a wait of days holds no process.

This changes a choice #314 made on purpose, so it is its own commit: one revert removes it. #314's test that un-annotated work is shut down and restarted still holds when the heartbeat is slower than the inactivity timeout, and is renamed for that.

The rest, on top of #314

It brings Python's session objects to the cancellation rule that sdk-sessions#39 merged for TypeScript, with that package as the reference.

Shared core 8.7.0

#314 builds against a branch of shared-core, "until shared-core releases it". 8.7.0 has the VM, and is what @restatedev/sessions builds against. The override is gone, and the two SDKs run the same VM.

One cancellation rule

For a handler without explicit_cancellation, #314 had two problems:

  • A cancellation with nothing pending was dropped. The driver failed the awaits pending when the signal came, and nothing else. A handler working outside Restate, or one whose ctx.db call was under way, never saw it; neither did the attempt that follows a cancelled one. Against a real server, a handler that cancels itself in its transaction and then compensates saw 0 cancellations.
  • With a journal codec, a commit could land and report as cancelled. There the database's store goes through Python, and a cancellation failed its reads and the ack of a commit whose writes were already out.

The rule now:

  • One carry, in the driver. A cancellation that failed nothing the handler awaits goes to its next await: a result of Restate's, or a call of ctx.db. Never an invocation id.
  • ctx.db takes it where the handler calls it (Attempt.asked): its statements, its transaction(), and a schema it ensures — before the call issues anything. What the session does for itself does not pass through there: the open, the object's schema, the sends a crashed attempt left, emptying the outbox after a commit, the save at the end.
  • The store's reads and acks are left to settle, on both stores.

So a ctx.db call that raises the cancellation did nothing — no write, no commit, no sends — and one already under way runs to its end. Database's docstring states the contract; none was written down for session objects before.

Three more places that differed from @restatedev/sessions

  • A call whose invocation id had not come yet was never cancelled. A call counted as done once its result was settled, and the cancellation had just settled it with its own failure, so the step that cancels calls skipped it and the callee ran on. A call is now over when Restate answers it; the handler's end waits for the id.
  • Once the attempt was over, a carried cancellation still made ctx.db raise a terminal 409, where the handler's awaits of Restate raise the attempt's end. The same fix is in sdk-sessions#41.
  • A dead attempt's model request ran on to its end, beside the one the next attempt made, and was paid for twice. Stop.guard let go of its work when the handler's task was cancelled. With a model that outlasts the inactivity timeout twice and then answers: three requests started and all three completed; now two are cancelled and one completes.

Tests

  • tests/sessions_wire.py (new): Restate's side of the service protocol played in process, ported from the TypeScript harness. The driver was only tested against a real server, where the moment a cancel signal arrives cannot be chosen. It starts an attempt with only the database's head, as an invocation does, so the store's reads reach it; and what follows a cancel signal is held until the attempt has acted on the signal, with no clock involved.
  • tests/sessions_driver.py (new, both stores, 48 cases): where a cancellation lands, and when an attempt heartbeats.
  • tests/sessions_transaction_cancel.py (new): a transaction under a cancellation either committed, with each of its sends made once, or did nothing, and says which. Two sweeps deliver the cancel signal before each thing the attempt asks Restate for, and read the rows back through a second attempt.
  • tests/sessions_saga.py (new): a saga that keeps its progress in ctx.db, cancelled at each step, crashed in the middle of its compensation, with a step that fails, and shut down during a long step.
  • Server tests: the twin of sdk-sessions' cancelledCommit; local work that keeps its attempt; an agent whose model outlasts the inactivity timeout; a dead attempt's model request; and the four expected failures above.

The driver tests fail against each of these wrong implementations, tried one at a time: no carry; a carry set although an await was failed; the outbox emptied after a commit taking the carry; a carry set for a handler with explicit cancellation; a cancellation failing the store's reads, key listing or acks; a call left running because its invocation id came after the cancellation; a carry delivered after the attempt ended; no heartbeat without an annotation; a heartbeat while an answer is outstanding.

Run locally: 504 fast tests, ruff, pyright, mypy; and, against a local Restate 1.8.0, the server-marked tests of sessions_harness, explicit_recovery, explicit_cancellation and the agents' (57 passing, 4 expected failures).

Not changed here

🤖 Generated with Claude Code

https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV

slinkydeveloper and others added 4 commits November 16, 2026 13:48
A session object's attempts no longer run on the SDK's journaled context with a selector
patched in to fake one await point over do_progress: they run on shared-core's
ExplicitRecoveryVM (restatedev/sdk-shared-core#118), which never replays and has no await
point at all.

- src/explicit_recovery.rs: PyExplicitRecoveryVM. Commands return handles, results come
  back as notifications; those of the database's own handles go straight to the Rust store.
- sessions/_driver.py: one attempt. One input pump, the only reader of the body; a tick
  after every call into the VM writes the output, settles the futures, and notices the end
  of the attempt (a shutdown when Restate closes the input, a failure, or the end).
  Cancellation: the VM's is_cancelled(), checked at every tick. An open annotation sends
  heartbeats, so a long model request is not shut down for inactivity.
- The store over the VM syncs a commit with one acked write of the database's head
  (KvStore::put_and_sync), instead of an empty run.
- Handler.attempt_runner: the server hands a session object's request to it.
- The selector and its working run (#312) are gone: nothing suspends on the cancel signal.
- shared-core is patched to the PR's branch until it is released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x.db call did nothing

On the explicit recovery VM a cancellation failed only the awaits pending in
the driver when it came. One that arrived with nothing pending was dropped:
a handler working outside Restate, or one whose database call was under way,
never saw it, and neither did the attempt that follows a cancelled one. With
a journal codec, where the database's store goes through Python, it failed
the store's reads and the ack of a commit instead, so a commit could land
and report as cancelled.

The driver now carries a cancellation that failed nothing the handler
awaits to its next await, once: a result of Restate's, or a call of
`ctx.db`. An invocation id is never the one it is carried to, as it is not
what a cancellation fails.

`ctx.db` takes the carry where the handler calls it (`Attempt.asked`):
its statements, its `transaction()`, and a schema it ensures, before the
call issues anything. What the session does for itself does not pass
through there: the open, the object's schema, the sends a crashed attempt
left, emptying the outbox after a commit, the save at the end. And the
store's reads and acks are left to settle on both stores.

So the rule is one rule, the same as @restatedev/sessions: a `ctx.db` call
that raises the cancellation did nothing, and one already under way runs to
its end. `Database`'s docstring states it.

A test's own host needs no change: `take_cancellation` is optional.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
…n compensates

Against a real server, on both stores: the handler sees one cancellation,
its transaction committed and made its send, and the compensation is stored
exactly once, made again where the cancellation reached it. A cancelled call
that had landed would show as a second row.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
…on lands

Until now the driver was only tested against a real server, where the
moment a cancel signal arrives cannot be chosen. `sessions_wire` plays
Restate's side of the service protocol in process: it answers state reads
from a dict, stores the writes, acks what asks for an ack, and answers
calls. It can send the cancel signal while a commit waits for its ack, with
nothing waiting, or as an entry an earlier attempt left.

`sessions_driver` runs on both stores:
- a commit under way when the signal arrives lands, and says so;
- the cancellation then fails the await after it, once;
- an attempt that follows a cancelled one fails its first call;
- a carried cancellation fails the next call of the database before it
  writes or sends anything, and a read that asks Restate nothing sees it;
- a schema the handler ensures takes it as its statements do;
- it fails a call's result, not the invocation id made first, and a send
  does not swallow it.

The two commit-under-way cases run on a key an earlier invocation wrote, and
assert the attempt made one commit: on a key never written the first commit
is the schema's, made as the session opens, and the signal would land there.

Each was seen to fail without what it tests: twelve cases with the carry
off, and the two commit-under-way cases on the Python store when a
cancellation fails the store's waits.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

I reviewed the three commits stacked on explicit-recovery-vm: the cancellation carry in the driver, where ctx.db delivers it, and the new wire harness and its tests.

  • One carry, in AttemptDriver: _cancelled() now fails only the pending futures that are not in _left_to_settle. When it fails none of them, it sets _carried_cancellation, which is consumed exactly once. Either future() takes it, failing a call or attach result, or take_carried_cancellation() takes it for the database. Invocation ids and the store's reads and acks are never failed and never take the carry.
  • Store waits left to settle: VmHost.get, state_keys and sync now go through store_future. A commit under way on the Python (journal-codec) store therefore can no longer report as cancelled after its writes are out. On the VM store, the waits happen in Rust and never entered _futures.
  • Attempt.asked(): this delivers the carry before the call issues anything. It runs for AttemptDatabase.query_columns (views included, through _ReaderOf), transaction() (before the session opens) and handler-ensured schemas. It does not run for _start, _flush, _save or statements inside a transaction. This matches the Database contract the PR documents.
  • Tests: sessions_wire.py plays Restate's side of the protocol in process. sessions_driver.py runs 9 cases on both stores. One new server test covers a handler that cancels itself in its transaction and then compensates.

I checked these locally after a maturin develop rebuild. tests/sessions_driver.py passes all 18 cases. The fast sessions*/explicit* suites pass 78 tests. The timing-based cases have plenty of margin: the first commit takes about 10 ms, against the 150 ms cancel timer.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Test Results

  8 files  ±0    8 suites  ±0   3m 48s ⏱️ -54s
 71 tests ±0   71 ✅ ±0  0 💤 ±0  0 ❌ ±0 
322 runs  ±0  322 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit df889a9. ± Comparison against base commit 377d634.

♻️ This comment has been updated with latest results.

The extension built against a branch of shared-core until the VM was
released (restatedev/sdk-shared-core#118). 8.7.0 has it, and is what
@restatedev/sessions builds against: the two SDKs now run the same VM.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
… come

Two places where the driver did not do what @restatedev/sessions does.

A cancellation cancels the calls Restate has not answered, waiting for the
invocation id of one that has none yet. It skipped those: a call counted as
done once its result was settled, and the cancellation itself had just
settled it, with its failure. So a call made shortly before the
cancellation was reported as cancelled and its callee ran on. A call is now
over when Restate answers it.

And once the attempt is over, a cancellation still carried made a call of
the database raise it, a terminal 409, where the handler's awaits of Restate
raise the attempt's end. Whatever cleans up after a shutdown took an attempt
that will be retried for an invocation that is over.

The docstrings say what a cancellation does rather than that it fails one
await: every await pending fails, or else the next one, and a call made
while it is carried is made all the same.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
@igalshilman igalshilman changed the title Sessions: a cancellation reaches the handler once, and a cancelled ctx.db call did nothing Sessions: attempts on the explicit recovery VM, with one cancellation rule Nov 17, 2026
@igalshilman
igalshilman changed the base branch from explicit-recovery-vm to main November 17, 2026 06:46

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. One test gap is noted below. It is mergeable as-is.

Reviewed changes

I reviewed the two commits added since 0ad587d: the move to the shared-core 8.7.0 release, and the driver fix that cancels a call whose invocation id arrives after the cancel signal.

  • Moved to shared-core 8.7.0: The [patch.crates-io] git override is gone. The lock's checksum matches the published crate. Compared with the old git revision, SQL calls now go through on_runtime/off_host_thread. Python polls a SqlTask on the asyncio thread, outside any tokio task, so those calls still run inline as before. The extension builds, and 78 fast sessions*/explicit* tests pass.
  • Made a call count as over once Restate answers it: track() records each call by its result handle. _tick sets answered when that result's notification arrives. _cancelled and cancel_calls now skip calls that are answered rather than done(). Before, a call whose result the cancellation had just failed was skipped, so its callee ran on.
  • Stopped take_carried_cancellation once the attempt has ended: A ctx.db call now fails with the attempt's end, as an await of Restate does. The carry is left for the next attempt.
  • Updated the docstrings to state the cancellation contract: The Database and explicit_cancellation docstrings now describe it as it is: every pending await fails, or else the next one does. A call made while the cancellation is carried is still issued. The docstring example also checks error.message.

ℹ️ No test covers either fix in 56e6cf5

With all of 56e6cf5's _driver.py changes reverted, the 78 fast tests still pass. sessions_wire.RestateSide always sends a call's invocation id right away, so the "cancel before the id" case never comes up. I checked the fix with an ad-hoc test that sends the cancel signal first and the id 50 ms later. Against this head, the attempt sends the cancel command (0x0410) once the id arrives. With the old call.done() check, it sends none and the callee runs on. The PR body mentions a hardening commit is coming; this case could go into it.

Technical details
# Regression tests for 56e6cf5

## Affected sites
- python/restate/sessions/_driver.py:458,464 — `cancel_calls` skips `answered` (was `done()`); no test fails if reverted
- python/restate/sessions/_driver.py:414 — `take_carried_cancellation` returns False once `ended`; no test covers it

## Required outcome
- A driver case (both stores) where the cancel signal arrives before a call's invocation id, asserting that a cancel-invocation command (`0x0410`) is sent after the id arrives, and that the handler sees `TerminalError("cancelled", 409)`.
- Optionally, a case where a cancellation is carried when the attempt ends, asserting that a `ctx.db` call does not raise the 409.

## Suggested approach
Subclass `RestateSide` and override `_answer`. For `TYPE.call` to key `k`: bump `_index`, `_push(CANCEL_SIGNAL)`, then `loop.call_later(0.05, self._push, frame(TYPE.call_invocation_id_completion, uint(1, message.int_of(10)) + bytes_(16, "inv_late")))`. The handler awaits `ctx.generic_call("Callee", "echo", "A", key="k")`. Assert that `TYPE` `0x0410` appears in `side.sent()` after the call. Verified locally: it passes on 56e6cf5 and fails with the old `call.done()` check.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…be wrong

The driver tests passed against several wrong implementations. They now
fail against each of these, tried one at a time:

- a cancellation carried even though it failed an await (delivered twice);
- the outbox emptied after a commit taking the carry, which reports a
  committed transaction as cancelled;
- a carry set for a handler with explicit cancellation;
- a cancellation failing the store's reads, its key listing or its acks;
- a call left running because its invocation id came after the
  cancellation;
- a carry delivered after the attempt ended.

Three things in the harness made that possible. It sent the whole state
with the start, so the store never asked Restate for anything: it sends
only the database's head, as an invocation does. It gave the ack of a
commit twenty milliseconds after the cancel signal, which on a stalled
machine reads both at once and proves nothing: it now holds what follows
the signal until the attempt has acted on it, with no clock involved. And
it always gave a call its invocation id at once.

New cases: a call and a commit made at once, a view, the key's other
database, a transaction whose cancelled body never ran. No test waits on
wall-clock time for what it asserts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. This commit closes the test gap noted on 56e6cf5.

Reviewed changes

I reviewed ac2fd9f, the only commit since 56e6cf5. It changes tests only: tests/sessions_driver.py and tests/sessions_wire.py.

  • Removed the timing dependence from the cancellation tests. The Gate fixture and RestateSide._push_alone/handed_off() now hold back whatever follows a cancel signal until the attempt has acted on it. They count event-loop turns, not seconds. The Cancelling timer and the asyncio.sleep calls are gone.
  • Made the harness preload only the head of the database. start now sends HEAD_KEY alone, with partial_state set. The rest of the state is read on the wire, so the store's reads can be cancelled while they are in flight. I checked field 5 (partial_state) and the move of the key listing to field 17 (StateKeys) against shared-core 8.7.0's protocol.proto.
  • Added driver cases for each of the PR's cancellation claims. They cover:
    • a cancellation is not delivered twice;
    • emptying the outbox after a commit does not take the carry;
    • handlers with explicit cancellation;
    • a cancellation landing on a read of the state, or on a listing of its keys;
    • a cancellation before and after a call's invocation id;
    • no carry delivered once the attempt has ended;
    • gather order, views, and another database of the key.

I rebuilt the extension with maturin develop and ran the tests:

  • tests/sessions_driver.py: all 42 cases passed in each of 8 runs in a row.
  • Fast sessions*/explicit* suites: 102 passed.
  • ruff: clean.

I also tried three wrong implementations of _driver.py, and a test fails on each one:

  • done() instead of answered in the cancel paths;
  • no ended check in take_carried_cancellation;
  • _carried_cancellation always set.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…nnotated or not

An attempt told Restate that it makes progress only while an annotation was
open, and only the OpenAI agents open one. Everything else that works
outside Restate for longer than the inactivity timeout (a minute, by
default) was shut down and restarted, over and over: a model request of
Pydantic AI, ADK or LangGraph, a tool, a summarizer, any handler of a
session object. Each attempt made the request again, and none got past it.

On main a turn is kept by its working run, which this branch removed with
the selector. An agent whose model takes three seconds, with an inactivity
timeout of one, answers there after one request; here it asked its model
twenty times in twenty seconds and never answered
(`tests/agents_long_work_server.py`).

The attempt now heartbeats for as long as its handler runs, when either
holds:

- nothing of Restate's is outstanding for the handler, so it is working on
  its own. This is the rule of @restatedev/sessions. The reads and acks of
  the database's store do not count: the database waits for those.
- an annotation is open, as before: it says the handler works beside a call
  it waits for.

With an answer of Restate's outstanding and no annotation it stays silent,
so that Restate shuts the attempt down and a wait of days holds no process.

The test that un-annotated work is shut down and restarted still holds when
the heartbeat is slower than the inactivity timeout, and is named for that;
beside it, the same work now finishes in its first attempt.

The agents' docs said a turn works on after Restate closed its input, and
advised raising the inactivity timeout for long steps. Neither is so here.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. One informational note on the new heartbeat rule is below.

Reviewed changes

I reviewed 14a0604, the only commit since ac2fd9f. It makes an attempt heartbeat while its handler works on its own, whether or not an annotation is open.

  • Started the heartbeat for every attempt: run() now starts _heartbeat next to the handler, and _end still cancels it. trace_opened/trace_closed now only keep the count.
  • Skipped a beat only while Restate's answer is outstanding and no annotation is open: _awaits_restate() looks for any pending handle in _futures that is not a store wait. Those are call results, call and send invocation ids, and attaches. The new _store_waits keeps the database's reads and acks out of that count.
  • Updated the docs to match: docs/agents/turns.md and the chat_turn docstring now say that a turn working on its own keeps its attempt past the inactivity timeout, and that a turn waiting on Restate does not.
  • Added tests for both sides of the rule: There are three new driver cases: work with nothing outstanding, work beside a call with no annotation, and work beside a call with one. sessions_harness has a 0.3 s heartbeat twin of the old shutdown test, which is renamed. agents_long_work_server.py asks a 3 s model once, with a 1 s inactivity timeout.

I checked this locally after a maturin develop --release rebuild:

  • tests/sessions_driver.py: all 48 cases pass.
  • ruff: clean.
  • Two wrong versions of the rule each fail 2 tests: never heartbeating, and heartbeating only while an annotation is open (the rule before this commit).

I also checked every agent ctx.call site: _turn.py, _object.py, _tools.py and the LangGraph nodes and graph. Each one awaits its call inline, so no agent leaves a result pending while its model runs.

ℹ️ A dropped call result keeps the attempt silent until Restate answers it

generic_call registers the result future in _futures when the call is made, not when it is awaited. A call that the handler never awaits, or whose await it cancels (for example with asyncio.wait_for after a timeout), therefore counts as outstanding until the callee answers. Any local work after it then runs without heartbeats. If that work outlasts the inactivity timeout, the attempt is shut down and restarted, which is the failure this commit fixes. This is consistent with the PR's stated rule and with the existing gap where _cancelled treats a dropped result as awaited. Nothing in the agents hits it today. It is mergeable as-is, but it is worth a line in the HEARTBEAT_INTERVAL docstring, or a shared decision with sdk-sessions on what counts as awaited.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

When Restate shut an attempt down in the middle of a model request, the
request ran on to its end in the dead attempt, beside the one the next
attempt made, and was paid for twice. The docs say a dead attempt stops, a
model's request included.

The request runs in a task of `Stop.guard`, which waits for it or for the
stop, whichever comes first. The end of an attempt cancels the handler's
task, and that cancellation landed on the guard's wait: the guard let go of
its watcher and left the request running, with nobody awaiting it. The
guard now cancels its work when it is itself cancelled. Nested guards pass
it on, down to the request.

With a model that outlasts the inactivity timeout twice and then answers:
three requests started and all three completed before; now two are
cancelled and one completes. On main the guard is the same, but nothing
shuts a turn down there, so it does not show.

Also a sentence in the heartbeat's docstring: an answer of Restate's counts
as outstanding from the call to its answer, awaited or not.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The dropped-result gap from the last review is now documented.

Reviewed changes

I reviewed 1cd3919, the only commit since 14a0604.

  • Cancelled a guard's work when the guard itself is cancelled. Stop.guard now cancels work when the task awaiting it is cancelled, and attaches _retrieve so that asyncio logs no unretrieved exception. Before this commit, the end of an attempt cancelled the handler's task but left the model request running. chat_turn's finally also cancels the watchers, so on_attempt_end never stopped the request either. answer_once is the only caller of guard in the SDK, and a framework's own guard runs inside it, so the cancellation reaches both.
  • Documented when a call's answer counts as outstanding. The _driver.py module docstring now says that work after a call whose result is left aside gets no heartbeat until Restate answers that call. This covers the informational note from the last review.
  • Added tests/agents_dead_attempt_server.py. Its server test runs two slow model requests against a 1 s inactivity timeout and checks that each dead attempt's request is cancelled and that exactly one request completes. Its in-process test checks guard directly.

I checked this locally:

  • The in-process test runs under -m "not server". It fails when the new except BaseException branch is removed.
  • The 297 fast agents_* tests pass, and ruff is clean.
  • I did not run the server test, because the runner has no Docker.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…estatedev/sessions tests them

The two things a session object has to hold, driven on the wire on both
stores, from the tests of the same name in @restatedev/sessions.

A saga (`sessions_saga.py`). A handler has no journal to replay, so the
saga keeps its progress in `ctx.db` and resumes from what is stored:
cancelled while a step's call is pending, cancelled between two steps, a
crash in the middle of the compensation that the next attempt finishes
with the same idempotency keys, a step that fails terminally, and a
shutdown while a step's call is outstanding, after which the next attempt
calls again with the same key and undoes nothing. One case writes down a
limit: a call made while a cancellation is carried is still made, and its
callee is not cancelled; the saga undoes that step too.

A transaction under a cancellation (`sessions_transaction_cancel.py`):
it either committed, with each of its sends made once, or did nothing, and
it says which. Two sweeps deliver the cancel signal before each thing the
attempt asks Restate for, and read the rows back through a second attempt.
In the second sweep the transaction is not the handler's first call, so
the signal is also carried into it: it is then refused, with no row and no
send.

Where Python differs from @restatedev/sessions: at a shutdown the
handler's task is cancelled, so its await raises asyncio's cancellation
where TypeScript's rejects with `AttemptShutDown`. Neither is a terminal
error, which is what a saga needs; two tests say so.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
… waits for a call

Expected failures, until shared-core has the fix.

Restate shuts down an attempt that waits for a call, and starts the next
when the call answers. On the explicit recovery VM the suspension names
the call alone, so cancelling the invocation meanwhile does nothing until
the call answers, and nothing at all if it never does. On main the
invocation ends at once.

For a session object: a cancelled invocation stays suspended, with or
without explicit cancellation. For an agent: `stop` does not end a turn
that waits for an approval once its attempt was shut down.

shared-core's branch `claude/cancel-wakes-a-shut-down-attempt` names the
cancellation in the suspension unless the invocation was cancelled
already. Built against it, these four pass, with the rest of the suite. A
third test checks what kept the cancellation out of the suspension: that a
cancelled invocation which goes on waiting is not resumed over and over.

The tests are strict expected failures: with the release that has the fix
they pass, which fails them, and the marker comes off.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The two new commits change tests only.

Reviewed changes

I reviewed 2af0fd5 and df889a9, the two commits since 1cd3919. They touch only tests/, and no production code changes.

  • Added tests/sessions_saga.py. It runs a saga that keeps its progress in ctx.db, on both stores. The cases cover a cancellation at each step, a crash in the middle of compensating, a step that fails terminally, and a shutdown during a step. test_d pins the documented gap: a cancellation carried to a call still issues the call and sends no cancel command for it. The shutdown cases assert that the handler sees asyncio.CancelledError, which is where Python differs from AttemptShutDown in @restatedev/sessions.
  • Added tests/sessions_transaction_cancel.py. Two sweeps send the cancel signal before each thing the attempt asks Restate for. Each run checks that the transaction either committed, with its row stored and its send made once, or raised the cancellation with no row and no send. It also checks that exactly one cancellation reaches the handler. Further cases cover a body that awaits a call, a carried cancellation that refuses a transaction before it begins, a shutdown while a commit's ack is outstanding, and a body that raises.
  • Extended sessions_wire.RestateSide. The new knobs are cancel_before_answer, withhold_answer, hold, fail, id_per_call and cancel_on_commit=n. It now records asks, calls, sends and cancels, and there are new outcome() and commits() helpers.
  • Added four server tests as strict expected failures. Three are in sessions_suspended_server.py and one is in agents_suspended_stop_server.py. They cover a cancellation that does not reach an attempt shut down while it waits for a call. Each carries xfail(strict=True), so it will start failing, and its marker must come off, once a shared-core release fixes the gap.

I checked this locally after a maturin develop --release rebuild:

  • The 38 new fast cases pass in about 1 s.
  • The fast sessions*/explicit* suites pass 146 tests, and ruff is clean.
  • I tried two wrong implementations, and the new files fail on each: store_future not leaving the store's waits to settle, and Attempt.asked doing nothing.
  • I did not run the server tests, because the runner has no Docker.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@igalshilman
igalshilman merged commit c16e4ed into main Nov 17, 2026
12 checks passed
@igalshilman
igalshilman deleted the claude/sessions-cancellation-once branch November 17, 2026 08:19
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 17, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants