Repository navigation
Sessions: attempts on the explicit recovery VM, with one cancellation rule - #316
Conversation
A session object's attempts no longer run on the SDK's journaled context with a selector patched in to fake one await point over do_progress: they run on shared-core's ExplicitRecoveryVM (restatedev/sdk-shared-core#118), which never replays and has no await point at all. - src/explicit_recovery.rs: PyExplicitRecoveryVM. Commands return handles, results come back as notifications; those of the database's own handles go straight to the Rust store. - sessions/_driver.py: one attempt. One input pump, the only reader of the body; a tick after every call into the VM writes the output, settles the futures, and notices the end of the attempt (a shutdown when Restate closes the input, a failure, or the end). Cancellation: the VM's is_cancelled(), checked at every tick. An open annotation sends heartbeats, so a long model request is not shut down for inactivity. - The store over the VM syncs a commit with one acked write of the database's head (KvStore::put_and_sync), instead of an empty run. - Handler.attempt_runner: the server hands a session object's request to it. - The selector and its working run (#312) are gone: nothing suspends on the cancel signal. - shared-core is patched to the PR's branch until it is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x.db call did nothing On the explicit recovery VM a cancellation failed only the awaits pending in the driver when it came. One that arrived with nothing pending was dropped: a handler working outside Restate, or one whose database call was under way, never saw it, and neither did the attempt that follows a cancelled one. With a journal codec, where the database's store goes through Python, it failed the store's reads and the ack of a commit instead, so a commit could land and report as cancelled. The driver now carries a cancellation that failed nothing the handler awaits to its next await, once: a result of Restate's, or a call of `ctx.db`. An invocation id is never the one it is carried to, as it is not what a cancellation fails. `ctx.db` takes the carry where the handler calls it (`Attempt.asked`): its statements, its `transaction()`, and a schema it ensures, before the call issues anything. What the session does for itself does not pass through there: the open, the object's schema, the sends a crashed attempt left, emptying the outbox after a commit, the save at the end. And the store's reads and acks are left to settle on both stores. So the rule is one rule, the same as @restatedev/sessions: a `ctx.db` call that raises the cancellation did nothing, and one already under way runs to its end. `Database`'s docstring states it. A test's own host needs no change: `take_cancellation` is optional. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
…n compensates Against a real server, on both stores: the handler sees one cancellation, its transaction committed and made its send, and the compensation is stored exactly once, made again where the cancellation reached it. A cancelled call that had landed would show as a second row. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
…on lands Until now the driver was only tested against a real server, where the moment a cancel signal arrives cannot be chosen. `sessions_wire` plays Restate's side of the service protocol in process: it answers state reads from a dict, stores the writes, acks what asks for an ack, and answers calls. It can send the cancel signal while a commit waits for its ack, with nothing waiting, or as an entry an earlier attempt left. `sessions_driver` runs on both stores: - a commit under way when the signal arrives lands, and says so; - the cancellation then fails the await after it, once; - an attempt that follows a cancelled one fails its first call; - a carried cancellation fails the next call of the database before it writes or sends anything, and a read that asks Restate nothing sees it; - a schema the handler ensures takes it as its statements do; - it fails a call's result, not the invocation id made first, and a send does not swallow it. The two commit-under-way cases run on a key an earlier invocation wrote, and assert the attempt made one commit: on a key never written the first commit is the schema's, made as the session opens, and the signal would land there. Each was seen to fail without what it tests: twelve cases with the carry off, and the two commit-under-way cases on the Python store when a cancellation fails the store's waits. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
I reviewed the three commits stacked on explicit-recovery-vm: the cancellation carry in the driver, where ctx.db delivers it, and the new wire harness and its tests.
- One carry, in
AttemptDriver:_cancelled()now fails only the pending futures that are not in_left_to_settle. When it fails none of them, it sets_carried_cancellation, which is consumed exactly once. Eitherfuture()takes it, failing a call or attach result, ortake_carried_cancellation()takes it for the database. Invocation ids and the store's reads and acks are never failed and never take the carry. - Store waits left to settle:
VmHost.get,state_keysandsyncnow go throughstore_future. A commit under way on the Python (journal-codec) store therefore can no longer report as cancelled after its writes are out. On the VM store, the waits happen in Rust and never entered_futures. Attempt.asked(): this delivers the carry before the call issues anything. It runs forAttemptDatabase.query_columns(views included, through_ReaderOf),transaction()(before the session opens) and handler-ensured schemas. It does not run for_start,_flush,_saveor statements inside a transaction. This matches theDatabasecontract the PR documents.- Tests:
sessions_wire.pyplays Restate's side of the protocol in process.sessions_driver.pyruns 9 cases on both stores. One new server test covers a handler that cancels itself in its transaction and then compensates.
I checked these locally after a maturin develop rebuild. tests/sessions_driver.py passes all 18 cases. The fast sessions*/explicit* suites pass 78 tests. The timing-based cases have plenty of margin: the first commit takes about 10 ms, against the 150 ms cancel timer.
claude-opus-5-5 | 𝕏
The extension built against a branch of shared-core until the VM was released (restatedev/sdk-shared-core#118). 8.7.0 has it, and is what @restatedev/sessions builds against: the two SDKs now run the same VM. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
… come Two places where the driver did not do what @restatedev/sessions does. A cancellation cancels the calls Restate has not answered, waiting for the invocation id of one that has none yet. It skipped those: a call counted as done once its result was settled, and the cancellation itself had just settled it, with its failure. So a call made shortly before the cancellation was reported as cancelled and its callee ran on. A call is now over when Restate answers it. And once the attempt is over, a cancellation still carried made a call of the database raise it, a terminal 409, where the handler's awaits of Restate raise the attempt's end. Whatever cleans up after a shutdown took an attempt that will be retried for an invocation that is over. The docstrings say what a cancellation does rather than that it fails one await: every await pending fails, or else the next one, and a call made while it is carried is made all the same. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
There was a problem hiding this comment.
✅ No new issues found. One test gap is noted below. It is mergeable as-is.
Reviewed changes
I reviewed the two commits added since 0ad587d: the move to the shared-core 8.7.0 release, and the driver fix that cancels a call whose invocation id arrives after the cancel signal.
- Moved to shared-core
8.7.0: The[patch.crates-io]git override is gone. The lock's checksum matches the published crate. Compared with the old git revision, SQL calls now go throughon_runtime/off_host_thread. Python polls aSqlTaskon the asyncio thread, outside any tokio task, so those calls still run inline as before. The extension builds, and 78 fastsessions*/explicit*tests pass. - Made a call count as over once Restate answers it:
track()records each call by its result handle._ticksetsansweredwhen that result's notification arrives._cancelledandcancel_callsnow skip calls that areansweredrather thandone(). Before, a call whose result the cancellation had just failed was skipped, so its callee ran on. - Stopped
take_carried_cancellationonce the attempt has ended: Actx.dbcall now fails with the attempt's end, as an await of Restate does. The carry is left for the next attempt. - Updated the docstrings to state the cancellation contract: The
Databaseandexplicit_cancellationdocstrings now describe it as it is: every pending await fails, or else the next one does. A call made while the cancellation is carried is still issued. The docstring example also checkserror.message.
ℹ️ No test covers either fix in 56e6cf5
With all of 56e6cf5's _driver.py changes reverted, the 78 fast tests still pass. sessions_wire.RestateSide always sends a call's invocation id right away, so the "cancel before the id" case never comes up. I checked the fix with an ad-hoc test that sends the cancel signal first and the id 50 ms later. Against this head, the attempt sends the cancel command (0x0410) once the id arrives. With the old call.done() check, it sends none and the callee runs on. The PR body mentions a hardening commit is coming; this case could go into it.
Technical details
# Regression tests for 56e6cf5
## Affected sites
- python/restate/sessions/_driver.py:458,464 — `cancel_calls` skips `answered` (was `done()`); no test fails if reverted
- python/restate/sessions/_driver.py:414 — `take_carried_cancellation` returns False once `ended`; no test covers it
## Required outcome
- A driver case (both stores) where the cancel signal arrives before a call's invocation id, asserting that a cancel-invocation command (`0x0410`) is sent after the id arrives, and that the handler sees `TerminalError("cancelled", 409)`.
- Optionally, a case where a cancellation is carried when the attempt ends, asserting that a `ctx.db` call does not raise the 409.
## Suggested approach
Subclass `RestateSide` and override `_answer`. For `TYPE.call` to key `k`: bump `_index`, `_push(CANCEL_SIGNAL)`, then `loop.call_later(0.05, self._push, frame(TYPE.call_invocation_id_completion, uint(1, message.int_of(10)) + bytes_(16, "inv_late")))`. The handler awaits `ctx.generic_call("Callee", "echo", "A", key="k")`. Assert that `TYPE` `0x0410` appears in `side.sent()` after the call. Verified locally: it passes on 56e6cf5 and fails with the old `call.done()` check.claude-opus-5-5 | 𝕏
…be wrong The driver tests passed against several wrong implementations. They now fail against each of these, tried one at a time: - a cancellation carried even though it failed an await (delivered twice); - the outbox emptied after a commit taking the carry, which reports a committed transaction as cancelled; - a carry set for a handler with explicit cancellation; - a cancellation failing the store's reads, its key listing or its acks; - a call left running because its invocation id came after the cancellation; - a carry delivered after the attempt ended. Three things in the harness made that possible. It sent the whole state with the start, so the store never asked Restate for anything: it sends only the database's head, as an invocation does. It gave the ack of a commit twenty milliseconds after the cancel signal, which on a stalled machine reads both at once and proves nothing: it now holds what follows the signal until the attempt has acted on it, with no clock involved. And it always gave a call its invocation id at once. New cases: a call and a commit made at once, a view, the key's other database, a transaction whose cancelled body never ran. No test waits on wall-clock time for what it asserts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
There was a problem hiding this comment.
✅ No new issues found. This commit closes the test gap noted on
56e6cf5.
Reviewed changes
I reviewed ac2fd9f, the only commit since 56e6cf5. It changes tests only: tests/sessions_driver.py and tests/sessions_wire.py.
- Removed the timing dependence from the cancellation tests. The
Gatefixture andRestateSide._push_alone/handed_off()now hold back whatever follows a cancel signal until the attempt has acted on it. They count event-loop turns, not seconds. TheCancellingtimer and theasyncio.sleepcalls are gone. - Made the harness preload only the head of the database.
startnow sendsHEAD_KEYalone, withpartial_stateset. The rest of the state is read on the wire, so the store's reads can be cancelled while they are in flight. I checked field5(partial_state) and the move of the key listing to field17(StateKeys) against shared-core 8.7.0'sprotocol.proto. - Added driver cases for each of the PR's cancellation claims. They cover:
- a cancellation is not delivered twice;
- emptying the outbox after a commit does not take the carry;
- handlers with explicit cancellation;
- a cancellation landing on a read of the state, or on a listing of its keys;
- a cancellation before and after a call's invocation id;
- no carry delivered once the attempt has ended;
gatherorder, views, and another database of the key.
I rebuilt the extension with maturin develop and ran the tests:
tests/sessions_driver.py: all 42 cases passed in each of 8 runs in a row.- Fast
sessions*/explicit*suites: 102 passed. ruff: clean.
I also tried three wrong implementations of _driver.py, and a test fails on each one:
done()instead ofansweredin the cancel paths;- no
endedcheck intake_carried_cancellation; _carried_cancellationalways set.
claude-opus-5-5 | 𝕏
…nnotated or not An attempt told Restate that it makes progress only while an annotation was open, and only the OpenAI agents open one. Everything else that works outside Restate for longer than the inactivity timeout (a minute, by default) was shut down and restarted, over and over: a model request of Pydantic AI, ADK or LangGraph, a tool, a summarizer, any handler of a session object. Each attempt made the request again, and none got past it. On main a turn is kept by its working run, which this branch removed with the selector. An agent whose model takes three seconds, with an inactivity timeout of one, answers there after one request; here it asked its model twenty times in twenty seconds and never answered (`tests/agents_long_work_server.py`). The attempt now heartbeats for as long as its handler runs, when either holds: - nothing of Restate's is outstanding for the handler, so it is working on its own. This is the rule of @restatedev/sessions. The reads and acks of the database's store do not count: the database waits for those. - an annotation is open, as before: it says the handler works beside a call it waits for. With an answer of Restate's outstanding and no annotation it stays silent, so that Restate shuts the attempt down and a wait of days holds no process. The test that un-annotated work is shut down and restarted still holds when the heartbeat is slower than the inactivity timeout, and is named for that; beside it, the same work now finishes in its first attempt. The agents' docs said a turn works on after Restate closed its input, and advised raising the inactivity timeout for long steps. Neither is so here. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
There was a problem hiding this comment.
✅ No new issues found. One informational note on the new heartbeat rule is below.
Reviewed changes
I reviewed 14a0604, the only commit since ac2fd9f. It makes an attempt heartbeat while its handler works on its own, whether or not an annotation is open.
- Started the heartbeat for every attempt:
run()now starts_heartbeatnext to the handler, and_endstill cancels it.trace_opened/trace_closednow only keep the count. - Skipped a beat only while Restate's answer is outstanding and no annotation is open:
_awaits_restate()looks for any pending handle in_futuresthat is not a store wait. Those are call results, call and send invocation ids, and attaches. The new_store_waitskeeps the database's reads and acks out of that count. - Updated the docs to match:
docs/agents/turns.mdand thechat_turndocstring now say that a turn working on its own keeps its attempt past the inactivity timeout, and that a turn waiting on Restate does not. - Added tests for both sides of the rule: There are three new driver cases: work with nothing outstanding, work beside a call with no annotation, and work beside a call with one.
sessions_harnesshas a 0.3 s heartbeat twin of the old shutdown test, which is renamed.agents_long_work_server.pyasks a 3 s model once, with a 1 s inactivity timeout.
I checked this locally after a maturin develop --release rebuild:
tests/sessions_driver.py: all 48 cases pass.ruff: clean.- Two wrong versions of the rule each fail 2 tests: never heartbeating, and heartbeating only while an annotation is open (the rule before this commit).
I also checked every agent ctx.call site: _turn.py, _object.py, _tools.py and the LangGraph nodes and graph. Each one awaits its call inline, so no agent leaves a result pending while its model runs.
ℹ️ A dropped call result keeps the attempt silent until Restate answers it
generic_call registers the result future in _futures when the call is made, not when it is awaited. A call that the handler never awaits, or whose await it cancels (for example with asyncio.wait_for after a timeout), therefore counts as outstanding until the callee answers. Any local work after it then runs without heartbeats. If that work outlasts the inactivity timeout, the attempt is shut down and restarted, which is the failure this commit fixes. This is consistent with the PR's stated rule and with the existing gap where _cancelled treats a dropped result as awaited. Nothing in the agents hits it today. It is mergeable as-is, but it is worth a line in the HEARTBEAT_INTERVAL docstring, or a shared decision with sdk-sessions on what counts as awaited.
claude-opus-5-5 | 𝕏
When Restate shut an attempt down in the middle of a model request, the request ran on to its end in the dead attempt, beside the one the next attempt made, and was paid for twice. The docs say a dead attempt stops, a model's request included. The request runs in a task of `Stop.guard`, which waits for it or for the stop, whichever comes first. The end of an attempt cancels the handler's task, and that cancellation landed on the guard's wait: the guard let go of its watcher and left the request running, with nobody awaiting it. The guard now cancels its work when it is itself cancelled. Nested guards pass it on, down to the request. With a model that outlasts the inactivity timeout twice and then answers: three requests started and all three completed before; now two are cancelled and one completes. On main the guard is the same, but nothing shuts a turn down there, so it does not show. Also a sentence in the heartbeat's docstring: an answer of Restate's counts as outstanding from the call to its answer, awaited or not. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
There was a problem hiding this comment.
✅ No new issues found. The dropped-result gap from the last review is now documented.
Reviewed changes
I reviewed 1cd3919, the only commit since 14a0604.
- Cancelled a guard's work when the guard itself is cancelled.
Stop.guardnow cancelsworkwhen the task awaiting it is cancelled, and attaches_retrieveso that asyncio logs no unretrieved exception. Before this commit, the end of an attempt cancelled the handler's task but left the model request running.chat_turn'sfinallyalso cancels the watchers, soon_attempt_endnever stopped the request either.answer_onceis the only caller ofguardin the SDK, and a framework's ownguardruns inside it, so the cancellation reaches both. - Documented when a call's answer counts as outstanding. The
_driver.pymodule docstring now says that work after a call whose result is left aside gets no heartbeat until Restate answers that call. This covers the informational note from the last review. - Added
tests/agents_dead_attempt_server.py. Its server test runs two slow model requests against a 1 s inactivity timeout and checks that each dead attempt's request is cancelled and that exactly one request completes. Its in-process test checksguarddirectly.
I checked this locally:
- The in-process test runs under
-m "not server". It fails when the newexcept BaseExceptionbranch is removed. - The 297 fast
agents_*tests pass, andruffis clean. - I did not run the server test, because the runner has no Docker.
claude-opus-5-5 | 𝕏
…estatedev/sessions tests them The two things a session object has to hold, driven on the wire on both stores, from the tests of the same name in @restatedev/sessions. A saga (`sessions_saga.py`). A handler has no journal to replay, so the saga keeps its progress in `ctx.db` and resumes from what is stored: cancelled while a step's call is pending, cancelled between two steps, a crash in the middle of the compensation that the next attempt finishes with the same idempotency keys, a step that fails terminally, and a shutdown while a step's call is outstanding, after which the next attempt calls again with the same key and undoes nothing. One case writes down a limit: a call made while a cancellation is carried is still made, and its callee is not cancelled; the saga undoes that step too. A transaction under a cancellation (`sessions_transaction_cancel.py`): it either committed, with each of its sends made once, or did nothing, and it says which. Two sweeps deliver the cancel signal before each thing the attempt asks Restate for, and read the rows back through a second attempt. In the second sweep the transaction is not the handler's first call, so the signal is also carried into it: it is then refused, with no row and no send. Where Python differs from @restatedev/sessions: at a shutdown the handler's task is cancelled, so its await raises asyncio's cancellation where TypeScript's rejects with `AttemptShutDown`. Neither is a terminal error, which is what a saga needs; two tests say so. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
… waits for a call Expected failures, until shared-core has the fix. Restate shuts down an attempt that waits for a call, and starts the next when the call answers. On the explicit recovery VM the suspension names the call alone, so cancelling the invocation meanwhile does nothing until the call answers, and nothing at all if it never does. On main the invocation ends at once. For a session object: a cancelled invocation stays suspended, with or without explicit cancellation. For an agent: `stop` does not end a turn that waits for an approval once its attempt was shut down. shared-core's branch `claude/cancel-wakes-a-shut-down-attempt` names the cancellation in the suspension unless the invocation was cancelled already. Built against it, these four pass, with the rest of the suite. A third test checks what kept the cancellation out of the suspension: that a cancelled invocation which goes on waiting is not resumed over and over. The tests are strict expected failures: with the release that has the fix they pass, which fails them, and the marker comes off. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV
There was a problem hiding this comment.
✅ No new issues found. The two new commits change tests only.
Reviewed changes
I reviewed 2af0fd5 and df889a9, the two commits since 1cd3919. They touch only tests/, and no production code changes.
- Added
tests/sessions_saga.py. It runs a saga that keeps its progress inctx.db, on both stores. The cases cover a cancellation at each step, a crash in the middle of compensating, a step that fails terminally, and a shutdown during a step.test_dpins the documented gap: a cancellation carried to a call still issues the call and sends no cancel command for it. The shutdown cases assert that the handler seesasyncio.CancelledError, which is where Python differs fromAttemptShutDownin @restatedev/sessions. - Added
tests/sessions_transaction_cancel.py. Two sweeps send the cancel signal before each thing the attempt asks Restate for. Each run checks that the transaction either committed, with its row stored and its send made once, or raised the cancellation with no row and no send. It also checks that exactly one cancellation reaches the handler. Further cases cover a body that awaits a call, a carried cancellation that refuses a transaction before it begins, a shutdown while a commit's ack is outstanding, and a body that raises. - Extended
sessions_wire.RestateSide. The new knobs arecancel_before_answer,withhold_answer,hold,fail,id_per_callandcancel_on_commit=n. It now recordsasks,calls,sendsandcancels, and there are newoutcome()andcommits()helpers. - Added four server tests as strict expected failures. Three are in
sessions_suspended_server.pyand one is inagents_suspended_stop_server.py. They cover a cancellation that does not reach an attempt shut down while it waits for a call. Each carriesxfail(strict=True), so it will start failing, and its marker must come off, once a shared-core release fixes the gap.
I checked this locally after a maturin develop --release rebuild:
- The 38 new fast cases pass in about 1 s.
- The fast
sessions*/explicit*suites pass 146 tests, andruffis clean. - I tried two wrong implementations, and the new files fail on each:
store_futurenot leaving the store's waits to settle, andAttempt.askeddoing nothing. - I did not run the server tests, because the runner has no Docker.
claude-opus-5-5 | 𝕏

Self-contained on
main: this carries the commit of #314 (c5e1f7d, "Sessions: attempts on the explicit recovery VM, without the selector", unchanged and with its author) and ten commits on top of it. Merging this one brings in both; #314 can then be closed. Nothing here needs #314 merged first.Read this first: two regressions from
mainthat come with the VM1. Not fixed here: a cancellation does not reach an invocation that is suspended on a call
Restate shuts down an attempt that waits for a call, and starts the next one when the call answers. On the explicit recovery VM the suspension names the calls alone, so cancelling the invocation meanwhile does nothing until the call answers, and nothing at all if it never does.
Run against a real server, on
mainand here:mainit ends at once; here it stayssuspended, with or without explicit cancellation.stop: onmainthe turn ends; herestopanswers 200 and the turn stays suspended.It is in shared-core 8.7.0 (
TODO the cancellation is ignored for now), so #314 has it too, and so does @restatedev/sessions. The fix is on shared-core's branchclaude/cancel-wakes-a-shut-down-attempt: the suspension names the cancellation, unless the invocation was cancelled already. Built against that branch, the four tests below pass, with the rest of the suite. It needs a shared-core release and a bump here.The tests are in this PR as strict expected failures (
tests/sessions_suspended_server.py,tests/agents_suspended_stop_server.py): they start failing, and lose their marker, with the shared-core that fixes them.2. Fixed here, in
14a0604: an agent's turn longer than the inactivity timeout restarted foreverOn
maina turn's attempt is kept alive by its working run, which #314 removes with the selector. #314 heartbeats only while an annotation is open, and only the OpenAI agents open one. So anything else that works outside Restate for longer than the inactivity timeout (a minute, by default) was shut down and restarted, over and over: a model request of Pydantic AI, ADK or LangGraph, a tool, a summarizer.One test on both (
tests/agents_long_work_server.py: a model that takes 3 s, an inactivity timeout of 1 s): onmainthe model is asked once and the turn answers; on #314 the turn never answers, and the model was asked 20 times in 20 seconds.An attempt now heartbeats for as long as its handler runs, when either holds: nothing of Restate's is outstanding for the handler (the rule of @restatedev/sessions), or an annotation is open (as #314 has it, for work beside a call). With an answer of Restate's outstanding and no annotation it stays silent, so a wait of days holds no process.
This changes a choice #314 made on purpose, so it is its own commit: one revert removes it. #314's test that un-annotated work is shut down and restarted still holds when the heartbeat is slower than the inactivity timeout, and is renamed for that.
The rest, on top of #314
It brings Python's session objects to the cancellation rule that sdk-sessions#39 merged for TypeScript, with that package as the reference.
Shared core 8.7.0
#314 builds against a branch of shared-core, "until shared-core releases it". 8.7.0 has the VM, and is what @restatedev/sessions builds against. The override is gone, and the two SDKs run the same VM.
One cancellation rule
For a handler without
explicit_cancellation, #314 had two problems:ctx.dbcall was under way, never saw it; neither did the attempt that follows a cancelled one. Against a real server, a handler that cancels itself in its transaction and then compensates saw 0 cancellations.The rule now:
ctx.db. Never an invocation id.ctx.dbtakes it where the handler calls it (Attempt.asked): its statements, itstransaction(), and a schema it ensures — before the call issues anything. What the session does for itself does not pass through there: the open, the object's schema, the sends a crashed attempt left, emptying the outbox after a commit, the save at the end.So a
ctx.dbcall that raises the cancellation did nothing — no write, no commit, no sends — and one already under way runs to its end.Database's docstring states the contract; none was written down for session objects before.Three more places that differed from @restatedev/sessions
ctx.dbraise a terminal 409, where the handler's awaits of Restate raise the attempt's end. The same fix is in sdk-sessions#41.Stop.guardlet go of its work when the handler's task was cancelled. With a model that outlasts the inactivity timeout twice and then answers: three requests started and all three completed; now two are cancelled and one completes.Tests
tests/sessions_wire.py(new): Restate's side of the service protocol played in process, ported from the TypeScript harness. The driver was only tested against a real server, where the moment a cancel signal arrives cannot be chosen. It starts an attempt with only the database's head, as an invocation does, so the store's reads reach it; and what follows a cancel signal is held until the attempt has acted on the signal, with no clock involved.tests/sessions_driver.py(new, both stores, 48 cases): where a cancellation lands, and when an attempt heartbeats.tests/sessions_transaction_cancel.py(new): a transaction under a cancellation either committed, with each of its sends made once, or did nothing, and says which. Two sweeps deliver the cancel signal before each thing the attempt asks Restate for, and read the rows back through a second attempt.tests/sessions_saga.py(new): a saga that keeps its progress inctx.db, cancelled at each step, crashed in the middle of its compensation, with a step that fails, and shut down during a long step.cancelledCommit; local work that keeps its attempt; an agent whose model outlasts the inactivity timeout; a dead attempt's model request; and the four expected failures above.The driver tests fail against each of these wrong implementations, tried one at a time: no carry; a carry set although an await was failed; the outbox emptied after a commit taking the carry; a carry set for a handler with explicit cancellation; a cancellation failing the store's reads, key listing or acks; a call left running because its invocation id came after the cancellation; a carry delivered after the attempt ended; no heartbeat without an annotation; a heartbeat while an answer is outstanding.
Run locally: 504 fast tests, ruff, pyright, mypy; and, against a local Restate 1.8.0, the server-marked tests of
sessions_harness,explicit_recovery,explicit_cancellationand the agents' (57 passing, 4 expected failures).Not changed here
AttemptShutDown. As Sessions: attempts on the explicit recovery VM, without the selector #314 has it; neither is a terminal error.startas a send. Left as Sessions: attempts on the explicit recovery VM, without the selector #314 has them.compacthandler can swallow the cancellation: it runs without explicit cancellation and wrapssummarizeinexcept Exception. The same onmain.🤖 Generated with Claude Code
https://claude.ai/code/session_01RePHkSS8KRKmJ93hcd9asV