Skip to content

feat(extras): suspend a session MicroVM after a job - #845

Draft
ParidelPooya wants to merge 11 commits into
mainfrom
feat/microvm-suspend-after
Draft

ParidelPooya wants to merge 11 commits into
mainfrom
feat/microvm-suspend-after

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds suspend_after to vm.invoke. A session job with suspend_after=True asks the worker to suspend its MicroVM when the job ends. The next job of the session resumes the MicroVM. So a session can stop paying for a running MicroVM during a long wait, without auto-suspend.

Why

  1. A session has one way to suspend its MicroVM today: auto-suspend after an idle time.
  2. The developer often knows the exact point where the MicroVM can suspend, such as the job before a long callback or wait.
  3. A suspend from the durable function would need its own durable step, and it could freeze a job that still runs.
  4. So the worker suspends the MicroVM after the job, once no other work runs. The durable function only marks the job.

The JavaScript SDK adds the same option in aws/aws-durable-execution-sdk-js#1105.

Changes

Worker (aws-durable-execution-sdk-python-microvm-worker):

  • The job request can carry "suspendAfter": true. The worker accepts only JSON true. The job document stays version 1, so older workers ignore the field.
  • The worker answers such a request with the header X-Microvm-Suspend-After: accepted. A second delivery gets the header when the first request carried the field.
  • The worker suspends only the MicroVM that the run hook named, never one from a job request.
  • When a suspend_after job ends, after its report, the worker records a request. The request fires when no job, report, run hook, or suspend call runs, and the worker is not terminating, closed, or refusing jobs.
  • The call uses the existing suspend path: the refusal of jobs, one attempt, no attempt after a resume hook, the outcome classification, and the 30-second grace.
  • A request is done when any suspend starts, or when a suspend refusal is in effect. A terminate hook and close() clear it. A resume hook never sets it.
  • Without auto-suspend, a suspend refused for now is not retried.
  • New log lines: suspending the MicroVM after a job and cannot suspend the MicroVM after a job: the run hook named no MicroVM. The failed-suspend warning adds trigger (idle or job) as its last field.
  • Decisions are made under _locked(), and the actions run after the block.

Extras (aws-durable-execution-sdk-python-extras):

  • MicrovmInvokeOptions.suspend_after: bool = False. microvm() rejects it with ValidationError, and a value that is not a bool is rejected, both before any durable operation.
  • The session sets an in-memory flag before the job's child context. From then on, each job checks the state before delivery, and resumes a suspended MicroVM, as in resume mode.
  • The flag is not rebuilt when a completed map item, parallel branch, or child context replays. So every session job also has a fallback. A delivery that gets HTTP 502 with x-aws-proxy-error: MICROVM_CONNECT_FAILED, or no answer, makes one state check, resumes a suspending or suspended MicroVM, and sends the job again. It runs at most one check per 2 seconds.
  • A denied GetMicrovm in the fallback logs one warning per job, and the delivery keeps retrying. A denied ResumeMicrovm while the MicroVM is suspended fails the job at once, and the message names lambda:ResumeMicrovm.
  • A 202 without the acknowledgement header logs one warning per session: the worker is too old, and the MicroVM keeps running.
  • README sections and docstrings are updated. The docs say when to use the option, and when not: on back-to-back jobs, on map items, and on the last job.

Examples:

  • New example microvm_session_suspend_after.py and cloud test test_microvm_session_suspend_after.py. The session has no auto-suspend. The first job writes a file with suspend_after=True. A callback gate holds the second job until the test sees the MicroVM SUSPENDED. The second job resumes the MicroVM and reads the file.

Testing

  • Worker unit tests cover each rule: the target, the fire conditions, the re-checks, the clears, the acknowledgement, the logs, and the interaction with auto-suspend and the lifecycle hooks.
  • Extras unit tests cover the validation, the field, the warning, the flag, the fallback trigger, the rate limit, and each error rule.
  • Composed tests run the session against the real worker: a suspend_after job and then a job that resumes the MicroVM; a map with one suspend_after item; a replay that lost the flag and recovers through the fallback; an old worker without the acknowledgement.
  • hatch run dev-microvm-worker:cov: 519 tests pass, worker.py at 100%. hatch run dev-extras:cov: 450 tests pass, 100%. Both typechecks, hatch fmt --check, dev-examples:test, and lintcommit.py pass.
  • All 4 MicroVM cloud example tests passed against the live service, including the new one. The MicroVM suspended after the first job, and the second job resumed it and read the file.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

A SuspendMicrovm retry can run after the MicroVM resumed and freeze the
next job. The default client now makes one attempt, and the before-send
handler starts no attempt after a resume hook.
A resend of a known job got 503 while the worker refused jobs. The
durable function then resumed the MicroVM only to get the duplicate
answer. The worker now answers a known callback ID with 202 first.
The 503 note for a resent job said that nothing would suspend the
MicroVM again. The idle time restarts after the refusal, so the worker
does suspend it again. The real cost is a resume of the MicroVM for
nothing.

A resume hook can stop the first SuspendMicrovm attempt. Nothing was
sent then. The classification docs now say that an earlier attempt, if
there was one, may have suspended the MicroVM.

The README now states the cost of one attempt. A first-attempt 5xx or
dropped connection counts as unknown, and the worker refuses jobs for up
to 30 seconds on a MicroVM that did not suspend. It also states why a
resent job whose callback is gone gets 202 after the terminate hook.
The JavaScript worker logs an info line when a resume hook arrives
during its SuspendMicrovm call. This worker logged nothing until the
unknown warning at the end of the call. It now logs the same line, with
the same text and the call's microvmId.

The resume hook decides under the lock and logs after it. Only the
first resume hook during a call logs the line. A resume hook after the
call returned logs nothing, because no attempt is left to stop.

New listener tests cover a resume hook before the first attempt, a
suspend hook after that resume and before the call returns, a second
resume hook during the call, and a resume hook after the call returned.

The README documents the line. It also states that the JavaScript worker
cancels the attempt in flight, while this worker can only stop the next
attempt.
A job request with "suspendAfter": true asks the worker to suspend its
MicroVM when the job ends. The worker answers such a request with the
header X-Microvm-Suspend-After: accepted. It suspends only the MicroVM
that the run hook named, once no other work runs. The call uses the
same path as the suspend when idle.
A session job with suspend_after=True asks the worker to suspend the
MicroVM after the job. The next job of the session checks the state
and resumes the MicroVM. A job whose delivery gets a 502 with
MICROVM_CONNECT_FAILED, or no answer, also checks the state. So a
replay that lost the session's flag still resumes the MicroVM.
…after

# Conflicts:
#	packages/aws-durable-execution-sdk-python-extras/tests/microvm/worker_composed_test.py
#	packages/aws-durable-execution-sdk-python-microvm-worker/README.md
#	packages/aws-durable-execution-sdk-python-microvm-worker/src/aws_durable_execution_sdk_python_microvm_worker/worker.py
#	packages/aws-durable-execution-sdk-python-microvm-worker/tests/lifecycle_hooks_test.py
A suspend refusal means that the MicroVM is suspending already. So a
suspend request that meets one is done. The suspend hook now clears a
pending request, and a suspendAfter job that ends during a refusal sets
none. Before, the request fired after the next job, even when that job
did not ask for it. Tests cover a job that ends between the suspend and
resume hooks, and a request that waits when the suspend hook arrives.
The terminate test now asserts that the request is cleared.
A replay of a completed job calls vm.invoke again, so it sets the
suspend_after record again. Only a completed map item, parallel branch,
or child context skips its function. The suspend_after job itself
checks the state, because the record is set before the job starts.

A worker that takes more than 10 seconds to start listening gives the
first delivery no answer. The operation then makes one GetMicrovm call,
so a session that never suspends can make one extra call. The warning
about a worker that ignores the field is logged once per session in
each execution environment. The README snippet sets heartbeat_timeout
on both jobs, as the example does.
The map test with one suspend_after item relies on items that run at
once. MapConfig(max_concurrency=3) states that, instead of relying on
the default.
simulated.record(status)
self.send_response(status)
for name, value in headers.items():
self.send_header(name, value)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants