Repository navigation
feat(extras): suspend a session MicroVM after a job - #845
Draft
ParidelPooya wants to merge 11 commits into
Draft
ParidelPooya wants to merge 11 commits into
ParidelPooya wants to merge 11 commits into
Conversation
A SuspendMicrovm retry can run after the MicroVM resumed and freeze the next job. The default client now makes one attempt, and the before-send handler starts no attempt after a resume hook.
A resend of a known job got 503 while the worker refused jobs. The durable function then resumed the MicroVM only to get the duplicate answer. The worker now answers a known callback ID with 202 first.
The 503 note for a resent job said that nothing would suspend the MicroVM again. The idle time restarts after the refusal, so the worker does suspend it again. The real cost is a resume of the MicroVM for nothing. A resume hook can stop the first SuspendMicrovm attempt. Nothing was sent then. The classification docs now say that an earlier attempt, if there was one, may have suspended the MicroVM. The README now states the cost of one attempt. A first-attempt 5xx or dropped connection counts as unknown, and the worker refuses jobs for up to 30 seconds on a MicroVM that did not suspend. It also states why a resent job whose callback is gone gets 202 after the terminate hook.
The JavaScript worker logs an info line when a resume hook arrives during its SuspendMicrovm call. This worker logged nothing until the unknown warning at the end of the call. It now logs the same line, with the same text and the call's microvmId. The resume hook decides under the lock and logs after it. Only the first resume hook during a call logs the line. A resume hook after the call returned logs nothing, because no attempt is left to stop. New listener tests cover a resume hook before the first attempt, a suspend hook after that resume and before the call returns, a second resume hook during the call, and a resume hook after the call returned. The README documents the line. It also states that the JavaScript worker cancels the attempt in flight, while this worker can only stop the next attempt.
…spend-retry-and-duplicates
A job request with "suspendAfter": true asks the worker to suspend its MicroVM when the job ends. The worker answers such a request with the header X-Microvm-Suspend-After: accepted. It suspends only the MicroVM that the run hook named, once no other work runs. The call uses the same path as the suspend when idle.
A session job with suspend_after=True asks the worker to suspend the MicroVM after the job. The next job of the session checks the state and resumes the MicroVM. A job whose delivery gets a 502 with MICROVM_CONNECT_FAILED, or no answer, also checks the state. So a replay that lost the session's flag still resumes the MicroVM.
…after # Conflicts: # packages/aws-durable-execution-sdk-python-extras/tests/microvm/worker_composed_test.py # packages/aws-durable-execution-sdk-python-microvm-worker/README.md # packages/aws-durable-execution-sdk-python-microvm-worker/src/aws_durable_execution_sdk_python_microvm_worker/worker.py # packages/aws-durable-execution-sdk-python-microvm-worker/tests/lifecycle_hooks_test.py
A suspend refusal means that the MicroVM is suspending already. So a suspend request that meets one is done. The suspend hook now clears a pending request, and a suspendAfter job that ends during a refusal sets none. Before, the request fired after the next job, even when that job did not ask for it. Tests cover a job that ends between the suspend and resume hooks, and a request that waits when the suspend hook arrives. The terminate test now asserts that the request is cleared.
A replay of a completed job calls vm.invoke again, so it sets the suspend_after record again. Only a completed map item, parallel branch, or child context skips its function. The suspend_after job itself checks the state, because the record is set before the job starts. A worker that takes more than 10 seconds to start listening gives the first delivery no answer. The operation then makes one GetMicrovm call, so a session that never suspends can make one extra call. The warning about a worker that ignores the field is logged once per session in each execution environment. The README snippet sets heartbeat_timeout on both jobs, as the example does.
The map test with one suspend_after item relies on items that run at once. MapConfig(max_concurrency=3) states that, instead of relying on the default.
| simulated.record(status) | ||
| self.send_response(status) | ||
| for name, value in headers.items(): | ||
| self.send_header(name, value) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds
suspend_aftertovm.invoke. A session job withsuspend_after=Trueasks the worker to suspend its MicroVM when the job ends. The next job of the session resumes the MicroVM. So a session can stop paying for a running MicroVM during a long wait, without auto-suspend.Why
The JavaScript SDK adds the same option in aws/aws-durable-execution-sdk-js#1105.
Changes
Worker (
aws-durable-execution-sdk-python-microvm-worker):"suspendAfter": true. The worker accepts only JSONtrue. The job document stays version 1, so older workers ignore the field.X-Microvm-Suspend-After: accepted. A second delivery gets the header when the first request carried the field.suspend_afterjob ends, after its report, the worker records a request. The request fires when no job, report, run hook, or suspend call runs, and the worker is not terminating, closed, or refusing jobs.resumehook, the outcome classification, and the 30-second grace.terminatehook andclose()clear it. Aresumehook never sets it.suspending the MicroVM after a jobandcannot suspend the MicroVM after a job: the run hook named no MicroVM. The failed-suspend warning addstrigger(idleorjob) as its last field._locked(), and the actions run after the block.Extras (
aws-durable-execution-sdk-python-extras):MicrovmInvokeOptions.suspend_after: bool = False.microvm()rejects it withValidationError, and a value that is not aboolis rejected, both before any durable operation.mapitem,parallelbranch, or child context replays. So every session job also has a fallback. A delivery that gets HTTP 502 withx-aws-proxy-error: MICROVM_CONNECT_FAILED, or no answer, makes one state check, resumes a suspending or suspended MicroVM, and sends the job again. It runs at most one check per 2 seconds.GetMicrovmin the fallback logs one warning per job, and the delivery keeps retrying. A deniedResumeMicrovmwhile the MicroVM is suspended fails the job at once, and the message nameslambda:ResumeMicrovm.mapitems, and on the last job.Examples:
microvm_session_suspend_after.pyand cloud testtest_microvm_session_suspend_after.py. The session has no auto-suspend. The first job writes a file withsuspend_after=True. A callback gate holds the second job until the test sees the MicroVM SUSPENDED. The second job resumes the MicroVM and reads the file.Testing
suspend_afterjob and then a job that resumes the MicroVM; amapwith onesuspend_afteritem; a replay that lost the flag and recovers through the fallback; an old worker without the acknowledgement.hatch run dev-microvm-worker:cov: 519 tests pass,worker.pyat 100%.hatch run dev-extras:cov: 450 tests pass, 100%. Both typechecks,hatch fmt --check,dev-examples:test, andlintcommit.pypass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.