Repository navigation
fix(microvm-worker): keep a suspend_after request for the run hook - #848
Merged
ParidelPooya merged 6 commits intoOct 27, 2026
Merged
Conversation
HTTP jobs can arrive before the run hook. A suspendAfter job that ended before the run hook found no target, so the worker logged the skip warning and dropped the request. The run hook then set the target, but its re-check found no request. So the MicroVM kept running, although the 202 answer had acknowledged the option. A due request with no target now stays pending until the first accepted run hook. The re-check at the end of that hook fires the request when the hook named a MicroVM. When the hook named none, the worker logs the skip warning and drops the request.
The terminate hook and close() clear a pending suspendAfter request, and a job end after either records none. A suspend refusal clears the request too, and a job end during a refusal records none. So the terminating, closed, and suspending checks in the fire conditions never decide a reachable case. This matches the JavaScript worker. The fire conditions now check only for running work. The code comment and the README state one rule: terminate and close drop the request, and a job end after either records none.
A suspendAfter job that ends after the terminate hook must record no request. The fire conditions no longer check the terminate hook. So the record check in the job end is the only thing that stops this suspend, and no test covered it. The new test fails when that check loses the terminate term. A second test matches the JavaScript suite. close() drops a request that waits for the run hook, so the run hook starts no suspend.
A suspendAfter request that comes before the first accepted run hook waits for that hook. An image without the run hook never sends it. So the request waits for good, and the worker logged nothing. The earlier skip warning was the only sign of that setup. The worker now logs a warning with callbackId when a request starts to wait because no run hook has been accepted yet. It logs the warning once per worker, so an image without the run hook logs it once, not for each job. The worker decides under the lock and logs after it. The README lists the warning.
A run hook body that fails validation is not accepted, and does not end the wait of a suspendAfter request. The next valid run hook ends it. The README said "the first run hook". It now says "the first accepted run hook", as the code comments do.
ParidelPooya
marked this pull request as ready for review
October 27, 2026 23:30
ParidelPooya
deleted the
fix/microvm-worker-suspend-after-before-run-hook
branch
October 27, 2026 23:31
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR follows up review findings on the matching JS PR aws/aws-durable-execution-sdk-js#1105. The same code exists in the Python MicroVM worker.
suspendAfterrequest that is due before the first accepted run hook now waits for that hook.terminating,closed, orsuspending, because no reachable case reaches them.Why
HTTP jobs can arrive before the run hook. A
suspendAfterjob that ended before the run hook found no target. The worker then logged the skip warning and dropped the request. The run hook set the target later, but its re-check found no request. So the MicroVM kept running during the customer's wait, although the 202 answer had acknowledged the option.With the wait, an image without the run hook never ends the wait. The skip warning no longer appears for such an image. So the worker needs another warning to show that setup.
The terminate hook and
close()clear the request, and a job end after either records none. A suspend refusal clears the request, and a job end during a refusal records none. So the three state checks in the fire conditions cannot change a decision. The JavaScript worker already has nosuspendingcheck for the same reason.Changes
_take_job_suspendkeeps a due request pending while no target exists and no run hook has been accepted. The re-check at the end of the first accepted run hook fires the request when the hook named a MicroVM. When the hook named none, the worker logscannot suspend the MicroVM after a job: the run hook named no MicroVMwithcallbackId, and drops the request. A run hook that fails validation is not accepted, and does not end the wait.a job asked to suspend the MicroVM before the run hook arrived. The worker waits for the run hook to name the MicroVM.withcallbackId. It logs the warning once per worker, not once per job. The job end decides under the lock and logs after it._take_job_suspendno longer checks_terminating,_closed, or_suspending. The clears in the terminate hook,close(), and the suspend paths, and the job-end rule, own cancellation.Testing
suspendAfterjob ends before the run hook arrives. The run hook then arrives. The worker suspends the run hook's MicroVM once, and logs the info line once. This test fails onmain.main, becausemainwarns at the job end.suspendAfterjob ends after the terminate hook. The worker records no request and starts noSuspendMicrovmcall. The record check in the job end is now the only thing that stops this suspend. Removing its_terminatingterm fails this test.close()drops a request that waits for the run hook. The run hook that follows starts no suspend.nonecase oftest_warns_and_does_not_suspend_when_no_run_hook_named_a_microvmis removed. It expected the skip warning when no run hook arrived. The worker now keeps the request until the first accepted run hook, so that case logs the wait warning instead, and the new wait tests cover it. The test's other case, a run hook without a payload before the job, still expects the skip warning and passes.hatch run dev-microvm-worker:cov: 524 passed,worker.py100% coverage.hatch run dev-microvm-worker:typecheckandhatch fmt --checkin the package directory pass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.