Skip to content

fix(microvm-worker): keep a suspend_after request for the run hook - #848

Merged
ParidelPooya merged 6 commits into
mainfrom
fix/microvm-worker-suspend-after-before-run-hook
Oct 27, 2026
Merged

ParidelPooya merged 6 commits into
mainfrom
fix/microvm-worker-suspend-after-before-run-hook

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR follows up review findings on the matching JS PR aws/aws-durable-execution-sdk-js#1105. The same code exists in the Python MicroVM worker.

  1. A suspendAfter request that is due before the first accepted run hook now waits for that hook.
  2. The worker logs a warning, once per worker, when a request starts to wait for the run hook.
  3. The fire conditions no longer check terminating, closed, or suspending, because no reachable case reaches them.

Why

HTTP jobs can arrive before the run hook. A suspendAfter job that ended before the run hook found no target. The worker then logged the skip warning and dropped the request. The run hook set the target later, but its re-check found no request. So the MicroVM kept running during the customer's wait, although the 202 answer had acknowledged the option.

With the wait, an image without the run hook never ends the wait. The skip warning no longer appears for such an image. So the worker needs another warning to show that setup.

The terminate hook and close() clear the request, and a job end after either records none. A suspend refusal clears the request, and a job end during a refusal records none. So the three state checks in the fire conditions cannot change a decision. The JavaScript worker already has no suspending check for the same reason.

Changes

  • _take_job_suspend keeps a due request pending while no target exists and no run hook has been accepted. The re-check at the end of the first accepted run hook fires the request when the hook named a MicroVM. When the hook named none, the worker logs cannot suspend the MicroVM after a job: the run hook named no MicroVM with callbackId, and drops the request. A run hook that fails validation is not accepted, and does not end the wait.
  • When a request starts to wait because no run hook has been accepted yet, the worker logs the warning a job asked to suspend the MicroVM before the run hook arrived. The worker waits for the run hook to name the MicroVM. with callbackId. It logs the warning once per worker, not once per job. The job end decides under the lock and logs after it.
  • _take_job_suspend no longer checks _terminating, _closed, or _suspending. The clears in the terminate hook, close(), and the suspend paths, and the job-end rule, own cancellation.
  • The code comments and the README section "Suspending after a job" describe these rules. The README says "first accepted run hook", and lists the new warning. The README states one rule for terminate and close: they drop the request, and a job end after either records none.

Testing

  • New test: a suspendAfter job ends before the run hook arrives. The run hook then arrives. The worker suspends the run hook's MicroVM once, and logs the info line once. This test fails on main.
  • New test: a job ends before a run hook that names no MicroVM. The worker logs the skip warning once, and does not suspend. This test fails on main, because main warns at the job end.
  • New test: a suspendAfter job ends after the terminate hook. The worker records no request and starts no SuspendMicrovm call. The record check in the job end is now the only thing that stops this suspend. Removing its _terminating term fails this test.
  • New test: close() drops a request that waits for the run hook. The run hook that follows starts no suspend.
  • New test: two jobs whose requests wait for the run hook log the wait warning once. A job whose run hook came first logs no wait warning.
  • The none case of test_warns_and_does_not_suspend_when_no_run_hook_named_a_microvm is removed. It expected the skip warning when no run hook arrived. The worker now keeps the request until the first accepted run hook, so that case logs the wait warning instead, and the new wait tests cover it. The test's other case, a run hook without a payload before the job, still expects the skip warning and passes.
  • hatch run dev-microvm-worker:cov: 524 passed, worker.py 100% coverage.
  • hatch run dev-microvm-worker:typecheck and hatch fmt --check in the package directory pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

HTTP jobs can arrive before the run hook. A suspendAfter job that ended
before the run hook found no target, so the worker logged the skip
warning and dropped the request. The run hook then set the target, but
its re-check found no request. So the MicroVM kept running, although
the 202 answer had acknowledged the option.

A due request with no target now stays pending until the first
accepted run hook. The re-check at the end of that hook fires the
request when the hook named a MicroVM. When the hook named none, the
worker logs the skip warning and drops the request.
The terminate hook and close() clear a pending suspendAfter request,
and a job end after either records none. A suspend refusal clears the
request too, and a job end during a refusal records none. So the
terminating, closed, and suspending checks in the fire conditions
never decide a reachable case. This matches the JavaScript worker.

The fire conditions now check only for running work. The code comment
and the README state one rule: terminate and close drop the request,
and a job end after either records none.
A suspendAfter job that ends after the terminate hook must record no
request. The fire conditions no longer check the terminate hook. So
the record check in the job end is the only thing that stops this
suspend, and no test covered it. The new test fails when that check
loses the terminate term.

A second test matches the JavaScript suite. close() drops a request
that waits for the run hook, so the run hook starts no suspend.
A suspendAfter request that comes before the first accepted run hook
waits for that hook. An image without the run hook never sends it. So
the request waits for good, and the worker logged nothing. The earlier
skip warning was the only sign of that setup.

The worker now logs a warning with callbackId when a request starts
to wait because no run hook has been accepted yet. It logs the warning
once per worker, so an image without the run hook logs it once, not
for each job. The worker decides under the lock and logs after it.
The README lists the warning.
A run hook body that fails validation is not accepted, and does not
end the wait of a suspendAfter request. The next valid run hook ends
it. The README said "the first run hook". It now says "the first
accepted run hook", as the code comments do.

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@ParidelPooya
ParidelPooya marked this pull request as ready for review October 27, 2026 23:30
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime October 27, 2026 23:30 — with GitHub Actions Active
@ParidelPooya
ParidelPooya merged commit f133b78 into main Oct 27, 2026
16 checks passed
@ParidelPooya
ParidelPooya deleted the fix/microvm-worker-suspend-after-before-run-hook branch October 27, 2026 23:31

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — c4994588 Deployed Oct 27, 2026 by ParidelPooya via notify / Generate notification summary #333
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants