diff --git a/docs/advanced/configuration.md b/docs/advanced/configuration.md index ed702ab0c1..e1b9f6389a 100644 --- a/docs/advanced/configuration.md +++ b/docs/advanced/configuration.md @@ -34,11 +34,22 @@ public class OrderProcessor extends DurableHandler { | `withLambdaClientBuilder()` | Custom AWS Lambda client | Auto-configured Lambda client | | `withSerDes()` | Serializer for step results | Jackson with default settings | | `withExecutorService()` | Thread pool for user-defined operations | Cached daemon thread pool | +| `withInvocationExecutorFactory()` | Factory for isolated per-invocation user executors | Disabled | | `withLoggerConfig()` | Logger behavior configuration | Suppress logs during replay | | `withPollingStrategy()` | Backend polling strategy | Exponential backoff: 1s base, 2x rate, FULL jitter, 10s max | | `withCheckpointDelay()` | How often the SDK checkpoints updates | `Duration.ofSeconds(0)` (as soon as possible) | -The `withExecutorService()` option configures the thread pool used for running user-defined operations. Internal SDK coordination (checkpoint batching, polling) runs on an SDK-managed thread pool. +The `withExecutorService()` option configures a caller-owned executor shared by every invocation that uses the configuration. The SDK never shuts it down. To isolate concurrent Lambda Managed Instances invocations, configure a factory instead: + +```java +DurableConfig.builder() + .withInvocationExecutorFactory(() -> Executors.newFixedThreadPool(4)) + .build(); +``` + +The factory is called once per invocation and may be called concurrently on Lambda Managed Instances. It must return a fresh executor. The invocation caller waits for actual admitted work and End before shutting that executor down, including after cancellation. Code that ignores interruption can delay this shutdown beyond the policy budget; executor termination is then observed with any cleanup time that remains. Returning an executor already leased to an active invocation is rejected. The factory and `withExecutorService()` are mutually exclusive. Both options run all user code, including plugin-enabled root handlers, steps, and child contexts. Internal SDK coordination such as checkpoint batching and polling remains on the SDK-managed executor. `toBuilder()` and the local test runner preserve the factory, including across replay/resume. + +A bounded executor supplied through `withExecutorService()` remains shared. On Lambda Managed Instances, concurrent roots can occupy all of its workers while waiting for queued steps. Use the invocation factory when capacity must be isolated between native invocations. Isolation does not define the capacity required by arbitrarily nested synchronous child contexts; size or design those separately. ### Dynamic plugin loading @@ -95,3 +106,36 @@ file retention. Custom serializers can override `serialize(value, SerDesContext)` and `deserialize(data, typeToken, SerDesContext)` to receive stable execution/entity identity. Existing implementations inherit defaults that call the original methods. + +## Invocation deadline cleanup + +The SDK captures the Lambda invocation's remaining time once and uses monotonic +cutoffs for admission and its wait policy. Deadline exhaustion selects a retryable +invocation interruption; it is not checkpointed as a business-operation failure. +Catching an admission rejection cannot turn the stopped invocation into a +successful fallback. Completed operation results and failures remain available for +replay. + +Cleanup allowances determine when the SDK requests cooperative cancellation and +stops ordinary admission. They do not make cancellation equivalent to actual task +exit. Finalization still waits for admitted user work, continuations and in-flight +checkpoint/poll transport, and runs the matching invocation End on its owner. +The runtime caller waits for that owner and +End to finish, including after a deadline or interruption. End is not moved to a +completion-observer thread or deferred until after the SDK call returns. + +Interruption is recorded, remaining required cleanup still runs, and the interrupt +flag is restored. A terminal execution result acknowledged by the +service remains authoritative for a handled deadline cutoff. An accepted result +does not suppress an otherwise-propagating delivery or metadata failure, and an +available JVM fatal keeps its original identity and priority. + +These cutoffs do not guarantee bounded caller return or worker reclamation. +Already-running application I/O, synchronous initial-state reads, custom clients, +plugin hooks and other code that ignores interruption can keep the caller waiting +for actual cleanup beyond the policy budget. An executor that runs the root inline, +including a saturated `CallerRunsPolicy`, can also block the runtime caller before +it reaches a timeout-aware wait. Use finite downstream I/O timeouts and executor +configurations that allow progress. Cancellation cannot undo a request already +accepted by the service; interrupted side effects still require the documented +retry and idempotency strategy. diff --git a/docs/design.md b/docs/design.md index 197ee6a7c7..1fc7f8aa3e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -145,19 +145,34 @@ public class MyHandler extends DurableHandler { | `lambdaClientBuilder` | Auto-created `LambdaClient` for current region, primed for performance (see [`DurableConfig.java`](../sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java)) | | `serDes` | `JacksonSerDes` | | `executorService` | `Executors.newCachedThreadPool()` (for user-defined operations only) | +| `invocationExecutorFactory` | Disabled; when configured, creates an SDK-owned executor for each invocation | | `loggerConfig` | `LoggerConfig.defaults()` (suppress replay logs) | | `pollingStrategy` | Exponential backoff: 1s base, 2x rate, FULL jitter, 10s max | | `checkpointDelay` | `Duration.ofSeconds(0)` (checkpoint as soon as possible) | ### Thread Pool Architecture +Each invocation tracks logical task outcomes separately from actual task exit. +Deadline and cleanup cutoffs fence admission and request cooperative cancellation, +while finalization still waits for admitted work and in-flight transport to finish. +The matching invocation End runs on its owner, and the runtime caller waits for +that owner and End before returning. Code that ignores interruption can therefore +outlast the policy budget; see [invocation deadline cleanup](advanced/configuration.md#invocation-deadline-cleanup). + The SDK uses two separate thread pools with distinct responsibilities: **User Executor (`DurableConfig.executorService`):** -- Runs user-defined operations (the code passed to `ctx.step()` and `ctx.stepAsync()`) +- Runs the root handler, steps, child-context user code, and map/parallel coordination - Configurable via `DurableConfig.builder().withExecutorService()` +- Caller-owned and shared across invocations; the SDK never shuts it down - Default: cached daemon thread pool +**Invocation Executor Factory (`DurableConfig.invocationExecutorFactory`):** +- Optional alternative to the shared user executor +- Creates one executor per invocation for all root, step, child-context user code, and map/parallel coordination +- SDK-owned: the invocation caller waits for actual admitted work and End before requesting executor shutdown, including after cancellation. Non-cooperative work can delay that shutdown beyond the policy budget. Executor termination is observed with any cleanup time that remains. +- Must return a fresh executor and cannot be combined with `withExecutorService()` + **Internal Executor (`InternalExecutor.INSTANCE`):** - Runs SDK coordination tasks: checkpoint batching, polling for wait completion - Dedicated cached thread pool with daemon threads named `durable-sdk-internal-*` diff --git a/lmi-tests/README.md b/lmi-tests/README.md index a88c5b73f9..97f7e50e7b 100644 --- a/lmi-tests/README.md +++ b/lmi-tests/README.md @@ -19,6 +19,15 @@ those original snapshots and retain the existing zero-work and +16-thread limits Newly discovered request identities invalidate the warm cohort instead of moving its original baseline or final measurement to another request. +The current deadline controls fence admission and request cooperative cancellation, +but the SDK still awaits actual owner cleanup, admitted transport and the matching +invocation End before returning. Non-cooperative work can therefore keep the +runtime caller blocked beyond its deadline. This remains a partial change for +[#726](https://github.com/aws/aws-durable-execution-sdk-java/issues/726), not proof +that every desired LMI lifecycle bound is satisfied. The acceptance limits below +remain unchanged; local tests do not replace a deployed cloud acceptance run, and +eventual cleanup cannot turn an earlier lifecycle violation into a pass. + ## Test design * Each fixture is a published durable function with LMI invocation concurrency @@ -30,7 +39,7 @@ its original baseline or final measurement to another request. SDK's minimum JDK prevents fixture-only changes from introducing newer JDK APIs into the shared Maven reactor. On manual dispatch, a separate serialized cloud job builds the selected commit, deploys all five - fixture functions, runs all 13 cases, then collects evidence. A persistent + fixture functions, runs all 12 gating cases, then collects evidence. A persistent CloudFormation stack owns all five functions and log groups. The stack, functions and bucket remain after both successful and failed test runs; later deployments update the same resources. Each uses 2 GiB / 1 vCPU and a single @@ -65,17 +74,14 @@ its original baseline or final measurement to another request. Each real checkpoint/poll call has a request-local call ID paired with its `finally` exit. Missing or mismatched exits prevent cleanup from reusing the fixture, and an exit after wrapper return remains a lifecycle failure. -* A bounded same-JVM root barrier establishes the fixed-pool reproduction. - `fixed2` uses a shared two-thread pool so two roots starve their first steps. - `nested2` uses four threads and a separate cohort-scoped child barrier. Both - child handlers must enter before either schedules nested map/parallel work. - The child barrier has an independent 8-second setup budget; the nested progress - budget starts when it releases, not when the roots passed their earlier barrier. - A broken child barrier fails setup without growing the pool; an escape after - successful child admission still fails the starvation assertion. - Other cases hold steps with private S3 control objects. The driver requires - distinct request IDs active in the same JVM. Placement has its own deadline - and failure category. Environment replacement is never worker recovery. +* A bounded same-JVM root barrier verifies that `fixed2` gives each invocation + its own two-thread executor. `nested2` is retained in the persistent stack for + baseline diagnostics but does not produce a gating LMI case. It uses four + shared threads and a separate cohort-scoped child barrier before scheduling + nested map/parallel work. Other cases hold steps with private S3 control + objects. The driver requires distinct request IDs active in the same JVM. + Placement has its own deadline and failure category. Environment replacement + is never worker recovery. * Timeout victims start before healthy peers, leaving the peers time to hold the other runtime slots during recovery. Diagnostics capture the real context deadline; no fake clock, context, checkpoint backend, or time-skipping runner @@ -170,8 +176,9 @@ python3 lmi-tests/cloud_suite.py collect ``` The deploy command creates or updates and retains `default1`, `default2`, `default8`, -`fixed2`, and `nested2` together. The test command requires all five and runs the -full suite. CI publishes one combined artifact, named `lmi-e2e-RUN-ATTEMPT`, with +`fixed2`, and `nested2` together. The test command requires all five and runs 12 +gating cases; `nested2` is retained but excluded. CI publishes one combined +artifact, named `lmi-e2e-RUN-ATTEMPT`, with the single `template.json` and per-function configuration snapshots. The CloudFormation execution role needs permission to manage the function scaling configuration; the driver calls `lambda:GetFunctionScalingConfig` to verify it @@ -219,7 +226,7 @@ barrier diagnostics never authorize a retry. Every attempted cohort remains in the escape ledger, so a later successful cohort cannot hide an earlier pool escape. `timeouts/*.json` distinguishes server timeout logs from an SDK deadline -cancellation that returns early with an invocation error. A client HTTP timeout +cancellation reported as an invocation error after owner cleanup. A client HTTP timeout is a collection error. Deadline victims use asynchronous service invocation so a longer durable retry does not consume the driver HTTP timeout; invocation outcomes come from request-correlated runtime logs. Invocation request failures @@ -251,9 +258,10 @@ a normally ordered retry cannot replace an admitted participant. After stopping durable executions, cleanup allows any pre-stop stubborn retry its configured residual hold (capped at 120 seconds), plus the existing 30-second collection margin and 20-second quiet period. This cleanup allowance does not alter -the original invocation deadline + 5s or progress/recovery acceptance limits. No virtual-thread executor variant is deployed -until its executor contract is defined; default cached and shared fixed pools -are covered separately. +the original invocation deadline + 5s or progress/recovery acceptance limits. +No virtual-thread executor variant is deployed until its executor contract is defined. +The cloud suite gates default cached and invocation-local fixed pools; the shared +fixed nesting fixture remains available as a non-gating baseline diagnostic. CloudFormation's native [FunctionScalingConfig](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-lambda-function-functionscalingconfig.html) sets the limits in the resource declaration. LMI provisioning and version behavior diff --git a/lmi-tests/cloud_suite.py b/lmi-tests/cloud_suite.py index 95d51e25bc..b1666623d9 100644 --- a/lmi-tests/cloud_suite.py +++ b/lmi-tests/cloud_suite.py @@ -25,7 +25,7 @@ OWNER = "java-sdk-lmi-e2e" DEFAULT_STACK = "java-lmi-e2e" FIXTURES = {"default1": (1, "default"), "default2": (2, "default"), - "default8": (8, "default"), "fixed2": (2, "fixed"), "nested2": (2, "nested")} + "default8": (8, "default"), "fixed2": (2, "invocation-fixed"), "nested2": (2, "nested")} PEER_LOG_OBSERVATION_SECONDS = 20 PEER_PLACEMENT_SECONDS = 60 @@ -201,7 +201,7 @@ def deploy_fixtures(manifest, tags, seconds=1800): wait_stack(manifest["stack"], expected, remaining_budget(deadline, seconds)) stack = existing_stack(manifest["stack"]) outputs = {output["OutputKey"]: output["OutputValue"] for output in stack["Outputs"]} - require(set(outputs) == set(FIXTURES), "The stack must expose all five LMI fixtures") + require(set(outputs) == set(FIXTURES), "The stack must expose all required LMI fixtures") for fixture in FIXTURES: arn = outputs[fixture] record_fixture(manifest, fixture, arn) @@ -893,10 +893,10 @@ def collect_warm_snapshots(cloud, items, seconds=30): def cases_for_fixture(cloud, fixture): if fixture == "default1": return [("baseline-concurrency1", lambda: replay_case(cloud, fixture, "baseline"))] - if fixture in {"fixed2", "nested2"}: - scenario = "fixed" if fixture == "fixed2" else "nested" - return [("fixed-two-roots" if fixture == "fixed2" else "fixed-nested-map-parallel", - lambda: fixed_case(cloud, fixture, scenario))] + if fixture == "fixed2": + return [("isolated-fixed-two-roots", lambda: fixed_case(cloud, fixture, "fixed"))] + if fixture == "nested2": + return [] cases = [(fixture + "-isolation", lambda: overlap_case(cloud, fixture)), (fixture + "-suspend-cleanup-replay", lambda: replay_case(cloud, fixture, "suspend")), (fixture + "-timeout-recovery", lambda: timeout_case(cloud, fixture))] @@ -1022,7 +1022,7 @@ def revalidate_cases(cloud, records): def run_tests(): manifest = json.loads(MANIFEST.read_text()) if set(manifest["functions"]) != set(FIXTURES): - raise PreconditionError("All five functions must be deployed before running the cloud suite") + raise PreconditionError("All required functions must be deployed before running the cloud suite") cloud = Cloud(manifest, ARTIFACTS) suite = ET.Element("testsuite", name="LMI cloud lifecycle") records = [] diff --git a/lmi-tests/cloud_support.py b/lmi-tests/cloud_support.py index e7a18b9aac..0ed28528be 100644 --- a/lmi-tests/cloud_support.py +++ b/lmi-tests/cloud_support.py @@ -213,7 +213,7 @@ def assert_fixed(events, markers, progress_start="BARRIER_PASSED"): require(len({e["requestId"] for e in entered}) == len(markers), "Missing distinct runtime invocations") require(max(e["sequence"] for e in entered) < min(e["sequence"] for e in passed), "Root invocations did not overlap at the barrier") - require(not selected(events, "ESCAPE"), "Shared fixed executor starved its own queued work") + require(not selected(events, "ESCAPE"), "Fixed executor starved its own queued work") # Child admission is one cohort-wide phase. A participant scheduled later must # not get a fresh progress budget after its peer has already left the barrier. cohort_start = min((e["nanos"] for e in selected(events, progress_start)), default=None) @@ -222,7 +222,7 @@ def assert_fixed(events, markers, progress_start="BARRIER_PASSED"): start = selected(events, progress_start, marker) started = cohort_start if progress_start == "CHILD_BARRIER_PASSED" else (start[0]["nanos"] if start else None) require(start and progress and 0 <= (progress[-1]["nanos"] - started) < 8_000_000_000, - "Shared executor did not progress within budget") + "Fixed executor did not progress within budget") def assert_nested_reached(events, markers): diff --git a/lmi-tests/src/main/java/software/amazon/lambda/durable/lmi/LifecycleHandler.java b/lmi-tests/src/main/java/software/amazon/lambda/durable/lmi/LifecycleHandler.java index 4db3041e24..b8ce3845fb 100644 --- a/lmi-tests/src/main/java/software/amazon/lambda/durable/lmi/LifecycleHandler.java +++ b/lmi-tests/src/main/java/software/amazon/lambda/durable/lmi/LifecycleHandler.java @@ -25,6 +25,7 @@ import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicReference; import software.amazon.awssdk.services.lambda.model.OperationType; import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.DurableContext; @@ -75,27 +76,42 @@ public void handleRequest(InputStream source, OutputStream destination, Context var input = WIRE.deserialize( new String(source.readAllBytes(), StandardCharsets.UTF_8), TypeToken.get(DurableExecutionInput.class)); var trace = new InvocationTrace(readUserInput(input), runtime, input.durableExecutionArn()); + var invocationExecutor = new AtomicReference(); trace.wrapperEnter(); var status = "THREW"; try { - var config = DurableConfig.builder() - .withExecutorService(shared.getExecutorService()) - .withDurableExecutionClient(new ObservedClient(shared.getDurableExecutionClient(), trace)) - .build(); + var configBuilder = DurableConfig.builder() + .withDurableExecutionClient(new ObservedClient(shared.getDurableExecutionClient(), trace)); + if ("invocation-fixed".equals(System.getenv("LMI_EXECUTOR"))) { + configBuilder.withInvocationExecutorFactory(() -> { + var executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(2); + invocationExecutor.set(executor); + return executor; + }); + } else { + configBuilder.withExecutorService(shared.getExecutorService()); + } + var config = configBuilder.build(); var response = DurableExecutor.execute( input, runtime, TypeToken.get(FixtureInput.class), - (value, context) -> handle(value, context, trace), + (value, context) -> handle(value, context, trace, executor(invocationExecutor)), config); destination.write(WIRE.serialize(response).getBytes(StandardCharsets.UTF_8)); status = response.status().name(); } finally { - trace.snapshot((ThreadPoolExecutor) shared.getExecutorService()); + trace.snapshot(executor(invocationExecutor)); trace.wrapperExit(status); } } + private ThreadPoolExecutor executor(AtomicReference invocationExecutor) { + return invocationExecutor.get() != null + ? invocationExecutor.get() + : (ThreadPoolExecutor) shared.getExecutorService(); + } + private static FixtureInput readUserInput(DurableExecutionInput input) { var operation = input.initialExecutionState().operations().stream() .filter(op -> op.type() == OperationType.EXECUTION) @@ -104,7 +120,8 @@ private static FixtureInput readUserInput(DurableExecutionInput input) { return JSON.deserialize(operation.executionDetails().inputPayload(), TypeToken.get(FixtureInput.class)); } - String handle(FixtureInput input, DurableContext context, InvocationTrace trace) { + String handle( + FixtureInput input, DurableContext context, InvocationTrace trace, ThreadPoolExecutor invocationExecutor) { trace.rootEnter(); try { if (!context.isReplaying() @@ -113,7 +130,7 @@ String handle(FixtureInput input, DurableContext context, InvocationTrace trace) trace.event("PLACEMENT_MISS"); return "PLACEMENT_MISS"; } - return runScenario(input, context, trace); + return runScenario(input, context, trace, invocationExecutor); } finally { if ("suspend".equals(input.scenario())) { trace.event("CLEANUP_ENTER"); @@ -124,13 +141,14 @@ String handle(FixtureInput input, DurableContext context, InvocationTrace trace) } } - private String runScenario(FixtureInput input, DurableContext context, InvocationTrace trace) { + private String runScenario( + FixtureInput input, DurableContext context, InvocationTrace trace, ThreadPoolExecutor invocationExecutor) { return switch (input.scenario()) { case "baseline", "replay", "suspend" -> replay(input, context, trace); case "hold", "probe" -> context.step("held-step", String.class, step -> hold(input, trace), NO_RETRY); case "timeout", "failure-inflight", "return-inflight" -> inFlight(input, context, trace); case "stubborn" -> stubborn(input, context, trace); - case "fixed", "nested" -> fixed(input, context, trace); + case "fixed", "nested" -> fixed(input, context, trace, invocationExecutor); case "success" -> context.step("success", String.class, step -> body(trace, "success", input.marker())); case "failure" -> context.step("failure", String.class, step -> fail(trace, input.marker()), NO_RETRY); default -> throw new IllegalArgumentException("Unknown fixture scenario"); @@ -226,8 +244,7 @@ private String stubborn(FixtureInput input, DurableContext context, InvocationTr return input.marker(); } - private String fixed(FixtureInput input, DurableContext context, InvocationTrace trace) { - var pool = (ThreadPoolExecutor) shared.getExecutorService(); + private String fixed(FixtureInput input, DurableContext context, InvocationTrace trace, ThreadPoolExecutor pool) { awaitRootBarrier(input, trace); if ("nested".equals(input.scenario())) { var value = context.step("success", String.class, step -> body(trace, "success", input.marker())); diff --git a/lmi-tests/tests/test_evidence.py b/lmi-tests/tests/test_evidence.py index d7a814e01a..cb0973bfbc 100644 --- a/lmi-tests/tests/test_evidence.py +++ b/lmi-tests/tests/test_evidence.py @@ -186,14 +186,15 @@ def test_artifacts_redact_control_credentials_in_nested_payloads(self): value = {"controlUrl": "secret", "InputPayload": '{"controlUrl":"https://bucket/key?X-Amz-Signature=secret"}'} self.assertNotIn("secret", json.dumps(scrub(value))) - def test_one_stack_declares_all_five_functions_and_their_scaling_limits(self): + def test_one_stack_declares_all_functions_and_their_scaling_limits(self): manifest = {"stack": "test", "role": "role", "bucket": "bucket", "provider": "provider", "commit": "sha", "invocationTimeout": 60, "codeSha256": "digest", "codeKey": "code/digest.jar", "runId": "run-1"} spec = template(manifest) functions = [r["Properties"] for r in spec["Resources"].values() if r["Type"] == "AWS::Lambda::Function"] - self.assertEqual(5, len(functions)) + self.assertEqual(len(FIXTURES), len(functions)) self.assertEqual(set(FIXTURES), set(spec["Outputs"])) - self.assertEqual(5, sum(r["Type"] == "AWS::Logs::LogGroup" for r in spec["Resources"].values())) + self.assertEqual( + len(FIXTURES), sum(r["Type"] == "AWS::Logs::LogGroup" for r in spec["Resources"].values())) concurrencies = set() for function in functions: capacity = function["CapacityProviderConfig"]["LambdaManagedInstancesCapacityProviderConfig"] @@ -213,7 +214,7 @@ def test_one_stack_declares_all_five_functions_and_their_scaling_limits(self): self.assertEqual({1, 2, 8}, concurrencies) fixed_executor = spec["Resources"]["fixed2Function"]["Properties"]["Environment"]["Variables"] nested_executor = spec["Resources"]["nested2Function"]["Properties"]["Environment"]["Variables"] - self.assertEqual("fixed", fixed_executor["LMI_EXECUTOR"]) + self.assertEqual("invocation-fixed", fixed_executor["LMI_EXECUTOR"]) self.assertEqual("nested", nested_executor["LMI_EXECUTOR"]) self.assertGreater(PEER_LOG_OBSERVATION_SECONDS, 9) @@ -226,7 +227,10 @@ def test_full_suite_preserves_all_cases_on_their_own_fixture(self): try: for fixture in FIXTURES: cases = cases_for_fixture(cloud, fixture) - self.assertTrue(cases) + if fixture == "nested2": + self.assertEqual([], cases) + else: + self.assertTrue(cases) names.extend(name for name, _ in cases) for _, run in cases: run() @@ -237,15 +241,15 @@ def test_full_suite_preserves_all_cases_on_their_own_fixture(self): finally: for p in patches: p.stop() - self.assertEqual(13, len(names)) - self.assertEqual(13, len(set(names))) + self.assertEqual(12, len(names)) + self.assertEqual(12, len(set(names))) @patch("cloud_suite.Cloud") - def test_cloud_suite_requires_all_five_functions(self, cloud): + def test_cloud_suite_requires_all_functions(self, cloud): with TemporaryDirectory() as directory: path = Path(directory) / "manifest.json" path.write_text(json.dumps({"functions": {"default1": {}}})) - with patch("cloud_suite.MANIFEST", path), self.assertRaisesRegex(PreconditionError, "All five"): + with patch("cloud_suite.MANIFEST", path), self.assertRaisesRegex(PreconditionError, "All required"): run_tests() cloud.assert_not_called() diff --git a/lmi-tests/tests/test_persistence.py b/lmi-tests/tests/test_persistence.py index bafc949e31..50f56aaa7b 100644 --- a/lmi-tests/tests/test_persistence.py +++ b/lmi-tests/tests/test_persistence.py @@ -23,6 +23,16 @@ def stack(self, status="CREATE_COMPLETE"): "Outputs": [{"OutputKey": key, "OutputValue": "function:" + key + ":$LATEST.PUBLISHED"} for key in FIXTURES]} + def test_fixed_fixture_uses_invocation_factory(self): + resources = template(self.manifest())["Resources"] + + self.assertEqual( + "invocation-fixed", + resources["fixed2Function"]["Properties"]["Environment"]["Variables"]["LMI_EXECUTOR"]) + self.assertEqual( + "nested", + resources["nested2Function"]["Properties"]["Environment"]["Variables"]["LMI_EXECUTOR"]) + @patch("cloud_suite.save") @patch("cloud_suite.wait_stack") @patch("cloud_suite.wait_for_idle_functions") @@ -51,7 +61,7 @@ def record_one(manifest, fixture, arn): self.assertEqual("run-2", after["Environment"]["Variables"]["LMI_TEST_RUN_ID"]) self.assertNotIn("TimeoutInMinutes", api.call_args_list[1].args[2]) self.assertEqual(["CREATE_COMPLETE", "UPDATE_COMPLETE"], [call.args[1] for call in wait.call_args_list]) - self.assertEqual(10, verify.call_count) + self.assertEqual(2 * len(FIXTURES), verify.call_count) idle.assert_called_once() @patch("cloud_suite.save") @@ -66,7 +76,7 @@ def test_unchanged_stack_still_gets_readback_without_recreation(self, api, exist deploy_fixtures(self.manifest(), []) self.assertEqual("update-stack", api.call_args.args[1]) wait.assert_not_called() - self.assertEqual(5, verify.call_count) + self.assertEqual(len(FIXTURES), verify.call_count) @patch("cloud_suite.save") @patch("cloud_suite.existing_stack") diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/HandlerMdcIntegrationTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/HandlerMdcIntegrationTest.java index dc28b137d7..55ef846b63 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/HandlerMdcIntegrationTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/HandlerMdcIntegrationTest.java @@ -24,6 +24,7 @@ import org.slf4j.MDC; import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.execution.ExecutionManager; import software.amazon.lambda.durable.model.ExecutionStatus; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; @@ -325,10 +326,19 @@ private static void awaitMdcLatch(CountDownLatch latch) { private static void awaitMdcCallerJoin(Thread caller) { var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(3); while (System.nanoTime() < deadline) { - if (caller.getState() == Thread.State.WAITING - && Arrays.stream(caller.getStackTrace()) + var state = caller.getState(); + var frames = caller.getStackTrace(); + var legacyJoin = Arrays.stream(frames) + .anyMatch(frame -> frame.getClassName().equals(CompletableFuture.class.getName()) + && frame.getMethodName().equals("join")); + var budgetedWait = Arrays.stream(frames) + .anyMatch(frame -> frame.getClassName().equals(ExecutionManager.class.getName()) + && frame.getMethodName().equals("awaitInvocationOutcome")) + && Arrays.stream(frames) .anyMatch(frame -> frame.getClassName().equals(CompletableFuture.class.getName()) - && frame.getMethodName().equals("join"))) return; + && frame.getMethodName().equals("get")); + if ((state == Thread.State.WAITING || state == Thread.State.TIMED_WAITING) && (legacyJoin || budgetedWait)) + return; LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(1)); } fail("Caller did not attach its completion observer before releasing the handler"); diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationPreparationAdmissionProbeTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationPreparationAdmissionProbeTest.java index 73a536a0f9..cc764e756f 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationPreparationAdmissionProbeTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationPreparationAdmissionProbeTest.java @@ -21,6 +21,7 @@ import software.amazon.lambda.durable.config.*; import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; import software.amazon.lambda.durable.execution.DurableExecutor; +import software.amazon.lambda.durable.execution.ExecutionManager; import software.amazon.lambda.durable.model.DurableExecutionInput; import software.amazon.lambda.durable.model.ExecutionStatus; import software.amazon.lambda.durable.model.WaitForConditionResult; @@ -273,7 +274,9 @@ private static void awaitDrain(Thread root) { var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(3); while (System.nanoTime() < deadline) { if (Arrays.stream(root.getStackTrace()) - .anyMatch(frame -> frame.getMethodName().equals("waitForRunningUserHandlers"))) return; + .anyMatch(frame -> frame.getClassName().equals(ExecutionManager.class.getName()) + && (frame.getMethodName().equals("waitForRunningUserHandlers") + || frame.getMethodName().equals("waitForRunningUserHandlersUntil")))) return; LockSupport.parkNanos(100_000); } fail("Root did not begin draining the pending child"); diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/LegacyEndThreadProbeTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/LegacyEndThreadProbeTest.java index dd74fccbbc..6e70d6503d 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/LegacyEndThreadProbeTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/LegacyEndThreadProbeTest.java @@ -19,6 +19,7 @@ import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.TypeToken; import software.amazon.lambda.durable.execution.DurableExecutor; +import software.amazon.lambda.durable.execution.ExecutionManager; import software.amazon.lambda.durable.model.DurableExecutionInput; import software.amazon.lambda.durable.model.ExecutionStatus; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; @@ -147,10 +148,19 @@ public void onInvocationEnd(InvocationEndInfo info) { private static void awaitCallerJoin(Thread caller) { var end = System.nanoTime() + TimeUnit.SECONDS.toNanos(3); while (System.nanoTime() < end) { - if (caller.getState() == Thread.State.WAITING - && Arrays.stream(caller.getStackTrace()) + var state = caller.getState(); + var frames = caller.getStackTrace(); + var legacyJoin = Arrays.stream(frames) + .anyMatch(frame -> frame.getClassName().equals(CompletableFuture.class.getName()) + && frame.getMethodName().equals("join")); + var budgetedWait = Arrays.stream(frames) + .anyMatch(frame -> frame.getClassName().equals(ExecutionManager.class.getName()) + && frame.getMethodName().equals("awaitInvocationOutcome")) + && Arrays.stream(frames) .anyMatch(frame -> frame.getClassName().equals(CompletableFuture.class.getName()) - && frame.getMethodName().equals("join"))) return; + && frame.getMethodName().equals("get")); + if ((state == Thread.State.WAITING || state == Thread.State.TIMED_WAITING) && (legacyJoin || budgetedWait)) + return; LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(1)); } fail("invocation caller did not reach its future join before releasing the handler"); diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/SerializationCancellationOtelIntegrationTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/SerializationCancellationOtelIntegrationTest.java new file mode 100644 index 0000000000..726f4ecb93 --- /dev/null +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/SerializationCancellationOtelIntegrationTest.java @@ -0,0 +1,144 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.otel; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; +import static software.amazon.lambda.durable.otel.SpanAttributes.DURABLE_INVOCATION_STATUS; + +import com.amazonaws.services.lambda.runtime.Context; +import io.opentelemetry.api.GlobalOpenTelemetry; +import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter; +import io.opentelemetry.sdk.trace.SdkTracerProvider; +import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.client.DurableExecutionClient; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.execution.DurableExecutor; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.serde.JacksonSerDes; + +@Timeout(20) +class SerializationCancellationOtelIntegrationTest { + @BeforeEach + @AfterEach + void clearState() { + GlobalOpenTelemetry.resetForTest(); + DeterministicIdGenerator.clearSharedStateForTest(); + DurableSamplingDecision.clearSharedStateForTest(); + OtelPluginAutoConfigurationState.resetInstalledForTest(); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void cancellationDuringSmallResultSerializationDoesNotCompleteWorkflow(boolean executionView) throws Exception { + var exporter = InMemorySpanExporter.create(); + var provider = SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)); + var pluginConfig = OtelPluginConfig.builder() + .contextExtractor(() -> null) + .enableMdc(false) + .build(); + var factory = executionView + ? ExecutionOtelPlugin.factory(provider, pluginConfig) + : InvocationOtelPlugin.factory(provider, pluginConfig); + var serializationEntered = new CountDownLatch(1); + var serializationInterrupted = new CountDownLatch(1); + var releaseSerialization = new CountDownLatch(1); + var caller = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var workers = Executors.newSingleThreadExecutor(); + var callers = Executors.newSingleThreadExecutor(); + var serializer = new JacksonSerDes() { + @Override + public String serialize(Object value) { + if ("result".equals(value)) { + serializationEntered.countDown(); + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (true) { + try { + assertTrue(releaseSerialization.await( + Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + break; + } catch (InterruptedException ignored) { + serializationInterrupted.countDown(); + } + } + } + return super.serialize(value); + } + }; + var config = DurableConfig.builder() + .withExecutorService(workers) + .withDurableExecutionClient(mock(DurableExecutionClient.class)) + .withSerDes(serializer) + .withPlugins(factory) + .build(); + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(10_000); + when(runtime.getAwsRequestId()).thenReturn("serialization-cancellation"); + var input = new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/otel/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.now()) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + try { + var response = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input, runtime, TypeToken.get(String.class), (value, context) -> "result", config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertTrue(serializationEntered.await(5, TimeUnit.SECONDS)); + caller.get().interrupt(); + assertTrue(serializationInterrupted.await(5, TimeUnit.SECONDS)); + assertThrows(TimeoutException.class, () -> response.get(100, TimeUnit.MILLISECONDS)); + releaseSerialization.countDown(); + var failure = assertInstanceOf( + UnrecoverableDurableExecutionException.class, + assertThrows(ExecutionException.class, () -> response.get(5, TimeUnit.SECONDS)) + .getCause()); + assertTrue(failure.isRetryable()); + assertTrue(callerInterrupted.get()); + var spans = exporter.getFinishedSpanItems(); + var invocations = spans.stream() + .filter(span -> span.getName().equals("Invocation")) + .toList(); + assertEquals(1, invocations.size(), "The real plugin must still export the completed invocation"); + assertAll( + () -> assertEquals( + "RETRYING", invocations.get(0).getAttributes().get(DURABLE_INVOCATION_STATUS)), + () -> assertTrue( + spans.stream().noneMatch(span -> span.getName().equals("Workflow")), + "A cancelled preparation must not emit a terminal successful Workflow span")); + } finally { + releaseSerialization.countDown(); + workers.shutdownNow(); + callers.shutdownNow(); + assertTrue(workers.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + exporter.close(); + } + } +} diff --git a/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java b/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java index 8967ad2e01..58a54e4ae8 100644 --- a/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java +++ b/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java @@ -10,6 +10,9 @@ import java.util.ArrayList; import java.util.Collections; import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.Operation; @@ -204,4 +207,36 @@ void await() { assertTrue(ex.getMessage().contains("Invalid suspension. No operation is pending.")); assertTrue(ex.getMessage().contains("unregistered-op")); } + + @Test + void preservesInvocationExecutorFactoryAcrossReinvocations() { + var executors = new CopyOnWriteArrayList(); + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> { + var executor = Executors.newFixedThreadPool(2); + executors.add(executor); + return executor; + }) + .build(); + var runner = LocalDurableTestRunner.create( + String.class, + (input, context) -> { + context.wait("wait", Duration.ofMinutes(1)); + return input; + }, + config); + + try { + var first = runner.run("test"); + runner.advanceTime(); + var second = runner.run("test"); + + assertEquals(ExecutionStatus.PENDING, first.getStatus()); + assertEquals(ExecutionStatus.SUCCEEDED, second.getStatus()); + assertEquals(2, executors.size()); + assertTrue(executors.stream().allMatch(ExecutorService::isTerminated)); + } finally { + executors.forEach(ExecutorService::shutdownNow); + } + } } diff --git a/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java b/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java index 6b9bda1071..2640f3ab29 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java @@ -8,9 +8,11 @@ import java.util.ArrayList; import java.util.List; import java.util.Objects; +import java.util.Optional; import java.util.Properties; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.function.Supplier; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider; @@ -95,6 +97,7 @@ public final class DurableConfig { private final DurableExecutionClient durableExecutionClient; private final SerDes serDes; private final ExecutorService executorService; + private final Supplier invocationExecutorFactory; private final LoggerConfig loggerConfig; private final PollingStrategy pollingStrategy; private final Duration checkpointDelay; @@ -103,6 +106,10 @@ public final class DurableConfig { private final List pluginFactories; private DurableConfig(Builder builder) { + if (builder.executorService != null && builder.invocationExecutorFactory != null) { + throw new IllegalArgumentException( + "ExecutorService and invocation executor factory cannot both be configured"); + } this.pluginFactories = builder.loadDynamicPlugins ? DynamicPluginLoader.loadConfiguredPluginFactories(builder.pluginFactories) : List.copyOf(builder.pluginFactories); @@ -112,6 +119,7 @@ private DurableConfig(Builder builder) { this.serDes = Objects.requireNonNullElseGet(builder.serDes, JacksonSerDes::new); this.executorService = Objects.requireNonNullElseGet(builder.executorService, DurableConfig::createDefaultExecutor); + this.invocationExecutorFactory = builder.invocationExecutorFactory; this.loggerConfig = Objects.requireNonNullElseGet(builder.loggerConfig, LoggerConfig::defaults); this.pollingStrategy = Objects.requireNonNullElse(builder.pollingStrategy, PollingStrategies.Presets.DEFAULT); this.checkpointDelay = Objects.requireNonNullElseGet(builder.checkpointDelay, () -> Duration.ofSeconds(0)); @@ -166,6 +174,19 @@ public ExecutorService getExecutorService() { return executorService; } + /** + * Gets the optional factory for invocation-owned executors. + * + *

When present, {@code ExecutionManager} calls the factory once per Lambda invocation and owns the returned + * executor's shutdown. {@link #getExecutorService()} remains the shared executor used when no factory is + * configured. + * + * @return configured invocation executor factory, if any + */ + public Optional> getInvocationExecutorFactory() { + return Optional.ofNullable(invocationExecutorFactory); + } + /** * Gets the configured LoggerConfig. * @@ -239,12 +260,16 @@ public Builder toBuilder() { var builder = new Builder() .withDurableExecutionClient(durableExecutionClient) .withSerDes(serDes) - .withExecutorService(executorService) .withLoggerConfig(loggerConfig) .withPollingStrategy(pollingStrategy) .withCheckpointDelay(checkpointDelay) .withDeserializeAfterSerialization(deserializeAfterSerialization) .withCheckpointEmptyMap(checkpointEmptyMap); + if (invocationExecutorFactory != null) { + builder.withInvocationExecutorFactory(invocationExecutorFactory); + } else { + builder.withExecutorService(executorService); + } builder.pluginFactories = new ArrayList<>(pluginFactories); builder.loadDynamicPlugins = false; return builder; @@ -341,6 +366,7 @@ public static final class Builder { private DurableExecutionClient durableExecutionClient; private SerDes serDes; private ExecutorService executorService; + private Supplier invocationExecutorFactory; private LoggerConfig loggerConfig; private PollingStrategy pollingStrategy; private Duration checkpointDelay; @@ -422,6 +448,26 @@ public Builder withExecutorService(ExecutorService executorService) { return this; } + /** + * Sets a factory that creates an isolated executor for each Lambda invocation. + * + *

The factory is called once by each {@code DurableExecutor.execute()} call and may be called concurrently + * on Lambda Managed Instances. It must return a fresh, non-null {@link ExecutorService}; the SDK shuts that + * executor down only after actual admitted work and End finish, then observes termination within the remaining + * cleanup budget. Running code that ignores interruption may delay shutdown beyond that budget. Do not return + * an executor shared with another invocation or application component. This option cannot be combined with + * {@link #withExecutorService(ExecutorService)}. + * + * @param invocationExecutorFactory factory for invocation-owned executors + * @return This builder + * @throws NullPointerException if invocationExecutorFactory is null + */ + public Builder withInvocationExecutorFactory(Supplier invocationExecutorFactory) { + this.invocationExecutorFactory = + Objects.requireNonNull(invocationExecutorFactory, "Invocation executor factory cannot be null"); + return this; + } + /** * Sets a custom LoggerConfig. If not set, defaults to suppressing replay logs. * diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcher.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcher.java index 2046e82e03..6a0fbf1985 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcher.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcher.java @@ -11,7 +11,9 @@ import java.util.concurrent.CompletionException; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicReference; import java.util.function.Consumer; import java.util.function.Function; @@ -28,7 +30,7 @@ * @param Request type */ public class ApiRequestDelayedBatcher { - private static final Duration MAX_DELAY = Duration.ofMinutes(60); + private static final long NO_FLUSH_DEADLINE = Long.MAX_VALUE; /** Maximum items allowed in a single batch */ private final int maxItemCount; @@ -134,6 +136,12 @@ void shutdown() { if (fatal != null) throw new CompletionException(fatal); } + /** Flushes pending requests without waiting beyond the caller's remaining cleanup budget. */ + void shutdown(Duration timeout) throws InterruptedException, ExecutionException, TimeoutException { + flush(); + flushingQueueFuture.get(Math.max(0, timeout.toNanos()), TimeUnit.NANOSECONDS); + } + /** Schedules pending requests immediately without waiting for the current batch. */ void flush() { synchronized (delayedBatch) { @@ -145,8 +153,8 @@ void flush() { /** clear the current batch and creates a new batch */ private void initializeDelayedBatch() { this.delayedBatch.clear(); - // MAX_DELAY is longer than a single Lambda invocation - this.delayedBatchFlushTime = System.nanoTime() + MAX_DELAY.toNanos(); + // No timer is scheduled until an item supplies its flush time. + this.delayedBatchFlushTime = NO_FLUSH_DEADLINE; // the timer future is created initially without a timeout until an item is added to the batch this.delayedBatchFlushTimer = new CompletableFuture<>(); diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/CheckpointManager.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/CheckpointManager.java index e3a0dd0b30..fb82a20c6e 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/CheckpointManager.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/CheckpointManager.java @@ -11,6 +11,8 @@ import java.util.Objects; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeoutException; import java.util.function.BooleanSupplier; import java.util.function.Consumer; import java.util.function.Supplier; @@ -19,9 +21,11 @@ import software.amazon.awssdk.awscore.exception.AwsServiceException; import software.amazon.awssdk.services.lambda.model.CheckpointUpdatedExecutionState; import software.amazon.awssdk.services.lambda.model.Operation; +import software.amazon.awssdk.services.lambda.model.OperationAction; import software.amazon.awssdk.services.lambda.model.OperationType; import software.amazon.awssdk.services.lambda.model.OperationUpdate; import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; import software.amazon.lambda.durable.retry.PollingStrategies; import software.amazon.lambda.durable.retry.PollingStrategy; import software.amazon.lambda.durable.util.DurableApiErrorClassifier; @@ -48,7 +52,9 @@ class CheckpointManager { private final BooleanSupplier tryStartCheckpointProcessing; private final Runnable onCheckpointProcessingFinish; private final Runnable onTokenRevoked; + private final Runnable requestAdmission; private final Object checkpointLock = new Object(); + private volatile boolean terminalExecutionAcknowledged; private volatile String checkpointToken; private boolean pollingStopped; // Guarded by pollingFutures. /** @@ -130,6 +136,50 @@ class CheckpointManager { Runnable onCheckpointProcessingFinish, Runnable onTokenRevoked, Supplier pluginFatal) { + this( + config, + durableExecutionArn, + checkpointToken, + callback, + tryStartCheckpointProcessing, + onCheckpointProcessingFinish, + onTokenRevoked, + pluginFatal, + () -> {}); + } + + CheckpointManager( + DurableConfig config, + String durableExecutionArn, + String checkpointToken, + Consumer> callback, + BooleanSupplier tryStartCheckpointProcessing, + Runnable onCheckpointProcessingFinish, + Runnable onTokenRevoked, + Runnable requestAdmission) { + this( + config, + durableExecutionArn, + checkpointToken, + callback, + tryStartCheckpointProcessing, + onCheckpointProcessingFinish, + onTokenRevoked, + () -> null, + requestAdmission); + } + + CheckpointManager( + DurableConfig config, + String durableExecutionArn, + String checkpointToken, + Consumer> callback, + BooleanSupplier tryStartCheckpointProcessing, + Runnable onCheckpointProcessingFinish, + Runnable onTokenRevoked, + Supplier pluginFatal, + Runnable requestAdmission) { + this.requestAdmission = requestAdmission; this.config = config; this.durableExecutionArn = durableExecutionArn; this.callback = callback; @@ -256,6 +306,15 @@ void shutdown() { checkpointApiRequestDelayedBatcher.shutdown(); } + void shutdown(Duration timeout) throws InterruptedException, ExecutionException, TimeoutException { + stopPolling(new IllegalStateException("CheckpointManager shutdown")); + checkpointApiRequestDelayedBatcher.shutdown(timeout); + } + + boolean isTerminalExecutionAcknowledged() { + return terminalExecutionAcknowledged; + } + private void stopPolling(Throwable cause) { List>> allFutures; synchronized (pollingFutures) { @@ -274,6 +333,11 @@ private void stopPolling(Throwable cause) { * nextMarker) */ List fetchAllPages(CheckpointUpdatedExecutionState checkpointUpdatedExecutionState) { + return fetchAllPages(checkpointUpdatedExecutionState, false); + } + + private List fetchAllPages( + CheckpointUpdatedExecutionState checkpointUpdatedExecutionState, boolean terminal) { rethrowFatalIfAborted(); List operations = new ArrayList<>(); if (checkpointUpdatedExecutionState == null) { @@ -285,6 +349,12 @@ List fetchAllPages(CheckpointUpdatedExecutionState checkpointUpdatedE var nextMarker = checkpointUpdatedExecutionState.nextMarker(); while (nextMarker != null && !nextMarker.isEmpty()) { rethrowFatalIfAborted(); + try { + requestAdmission.run(); + } catch (UnrecoverableDurableExecutionException failure) { + if (!terminal || !ExecutionManager.isLifecycleFailure(failure)) throw failure; + break; // Preserve inline/already-read metadata without issuing a new request after terminal ACK. + } var startTime = System.nanoTime(); try { var response = config.getDurableExecutionClient() @@ -312,6 +382,7 @@ private void checkpointBatch(List updates) { return entry.getValue().isEmpty(); }); } + // filter the null values from pollers var request = updates.stream().filter(Objects::nonNull).toList(); @@ -353,8 +424,15 @@ private void checkpointRequest(List request) { var startTime = System.nanoTime(); logger.debug("Calling durable checkpoint API with {} updates: {}", request.size(), request); rethrowFatalIfAborted(); + requestAdmission.run(); var response = config.getDurableExecutionClient().checkpoint(durableExecutionArn, checkpointToken, request); + if (request.stream() + .anyMatch(update -> + update.type() == OperationType.EXECUTION && update.action() == OperationAction.SUCCEED)) + terminalExecutionAcknowledged = true; rethrowFatalIfAborted(); + // A late response is durable at the service, but must not resume old SDK work after invocation cleanup. + if (!terminalExecutionAcknowledged) requestAdmission.run(); logger.debug("Durable checkpoint API called (latency={}ns): {}.", System.nanoTime() - startTime, response); rethrowFatalIfAborted(); @@ -366,7 +444,7 @@ private void checkpointRequest(List request) { var state = response.newExecutionState(); if (state != null) { - processUpdatedOperations(fetchAllPages(state)); + processUpdatedOperations(fetchAllPages(state, terminalExecutionAcknowledged)); } } diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java index 64fe95c5a2..2ceecec9c1 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java @@ -77,17 +77,66 @@ public static DurableExecutionOutput execute( BiFunction handler, DurableConfig config) { try (var manager = new ExecutionManager(input, config, lambdaContext)) { - manager.registerActiveThread(ROOT_THREAD_ID); - var invocation = new Invocation<>(input, lambdaContext, inputType, handler, config, manager); try { - return invocation.execute().join(); - } catch (CompletionException failure) { - ExceptionHelper.sneakyThrow(ExceptionHelper.unwrapCompletableFuture(failure)); + manager.registerActiveThread(ROOT_THREAD_ID); + var invocation = new Invocation<>(input, lambdaContext, inputType, handler, config, manager); + var completion = invocation.execute(); + var ready = invocation.outcome.applyToEither( + completion.handle((ignored, failure) -> new Outcome(null, failure)), Function.identity()); + var selected = manager.awaitInvocationOutcome(ready); + var originalFailure = selected.failure() != null + ? selected.failure() + : ExceptionHelper.unwrapInvocationFailure( + selected.result().failure()); + var finalFailure = manager.finishInvocation(originalFailure, completion); + var completedFailure = completion.isDone() + ? completion + .handle((ignored, failure) -> ExceptionHelper.unwrapInvocationFailure(failure)) + .join() + : null; + if (ExecutionManager.isFatal(completedFailure)) ExceptionHelper.sneakyThrow(completedFailure); + if (ExecutionManager.isFatal(finalFailure)) + ExceptionHelper.sneakyThrow(retainCompletionDiagnostic(finalFailure, completedFailure)); + if (ExecutionManager.isLifecycleFailure(finalFailure) && manager.isTerminalExecutionAcknowledged()) { + // ACK outranks a lifecycle cutoff, not an unrelated failure already reported by delivery. + if (completedFailure != null && !ExecutionManager.isLifecycleFailure(completedFailure)) + ExceptionHelper.sneakyThrow(completedFailure); + return DurableExecutionOutput.success(""); + } + if (finalFailure != null && (selected.failure() != null || finalFailure != originalFailure)) + ExceptionHelper.sneakyThrow(retainCompletionDiagnostic(finalFailure, completedFailure)); + // Preserve the stored failure after control precedence; newer JDKs wrap join() cancellation. + if (completedFailure != null) ExceptionHelper.sneakyThrow(completedFailure); + return completion.join(); + } catch (Throwable failure) { + var originalFailure = ExecutionManager.normalizeInvocationBoundaryFailure(failure); + // Finalize synchronous setup/admission/dispatch failures before automatic close observes them again. + var finalFailure = manager.finishInvocation(originalFailure); + ExceptionHelper.sneakyThrow(finalFailure != null ? finalFailure : originalFailure); return null; + } finally { + manager.markInvocationReported(); } } } + private static Throwable retainCompletionDiagnostic(Throwable primary, Throwable completion) { + if (completion == null || completion == primary) return primary; + // Re-observing the same delivery error as an SDK cleanup cause does not add a new diagnostic. + if (hasSdkLifecycleCause(primary, completion) || hasSdkLifecycleCause(completion, primary)) return primary; + for (var diagnostic : primary.getSuppressed()) { + if (diagnostic == completion || hasSdkLifecycleCause(diagnostic, completion)) return primary; + } + primary.addSuppressed(completion); + return primary; + } + + private static boolean hasSdkLifecycleCause(Throwable failure, Throwable cause) { + return failure.getClass() == UnrecoverableDurableExecutionException.class + && ExecutionManager.isLifecycleFailure(failure) + && failure.getCause() == cause; + } + /** Invocation-local state, accessed on the handler thread when plugins are present. */ private static final class Invocation { private final Context lambdaContext; @@ -102,6 +151,8 @@ private static final class Invocation { private final String xRayTraceId; private I userInput; private boolean started; + private boolean ended; + private CompletableFuture> outcome; private final AtomicReference losingBodyFatal = new AtomicReference<>(); private Invocation( @@ -128,9 +179,10 @@ private CompletableFuture execute() { var body = new CompletableFuture(); // Select the invocation outcome before scheduling, including with an inline executor. A suspension or // termination can win while the handler is still unwinding; its finally block must not replace it. - var outcome = manager.runUntilCompleteOrSuspend(body).handle(Outcome::new); + outcome = manager.runUntilCompleteOrSuspend(body).handle(Outcome::new); Supplier task = () -> { var bodyOutcome = Outcome.capture(this::invokeHandler); + manager.markRootHandlerExited(); bodyOutcome.complete(body); var selected = outcome.join(); var selectedCause = ExceptionHelper.unwrapInvocationFailure(selected.failure()); @@ -152,7 +204,13 @@ private CompletableFuture execute() { // No terminal response exists yet, so the service must retry this invocation. Plugins still // need their end hook. Classifying this as a handler failure would mark the execution FAILED, // so it is reported as a delivery failure: RETRYING, then the original exception is rethrown. - return failDelivery(drainFailure); + var selectedFailure = ExceptionHelper.unwrapInvocationFailure(selected.failure()); + if (ExecutionManager.isFatal(selectedFailure)) { + var diagnostic = ExecutionManager.ownerDrainDiagnostic(drainFailure); + if (selectedFailure != diagnostic) selectedFailure.addSuppressed(diagnostic); + return failDelivery(selectedFailure); + } + return failDelivery(ExecutionManager.selectOwnerDrainFailure(drainFailure)); } catch (Error fatal) { return failDelivery(fatal); } @@ -172,19 +230,39 @@ private O invokeHandler() { // Deserialize only once. Even failed input gets paired start/end hooks with a null input value. inputFailure = failure; } + checkInputAdmission(inputFailure); fireOnInvocationStart(); + checkInputAdmission(inputFailure); if (inputFailure != null) ExceptionHelper.sneakyThrow(inputFailure); var context = DurableContextImpl.createRootContext(manager, config, lambdaContext); DurableContextImpl.setCurrentContext(context); try (var ignored = DurableLogger.attachContext()) { return plugins.runHandler( - () -> handler.apply(userInput, context), + () -> { + manager.checkUserFunctionExecutionAllowed(); + return handler.apply(userInput, context); + }, () -> {}, manager::recordHandlerScopeFatal, manager::recordRootHandlerFatal); } } + private void checkInputAdmission(Throwable inputFailure) { + try { + manager.checkUserFunctionExecutionAllowed(); + } catch (Throwable admissionFailure) { + // Keep existing input Start/End behavior while admission remains open. If it closes during input + // preparation, the captured JVM fatal must win without starting any additional plugin or user work. + var inputCause = ExceptionHelper.unwrapInvocationFailure(inputFailure); + if (isFatal(inputCause)) { + manager.recordInvocationFatal((Error) inputCause); + throw manager.getInvocationFatal(); + } + ExceptionHelper.sneakyThrow(admissionFailure); + } + } + private void fireOnInvocationStart() { if (plugins.isEmpty()) return; var info = new InvocationInfo( @@ -205,6 +283,12 @@ private void fireOnInvocationStart() { private DurableExecutionOutput finishInvocation(O value, Throwable failure) { var fatal = manager.getInvocationFatal(); if (fatal != null) return finishFailure(fatal); + if (manager.isInvocationReported()) + return failDelivery( + manager.getLifecycleFailure() != null + ? manager.getLifecycleFailure() + : new IllegalStateException("Invocation has already returned")); + if (manager.getLifecycleFailure() != null) return failDelivery(manager.getLifecycleFailure()); // A withdrawn token means this invocation must stop and report PENDING. That overrides a handler success, // a handler failure, and a retryable failure, so it is checked before any of them. if (manager.isCheckpointTokenRevoked()) @@ -217,7 +301,7 @@ private DurableExecutionOutput finishInvocation(O value, Throwable failure) { } catch (Throwable deliveryFailure) { // Checkpointing a large result fails with SuspendExecutionException when the token is already gone. // That is a suspension, not a delivery error, so it reports PENDING rather than RETRYING. - if (ExceptionHelper.unwrapCompletableFuture(deliveryFailure) instanceof SuspendExecutionException) { + if (ExceptionHelper.unwrapInvocationFailure(deliveryFailure) instanceof SuspendExecutionException) { return finishFailure(new SuspendExecutionException(REVOKED_CHECKPOINT_TOKEN_MESSAGE)); } return failDelivery(deliveryFailure); @@ -273,7 +357,21 @@ private DurableExecutionOutput failDelivery(Throwable deliveryFailure) { } private void fireOnInvocationEnd(InvocationStatus status, Throwable error, Object result) { - if (!started) return; + if (!started || ended) return; + ended = true; + // End stays on its actual owner, including after a cutoff requested cancellation. The defensive + // reported-state guard must never reopen work or prepare another execution result. + var lifecycleFailure = manager.getLifecycleFailure(); + var terminal = status == InvocationStatus.SUCCEEDED || status == InvocationStatus.FAILED; + if ((manager.isInvocationReported() || (terminal && lifecycleFailure != null)) + && !manager.isTerminalExecutionAcknowledged() + && !ExecutionManager.isFatal(error)) { + status = InvocationStatus.RETRYING; + error = lifecycleFailure != null + ? lifecycleFailure + : new IllegalStateException("Invocation already ended before owner cleanup"); + result = null; + } plugins.onInvocationEnd(new InvocationEndInfo( requestId, executionArn, @@ -546,14 +644,13 @@ private static String handleLargePayload(ExecutionManager executionManager, Stri LAMBDA_RESPONSE_SIZE_LIMIT); // Checkpoint the large result and wait for it to complete - executionManager - .sendExecutionResultUpdate(OperationUpdate.builder() + executionManager.awaitCheckpointCompletion( + executionManager.sendExecutionResultUpdate(OperationUpdate.builder() .type(OperationType.EXECUTION) .id(executionManager.getExecutionOperation().id()) .action(OperationAction.SUCCEED) .payload(outputPayload) - .build()) - .join(); + .build())); // Return empty result, we checkpointed the data manually logger.debug("Execution completed (large response checkpointed)"); diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java index 06502b772a..bb6f5a9530 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java @@ -11,6 +11,7 @@ import java.util.Comparator; import java.util.HashMap; import java.util.HashSet; +import java.util.IdentityHashMap; import java.util.List; import java.util.Map; import java.util.Objects; @@ -18,6 +19,7 @@ import java.util.Set; import java.util.concurrent.CancellationException; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CompletionException; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutionException; import java.util.concurrent.Executor; @@ -26,6 +28,7 @@ import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; @@ -76,6 +79,24 @@ public class ExecutionManager implements SafeCloseable { private static final Logger logger = LoggerFactory.getLogger(ExecutionManager.class); private static final AtomicLong LOCAL_INVOCATION_SEQUENCE = new AtomicLong(); + private static final Set ACTIVE_INVOCATION_EXECUTORS = + Collections.synchronizedSet(Collections.newSetFromMap(new IdentityHashMap<>())); + + // Stop waiting for the root early enough to cancel tasks and flush checkpoints before the platform deadline. + private static final Duration EXECUTION_DEADLINE_HEADROOM = Duration.ofMillis(500); + // Leave a final margin for response serialization and return through the Lambda runtime wrapper. + private static final Duration RESPONSE_HEADROOM = Duration.ofMillis(100); + // Lambda supplies a deadline. This fallback limits the initial cancellation phase for local/null contexts; + // actual owner and transport exit are still awaited without a hard return bound. + private static final Duration MAX_CLEANUP_DURATION = Duration.ofSeconds(30); + private static final Duration MAX_GRACEFUL_DRAIN_DURATION = Duration.ofSeconds(5); + private static final Duration MAX_CANCELLATION_DRAIN_DURATION = Duration.ofSeconds(1); + private static final Duration EXECUTOR_SHUTDOWN_RESERVE = Duration.ofMillis(100); + private static final Duration CHECKPOINT_CLEANUP_RESERVE = Duration.ofMillis(100); + private static final String LIFECYCLE_ERROR_TYPE = + "software.amazon.lambda.durable.execution.InvocationLifecycleException"; + + record InvocationOutcome(T result, Throwable failure) {} enum LifecycleState { /** The invocation accepts new durable operations, executor tasks, checkpoints, and polls. */ @@ -100,6 +121,8 @@ enum LifecycleState { private final Context lambdaContext; private final AtomicReference executionMode; private final DurableConfig durableConfig; + private final ExecutorService executorService; + private final boolean ownsExecutorService; private final Set updatedOperationIdsSinceLastInvocation; private final Set initialOperationIds; @@ -119,7 +142,13 @@ enum LifecycleState { private final Map> activeExecutorTasks = new ConcurrentHashMap<>(); // The current task is shared across managers on a thread; admission verifies the exact handle's owner. private static final ThreadLocal> currentExecutorTask = new ThreadLocal<>(); - private LifecycleState lifecycleState = LifecycleState.OPEN; + private volatile LifecycleState lifecycleState = LifecycleState.OPEN; + private volatile boolean rootHandlerExited; + private final CompletableFuture rootHandlerExitSignal = new CompletableFuture<>(); + private volatile boolean invocationReported; + private boolean cleanupFinished; + private final AtomicReference lifecycleFailure = new AtomicReference<>(); + private final ThreadLocal cleanupInterrupted = ThreadLocal.withInitial(() -> false); // Guarded by admissionLock. Counts enqueue calls, not the lifetime of their returned futures. private int checkpointAdmissionsInFlight; @@ -183,7 +212,8 @@ public ExecutionManager(DurableExecutionInput input, DurableConfig config, Conte this::tryStartCheckpointProcessing, this::finishCheckpointProcessing, this::signalSuspension, - pluginFatal::get); + pluginFatal::get, + this::checkCheckpointRequestAllowed); this.operationStorage = checkpointManager.fetchAllPages(input.initialExecutionState()).stream() .collect(Collectors.toConcurrentMap(Operation::id, op -> op)); @@ -206,10 +236,80 @@ public ExecutionManager(DurableExecutionInput input, DurableConfig config, Conte if (executionOp == null) { throw new IllegalStateException("EXECUTION operation not found"); } - logger.debug("DurableExecution.execute() called"); - logger.debug("DurableExecutionArn: {}", durableExecutionArn); - logger.debug("Initial operations count: {}", operationStorage.size()); - logger.debug("EXECUTION operation found: {}", executionOp.id()); + var invocationExecutorFactory = config.getInvocationExecutorFactory(); + if (invocationExecutorFactory.isPresent()) { + this.executorService = Objects.requireNonNull( + invocationExecutorFactory.get().get(), "Invocation executor factory returned null"); + this.ownsExecutorService = true; + } else { + this.executorService = config.getExecutorService(); + this.ownsExecutorService = false; + } + var leaseAcquired = ownsExecutorService && ACTIVE_INVOCATION_EXECUTORS.add(executorService); + if (ownsExecutorService && !leaseAcquired) { + throw new IllegalStateException("Invocation executor factory returned an executor already in use"); + } + try { + if (ownsExecutorService && executorService.isShutdown()) { + throw new IllegalStateException("Invocation executor factory returned a shutdown executor"); + } + logger.debug("DurableExecution.execute() called"); + logger.debug("DurableExecutionArn: {}", durableExecutionArn); + logger.debug("Initial operations count: {}", operationStorage.size()); + logger.debug("EXECUTION operation found: {}", executionOp.id()); + } catch (Throwable initializationFailure) { + if (leaseAcquired) initializationFailure = cleanupFailedExecutorInitialization(initializationFailure); + ExceptionHelper.sneakyThrow(initializationFailure); + } + } + + private Throwable cleanupFailedExecutorInitialization(Throwable initializationFailure) { + var initial = inspectOwnedExecutorFailure(initializationFailure); + var initialCause = normalizeOwnedExecutorFailure(initial, null); + var initialFatal = isFatal(initialCause) ? (Error) initialCause : null; + var deadline = cleanupDeadlineNanos(); + try { + Throwable cleanupFailure; + try { + // No SDK work has been dispatched. Avoid repeating the state callback that failed validation. + cleanupFailure = attemptOwnedExecutorShutdown(deadline, null, false, initialFatal); + } catch (Throwable failure) { + cleanupFailure = failure; + } + var selected = combineOwnedExecutorFailures(initializationFailure, initial, cleanupFailure); + return cleanupFailure == null ? selected : forceStopOwnedExecutor(deadline, selected); + } finally { + ACTIVE_INVOCATION_EXECUTORS.remove(executorService); + restoreCleanupInterrupt(); + } + } + + private Throwable forceStopOwnedExecutor(long deadline, Throwable priorFailure) { + var failure = priorFailure; + try { + executorService.shutdownNow(); + } catch (Throwable shutdownFailure) { + failure = combineOwnedExecutorFailures(failure, shutdownFailure); + } + try { + if (!awaitOwnedExecutorTermination(deadline)) + failure = combineOwnedExecutorFailures( + failure, lifecycleFailure("Invocation executor did not terminate during forced cleanup", null)); + } catch (InterruptedException interrupted) { + cleanupInterrupted.set(true); + failure = combineOwnedExecutorFailures( + failure, lifecycleFailure("Interrupted while forcing invocation executor shutdown", interrupted)); + } catch (Throwable terminationFailure) { + failure = combineOwnedExecutorFailures(failure, terminationFailure); + } + return failure; + } + + private boolean awaitOwnedExecutorTermination(long deadline) throws InterruptedException { + var terminated = executorService.awaitTermination(remainingNanos(deadline), TimeUnit.NANOSECONDS); + // A later state-query failure must not retain an executor whose termination was already confirmed. + if (terminated) ACTIVE_INVOCATION_EXECUTORS.remove(executorService); + return terminated; } private static String resolveInvocationId(Context lambdaContext, String durableExecutionArn) { @@ -332,6 +432,14 @@ public void registerOperation(BaseDurableOperation operation, BaseDurableOperati operationCompletionLocks.put(operationId, completionLock); registeredOperations.put(operationId, operation); } + // Cancellation can publish outside admissionLock and miss an operation inserted after its broadcast. + // Recheck after publication and release this operation's waiters before rejecting its constructor. + try { + checkInvocationAdmission(); + } catch (RuntimeException | Error failure) { + operation.getCompletionFuture().completeExceptionally(failure); + throw failure; + } // Revocation is published before existing operations are suspended. Check after insertion so an // operation registered too late for that notification still releases its waiters. if (isCheckpointTokenRevoked()) { @@ -341,7 +449,7 @@ public void registerOperation(BaseDurableOperation operation, BaseDurableOperati /** Submits the invocation's root handler and records its executor task separately from its logical result. */ CompletableFuture submitRootTask(Supplier action) { - return submitExecutorTask(ExecutorTaskHandle.Role.ROOT, null, durableConfig.getExecutorService(), action); + return submitExecutorTask(ExecutorTaskHandle.Role.ROOT, null, executorService, action); } /** Submits an operation handler and records its executor task separately from its logical result. */ @@ -356,7 +464,7 @@ public void executeOperationTask( ExecutorTaskHandle task = registerExecutorTask( operationRole(operation), operation, - durableConfig.getExecutorService(), + executorService, () -> { started.set(true); command.run(); @@ -377,7 +485,7 @@ private CompletableFuture submitOperationTask( return submitExecutorTask( operationRole(operation), operation, - durableConfig.getExecutorService(), + executorService, () -> { action.run(); return null; @@ -407,7 +515,7 @@ void executeRootTask(Runnable command) { submitExecutorTask( ExecutorTaskHandle.Role.ROOT, null, - durableConfig.getExecutorService(), + executorService, () -> { command.run(); return null; @@ -441,6 +549,9 @@ private ExecutorTaskHandle registerExecutorTask( } try { + // The lifecycle fence can race task insertion after the check under admissionLock. The catch below + // settles actual-exit tracking if cancellation's task snapshot did not contain this registration. + checkInvocationAdmission(); dispatchExecutorTask(task, action, executor, preserveWorkerFailure); return task; } catch (RuntimeException | Error failure) { @@ -1035,15 +1146,124 @@ public CompletableFuture pollForOperationUpdates(String operationId, /** Reserved result delivery after operation admission has closed; the caller observes this checkpoint. */ CompletableFuture sendExecutionResultUpdate(OperationUpdate update) { + checkCheckpointRequestAllowed(); return checkpointManager.checkpoint(update); } + void awaitCheckpointCompletion(CompletableFuture future) { + var failure = observeCheckpointUntilDeadline(future); + // A cutoff may settle the logical observation early. Main's End contract still requires admitted + // transport and its publication callbacks to finish before the owner finalizes the invocation. + try { + checkpointManager.shutdown(); + } catch (Throwable transportFailure) { + failure = combineFailures(failure, ExceptionHelper.unwrapInvocationFailure(transportFailure)); + } + var completedFailure = future.isDone() + ? future.handle((ignored, problem) -> ExceptionHelper.unwrapInvocationFailure(problem)) + .join() + : null; + if (isFatal(completedFailure)) ExceptionHelper.sneakyThrow(completedFailure); + if (isFatal(failure)) ExceptionHelper.sneakyThrow(failure); + if (completedFailure != null && !isLifecycleFailure(completedFailure)) + ExceptionHelper.sneakyThrow(completedFailure); + if (checkpointManager.isTerminalExecutionAcknowledged()) return; + if (failure != null) ExceptionHelper.sneakyThrow(failure); + future.join(); + } + + private Throwable observeCheckpointUntilDeadline(CompletableFuture future) { + var deadline = + deadlineNanos == null ? addToNow(MAX_CLEANUP_DURATION) : deadlineNanos - RESPONSE_HEADROOM.toNanos(); + try { + future.get(remainingNanos(deadline), TimeUnit.NANOSECONDS); + return null; + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + return lifecycleFailure("Interrupted while waiting for checkpoint completion", interrupted); + } catch (TimeoutException timeout) { + return lifecycleFailure("Checkpoint completion exceeded the invocation deadline", timeout); + } catch (CancellationException cancelled) { + // Newer JDKs wrap get() cancellation. Preserve the future's actual failure and still drain transport. + return future.handle((ignored, failure) -> ExceptionHelper.unwrapInvocationFailure(failure)) + .join(); + } catch (ExecutionException failure) { + return ExceptionHelper.unwrapInvocationFailure(failure.getCause()); + } + } + + void markRootHandlerExited() { + rootHandlerExited = true; + rootHandlerExitSignal.complete(null); + } + + void markInvocationReported() { + invocationReported = true; + } + + boolean isInvocationReported() { + return invocationReported; + } + + UnrecoverableDurableExecutionException getLifecycleFailure() { + return lifecycleFailure.get(); + } + + boolean isTerminalExecutionAcknowledged() { + return checkpointManager.isTerminalExecutionAcknowledged(); + } + + /** Fences actual user callbacks, including callbacks delayed inside a plugin start hook. */ + public void checkUserFunctionExecutionAllowed() { + rethrowPluginFatalIfPresent(); + if (executionExceptionFuture.isCompletedExceptionally()) { + try { + executionExceptionFuture.join(); + } catch (CompletionException stopped) { + ExceptionHelper.sneakyThrow(ExceptionHelper.unwrapInvocationFailure(stopped)); + } + } + checkInvocationAdmission(); + } + + private void checkInvocationAdmission() { + rethrowPluginFatalIfPresent(); + if (invocationReported || lifecycleFailure.get() != null) { + var failure = lifecycleFailure.get(); + throw failure != null ? failure : lifecycleFailure("Invocation has already returned", null); + } + if (deadlineNanos != null && remainingNanos(deadlineNanos - EXECUTION_DEADLINE_HEADROOM.toNanos()) == 0) { + var failure = lifecycleFailure("Invocation deadline reached before user work", new TimeoutException()); + // An existing suspension/termination already owns the outcome; still reject this later admission. + if (executionExceptionFuture.isCompletedExceptionally()) throw failure; + // This check may hold admissionLock. Latch without running cancellation/completion observers here, + // so catching the rejection cannot turn the invocation into a successful fallback. + lifecycleFailure.compareAndSet(null, failure); + throw lifecycleFailure.get(); + } + } + + private void checkCheckpointRequestAllowed() { + rethrowPluginFatalIfPresent(); + if (invocationReported || lifecycleFailure.get() != null) { + var failure = lifecycleFailure.get(); + throw failure != null ? failure : lifecycleFailure("Invocation has already returned", null); + } + if (deadlineNanos != null && remainingNanos(deadlineNanos - RESPONSE_HEADROOM.toNanos()) == 0) { + var failure = lifecycleFailure("Invocation deadline reached before checkpoint I/O", new TimeoutException()); + if (executionExceptionFuture.isCompletedExceptionally()) throw failure; + lifecycleFailure.compareAndSet(null, failure); + throw lifecycleFailure.get(); + } + } + // ===== Utilities ===== /** Shutdown the checkpoint batcher. */ @Override public void close() { try { - drainOperations(); + var failure = finishInvocation(null, true); + if (failure != null) ExceptionHelper.sneakyThrow(failure); } finally { pluginRunner.releasePlugins(); } @@ -1105,7 +1325,7 @@ private void awaitUserHandler(BaseDurableOperation operation, DrainFailure drain drain.interrupted( "Interrupted while waiting for user handler " + operation.getOperationId(), interrupted); } catch (CancellationException cancelled) { - drain.record(new IllegalStateException( + drain.record(ownerDrainFailure( "User handler observation cancelled: " + operation.getOperationId(), cancelled)); return; } catch (Exception completedExceptionally) { @@ -1114,11 +1334,52 @@ private void awaitUserHandler(BaseDurableOperation operation, DrainFailure drain } } + /** Tags only synthetic owner-drain failures created after an invocation cancellation was recorded. */ + private RuntimeException ownerDrainFailure(String message, Throwable cause) { + var diagnostic = new IllegalStateException(message, cause); + var lifecycleCause = lifecycleFailure.get(); + return lifecycleCause == null ? diagnostic : new OwnerDrainFailure(diagnostic, lifecycleCause); + } + + private static final class OwnerDrainFailure extends RuntimeException { + private final IllegalStateException diagnostic; + private final UnrecoverableDurableExecutionException lifecycleCause; + + private OwnerDrainFailure( + IllegalStateException diagnostic, UnrecoverableDurableExecutionException lifecycleCause) { + super(diagnostic.getMessage(), diagnostic, false, false); + this.diagnostic = diagnostic; + this.lifecycleCause = lifecycleCause; + } + } + + static Throwable ownerDrainDiagnostic(Throwable failure) { + return failure instanceof OwnerDrainFailure owner ? owner.diagnostic : failure; + } + + static Throwable selectOwnerDrainFailure(Throwable failure) { + if (!(failure instanceof OwnerDrainFailure owner)) return failure; + for (var existing : owner.lifecycleCause.getSuppressed()) { + if (existing == owner.diagnostic) return owner.lifecycleCause; + } + owner.lifecycleCause.addSuppressed(owner.diagnostic); + return owner.lifecycleCause; + } + /** Retains cleanup failures while every required stage gets its finalization opportunity. */ - private static final class DrainFailure { + private final class DrainFailure { private Throwable failure; private boolean fatal; private boolean interrupted; + private final boolean lifecycleInterruptions; + + private DrainFailure() { + this(false); + } + + private DrainFailure(boolean lifecycleInterruptions) { + this.lifecycleInterruptions = lifecycleInterruptions; + } void run(Runnable stage) { try { @@ -1129,7 +1390,11 @@ void run(Runnable stage) { } void interrupted(String message, InterruptedException problem) { - if (!interrupted) record(new IllegalStateException(message, problem)); + if (!interrupted) + record( + lifecycleInterruptions + ? lifecycleFailure(message, problem) + : ownerDrainFailure(message, problem)); interrupted = true; } @@ -1139,17 +1404,24 @@ void preserveInterrupt() { @SuppressWarnings("removal") void record(Throwable problem) { - var cause = ExceptionHelper.unwrapInvocationFailure(problem); + var nextDiagnostic = ownerDrainDiagnostic(problem); + // Inspect wrappers once for fatal priority and aliases; retain the original diagnostic object below. + var cause = ExceptionHelper.unwrapAsyncFailure(nextDiagnostic); boolean nextFatal = cause instanceof VirtualMachineError || cause instanceof ThreadDeath; // Keep a transport wrapper intact: close may run while the same fatal already escapes the body. // Throwing that fatal object again from try-with-resources would trigger self-suppression. var next = problem; + var previousDiagnostic = ownerDrainDiagnostic(failure); if (failure == null || nextFatal && !fatal) { - if (failure != null && failure != next) next.addSuppressed(failure); + if (previousDiagnostic != null && previousDiagnostic != nextDiagnostic) + nextDiagnostic.addSuppressed(previousDiagnostic); failure = next; fatal = nextFatal; - } else if (failure != next) { - failure.addSuppressed(next); + } else if (previousDiagnostic != nextDiagnostic) { + // A shutdown re-observation of the same cancellation is not a new diagnostic. Keeping it here + // would create lifecycle -> drain -> lifecycle when the owner reports the canonical failure. + if (failure instanceof OwnerDrainFailure owner && cause == owner.lifecycleCause) return; + previousDiagnostic.addSuppressed(nextDiagnostic); } } @@ -1210,6 +1482,7 @@ private T admitCheckpoint(Supplier request, Supplier whenClosed) { } private void requireOperationAdmission() { + checkInvocationAdmission(); var task = currentExecutorTask.get(); if (lifecycleState == LifecycleState.DRAINING && task != null @@ -1224,6 +1497,7 @@ private void requireOperationAdmission() { } private void requireTaskAdmission(BaseDurableOperation operation) { + checkInvocationAdmission(); if (lifecycleState == LifecycleState.DRAINING && operation != null && registeredOperations.get(operation.getOperationId()) == operation) { @@ -1280,7 +1554,7 @@ private void awaitTaskExit(ExecutorTaskHandle task, DrainFailure drain) { private void validateExecutorPool() { // double check if the thread pool is empty - if (durableConfig.getExecutorService() instanceof ThreadPoolExecutor threadPoolExecutor) { + if (executorService instanceof ThreadPoolExecutor threadPoolExecutor) { var threadCount = threadPoolExecutor.getActiveCount(); // This may or may not be a problem because getActiveCount doesn't return an accurate number if (threadCount > 0) { @@ -1413,4 +1687,512 @@ public CompletableFuture runUntilCompleteOrSuspend(CompletableFuture u return null; }); } + /** Called only by the invocation caller, so root exit and owned-executor cleanup never self-join. */ + Throwable finishInvocation(Throwable originalFailure) { + return finishInvocation(originalFailure, false); + } + + Throwable finishInvocation(Throwable originalFailure, CompletableFuture completedRoot) { + return finishInvocation(originalFailure, false, completedRoot); + } + + private Throwable finishInvocation(Throwable originalFailure, boolean closeObservation) { + return finishInvocation(originalFailure, closeObservation, null); + } + + private Throwable finishInvocation( + Throwable originalFailure, boolean closeObservation, CompletableFuture completedRoot) { + if (cleanupFinished) return originalFailure; + beginDraining(); + var hardDeadline = cleanupDeadlineNanos(); + Throwable failure = null; + Throwable completionFatal = null; + if (Thread.interrupted()) cleanupInterrupted.set(true); + try { + if (cleanupInterrupted.get() && !getActiveExecutorTasks().isEmpty()) + throw new InterruptedException("Invocation caller was interrupted"); + stopCheckpointContinuationsUntil(hardDeadline); + failure = drainInvocationTasks(originalFailure, hardDeadline); + } catch (InterruptedException interrupted) { + cleanupInterrupted.set(true); + failure = signalLifecycleFailure( + lifecycleFailure("Interrupted while draining invocation tasks", interrupted)); + failure = settleInterruptedCleanup(hardDeadline, true, failure); + } catch (RuntimeException problem) { + failure = signalLifecycleFailure(problem); + } finally { + failure = finishActualInvocationCleanup(failure, ownsExecutorService && !closeObservation); + if (ownsExecutorService && !closeObservation && isFatal(originalFailure)) + failure = combineCompletionFailure(originalFailure, failure); + if (ownsExecutorService && !closeObservation && failure != null && failure == getInvocationFatal()) + completionFatal = failure; + if (completedRoot != null && completedRoot.isDone() && getInvocationFatal() == null) { + var rootFailure = + completedRoot.handle((ignored, problem) -> problem).join(); + // MDC restoration runs after End. Keep its fatal local so cleanup can retain later diagnostics + // without changing the selected End outcome or publishing new checkpoint/cancellation control. + if (isFatal(rootFailure)) { + completionFatal = rootFailure; + failure = combineCompletionFailure(rootFailure, failure); + } + } + try { + failure = shutdownOwnedExecutor(hardDeadline, failure); + } catch (Throwable shutdownFailure) { + // Executor callbacks, including fallback shutdown and state queries, must not skip finalization. + shutdownFailure = normalizeOwnedExecutorFailure( + inspectOwnedExecutorFailure(shutdownFailure), isFatal(failure) ? (Error) failure : null); + failure = combineFailures(failure, shutdownFailure); + var firstFatal = getInvocationFatal(); + if (firstFatal != null) failure = combineFailures(firstFatal, failure); + // Inline execution can already be throwing this fatal through try-with-resources close. + // Normal completion keeps the raw fatal ahead of lifecycle/ACK handling. + if (closeObservation && isFatal(failure)) failure = new CompletionException(failure); + } + cleanupFinished = true; + restoreCleanupInterrupt(); + } + var recordedFailure = lifecycleFailure.get(); + var selectedFailure = isFatal(originalFailure) || recordedFailure == null ? originalFailure : recordedFailure; + if (failure == null) return selectedFailure; + // Cleanup can return a later lifecycle failure without signaling it; retain the latched invocation failure. + if (isFatal(selectedFailure) || (recordedFailure != null && isLifecycleFailure(failure))) + return combineFailures(selectedFailure, failure); + return failure == completionFatal + ? combineCompletionFailure(failure, selectedFailure) + : combineFailures(failure, selectedFailure); + } + + private static Throwable combineCompletionFailure(Throwable completionFatal, Throwable additional) { + // The completed root may already carry this same delivery failure from MDC restoration handling. + for (var diagnostic : completionFatal.getSuppressed()) { + if (diagnostic == additional) return completionFatal; + } + return combineFailures(completionFatal, additional); + } + + private Throwable shutdownOwnedExecutor(long hardDeadline, Throwable priorFailure) { + if (!ownsExecutorService) return priorFailure; + Throwable result = priorFailure; + Throwable thrown = null; + try { + result = attemptOwnedExecutorShutdown( + hardDeadline, priorFailure, true, isFatal(priorFailure) ? (Error) priorFailure : null); + } catch (Throwable failure) { + thrown = isFatal(priorFailure) ? combineOwnedExecutorFailures(priorFailure, failure) : failure; + } + try { + if (executorService.isShutdown()) ACTIVE_INVOCATION_EXECUTORS.remove(executorService); + } catch (Throwable stateFailure) { + // The lease query must not replace the shutdown attempt's fatal or erase its diagnostics. + thrown = combineOwnedExecutorFailures(thrown != null ? thrown : result, stateFailure); + } + // Preserve the outer finalizer's first-fatal and explicit-close observation handling. + if (thrown != null) ExceptionHelper.sneakyThrow(thrown); + return result; + } + + private static Throwable combineOwnedExecutorFailures(Throwable earlier, Throwable later) { + return combineOwnedExecutorFailures(earlier, inspectOwnedExecutorFailure(earlier), later); + } + + private static Throwable combineOwnedExecutorFailures(Throwable earlier, OwnedFailure inspected, Throwable later) { + var earlierCause = normalizeOwnedExecutorFailure(inspected, null); + var laterCause = normalizeOwnedExecutorFailure( + inspectOwnedExecutorFailure(later), isFatal(earlierCause) ? (Error) earlierCause : null); + return combineFailures( + isFatal(earlierCause) ? earlierCause : earlier, isFatal(laterCause) ? laterCause : later); + } + + static Throwable normalizeInvocationBoundaryFailure(Throwable failure) { + var inspected = inspectOwnedExecutorFailure(failure); + var discarded = new ArrayList(); + // A retained application ExecutionException and its cause graph remain intact, including their diagnostics. + for (var wrapper : inspected.wrappers()) { + if (wrapper == inspected.selected()) break; + discarded.add(wrapper); + } + if (!discarded.isEmpty()) retainOwnedWrapperDiagnostics(inspected.selected(), inspected.selected(), discarded); + return inspected.selected(); + } + + private record OwnedFailure(Throwable selected, List wrappers) {} + + @SuppressWarnings("removal") + private static OwnedFailure inspectOwnedExecutorFailure(Throwable failure) { + // Match invocation unwrapping, retaining its observed wrappers without reading a cause accessor twice. + var wrappers = new ArrayList(); + var visited = Collections.newSetFromMap(new IdentityHashMap()); + Throwable applicationWrapper = null; + while (failure instanceof CompletionException || failure instanceof ExecutionException) { + if (!visited.add(failure)) break; + wrappers.add(failure); + if (applicationWrapper == null && failure instanceof ExecutionException) applicationWrapper = failure; + Throwable cause; + try { + cause = failure.getCause(); + } catch (VirtualMachineError | ThreadDeath fatal) { + failure = fatal; + break; + } catch (Throwable unreadableCause) { + break; + } + if (cause == null) break; + failure = cause; + } + return new OwnedFailure( + applicationWrapper != null && !isFatal(failure) ? applicationWrapper : failure, wrappers); + } + + private static Throwable normalizeOwnedExecutorFailure(OwnedFailure failure, Error preferredFatal) { + if (isFatal(failure.selected()) && !failure.wrappers().isEmpty()) { + var fatal = (Error) failure.selected(); + retainOwnedWrapperDiagnostics(preferredFatal != null ? preferredFatal : fatal, fatal, failure.wrappers()); + } + return failure.selected(); + } + + private static void retainOwnedWrapperDiagnostics(Throwable primary, Throwable cause, List wrappers) { + var omitted = Collections.newSetFromMap(new IdentityHashMap()); + omitted.addAll(wrappers); + var pending = new ArrayList(); + wrappers.forEach(wrapper -> Collections.addAll(pending, wrapper.getSuppressed())); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + for (var index = 0; index < pending.size(); index++) { + var diagnostic = pending.get(index); + if (diagnostic == primary || diagnostic == cause || omitted.contains(diagnostic) || !seen.add(diagnostic)) + continue; + var aliases = ownedAliasWrappers(diagnostic, primary, cause, omitted); + if (aliases != null) { + omitted.addAll(aliases); + aliases.forEach(alias -> Collections.addAll(pending, alias.getSuppressed())); + } else { + retainWrapperDiagnostic(primary, diagnostic); + } + } + } + + private static void retainWrapperDiagnostic(Throwable primary, Throwable diagnostic) { + // This transfers diagnostics from a discarded wrapper; it must not select a different failure. + for (var retained : primary.getSuppressed()) { + if (retained == diagnostic) return; + } + primary.addSuppressed(diagnostic); + } + + private static List ownedAliasWrappers( + Throwable failure, Throwable primary, Throwable cause, Set omitted) { + var aliases = new ArrayList(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + while (failure != primary && failure != cause && !omitted.contains(failure)) { + if (failure == null || !seen.add(failure)) return null; + var type = failure.getClass(); + if (type != CompletionException.class && type != ExecutionException.class) return null; + aliases.add(failure); + try { + failure = failure.getCause(); + } catch (Throwable unreadableCause) { + // An unreadable diagnostic link must not replace the selected fatal. + return null; + } + } + return aliases; + } + + private void requestOwnedExecutorShutdown(long deadline, boolean forceOnFailure, Error preferredFatal) { + try { + executorService.shutdown(); + } catch (Throwable shutdownFailure) { + if (preferredFatal != null) shutdownFailure = combineOwnedExecutorFailures(preferredFatal, shutdownFailure); + if (forceOnFailure) shutdownFailure = forceStopOwnedExecutor(deadline, shutdownFailure); + // The existing attempt and outer finalizer still classify the original failure and any later fatal. + ExceptionHelper.sneakyThrow(shutdownFailure); + } + } + + private Throwable attemptOwnedExecutorShutdown( + long hardDeadline, Throwable priorFailure, boolean forceOnShutdownFailure, Error preferredFatal) { + var executorDeadline = Math.max(System.nanoTime(), hardDeadline - CHECKPOINT_CLEANUP_RESERVE.toNanos()); + try { + // Constructor cleanup already recovers any failed attempt; normal finalization recovers this callback. + requestOwnedExecutorShutdown(executorDeadline, forceOnShutdownFailure, preferredFatal); + if (!getActiveExecutorTasks().isEmpty()) { + executorService.shutdownNow(); + } + if (awaitOwnedExecutorTermination(executorDeadline)) { + return priorFailure; + } + executorService.shutdownNow(); + if (awaitOwnedExecutorTermination(executorDeadline)) { + return priorFailure; + } + return combineFailures( + priorFailure, + lifecycleFailure("Invocation executor did not terminate before cleanup deadline", null)); + } catch (InterruptedException interrupted) { + cleanupInterrupted.set(true); + executorService.shutdownNow(); + return combineFailures( + priorFailure, lifecycleFailure("Interrupted while shutting down invocation executor", interrupted)); + } catch (RuntimeException failure) { + var cause = normalizeOwnedExecutorFailure(inspectOwnedExecutorFailure(failure), preferredFatal); + if (isFatal(cause)) ExceptionHelper.sneakyThrow(cause); + return combineFailures(priorFailure, lifecycleFailure("Invocation executor shutdown failed", failure)); + } + } + + private Throwable drainInvocationTasks(Throwable originalFailure, long hardDeadline) throws InterruptedException { + var taskDeadline = Math.max( + System.nanoTime(), + hardDeadline + - CHECKPOINT_CLEANUP_RESERVE.toNanos() + - (ownsExecutorService ? EXECUTOR_SHUTDOWN_RESERVE.toNanos() : 0)); + var recordedFailure = lifecycleFailure.get(); + // Fatal publication already stops operation peers. Retain the root/scope owner's cleanup allowance; + // only lifecycle cutoffs or interruption require immediate registry cancellation. + var abortFailure = recordedFailure != null + ? recordedFailure + : isLifecycleFailure(originalFailure) ? originalFailure : null; + var retrying = abortFailure != null; + if (retrying) signalLifecycleFailure(abortFailure); + else if (!awaitTasksUntil(minDeadline(taskDeadline, MAX_GRACEFUL_DRAIN_DURATION), false)) { + var failure = signalLifecycleFailure( + lifecycleFailure("Invocation user work exceeded its graceful cleanup budget", null)); + awaitTasksUntil(minDeadline(taskDeadline, MAX_CANCELLATION_DRAIN_DURATION), true); + return failure; + } + // The root body and operation work have exited. Let response preparation and End finish on their owner, + // within the remaining overall cleanup budget rather than charging an exporter to the user-work grace cap. + if (awaitTasksUntil(retrying ? minDeadline(taskDeadline, MAX_CANCELLATION_DRAIN_DURATION) : taskDeadline, true)) + return null; + return signalLifecycleFailure( + lifecycleFailure("Invocation lifecycle did not exit before the cleanup deadline", null)); + } + + private boolean awaitTasksUntil(long deadline, boolean includeRootLifecycle) throws InterruptedException { + while (true) { + List> tasks; + synchronized (admissionLock) { + tasks = getActiveExecutorTasks().stream() + .filter(task -> includeRootLifecycle + || task.role() != ExecutorTaskHandle.Role.ROOT + || !rootHandlerExited) + .toList(); + if (tasks.isEmpty()) { + // The final empty snapshot and admission closure are one transition. The internal, awaited + // EXECUTION-result path remains available; ordinary operations cannot enter after this point. + lifecycleState = LifecycleState.CLOSED; + return true; + } + } + var remaining = remainingNanos(deadline); + if (remaining == 0) return false; + try { + var exited = CompletableFuture.allOf( + tasks.stream().map(ExecutorTaskHandle::exit).toArray(CompletableFuture[]::new)); + var observed = !includeRootLifecycle + && tasks.stream().anyMatch(task -> task.role() == ExecutorTaskHandle.Role.ROOT) + ? CompletableFuture.anyOf(exited, rootHandlerExitSignal) + : exited; + observed.get(remaining, TimeUnit.NANOSECONDS); + } catch (ExecutionException impossible) { + throw new IllegalStateException("Executor task exit signal failed", impossible.getCause()); + } catch (TimeoutException timeout) { + return false; + } + } + } + + private void stopCheckpointContinuationsUntil(long deadline) throws InterruptedException { + List owners; + synchronized (activeThreads) { + closing = true; + owners = new ArrayList<>(checkpointContinuations.values()); + } + owners.forEach(ExecutionManager::stopContinuationOwner); + synchronized (activeThreads) { + while (!checkpointContinuations.isEmpty()) { + var remaining = remainingNanos(deadline); + if (remaining == 0) + throw lifecycleFailure( + "Checkpoint continuations exceeded the cleanup deadline", new TimeoutException()); + TimeUnit.NANOSECONDS.timedWait(activeThreads, remaining); + } + } + } + + private Throwable settleInterruptedCleanup(long deadline, boolean includeRoot, Throwable primary) { + while (remainingNanos(deadline) > 0) { + try { + stopCheckpointContinuationsUntil(deadline); + if (!awaitTasksUntil(deadline, includeRoot)) + return combineFailures( + primary, + lifecycleFailure( + "Interrupted cleanup exhausted its remaining budget", new TimeoutException())); + return primary; + } catch (InterruptedException interrupted) { + cleanupInterrupted.set(true); + // Keep the first interruption/failure, but continue best-effort joins with the remaining budget. + } catch (RuntimeException failure) { + return combineFailures(primary, failure); + } + } + return primary; + } + + private void restoreCleanupInterrupt() { + if (cleanupInterrupted.get()) Thread.currentThread().interrupt(); + cleanupInterrupted.remove(); + } + + private Throwable signalLifecycleFailure(Throwable failure) { + if (failure instanceof UnrecoverableDurableExecutionException problem + && LIFECYCLE_ERROR_TYPE.equals(problem.getErrorObject().errorType())) { + lifecycleFailure.compareAndSet(null, problem); + // End and the caller share the first selected lifecycle failure; keep later cleanup diagnostics. + failure = combineFailures(lifecycleFailure.get(), failure); + } + // Interruption must precede arbitrary inline completion observers. + var current = currentExecutorTask.get(); + getActiveExecutorTasks().stream().filter(task -> task != current).forEach(task -> task.cancel(true)); + executionExceptionFuture.completeExceptionally(failure); + stopAllOperations(failure); + return failure; + } + + InvocationOutcome awaitInvocationOutcome(CompletableFuture future) { + try { + var value = deadlineNanos == null + ? future.get() + : future.get( + remainingNanos(deadlineNanos - EXECUTION_DEADLINE_HEADROOM.toNanos()), + TimeUnit.NANOSECONDS); + return new InvocationOutcome<>(value, null); + } catch (InterruptedException interrupted) { + cleanupInterrupted.set(true); + var failure = lifecycleFailure("Invocation runtime thread was interrupted", interrupted); + lifecycleFailure.compareAndSet(null, failure); + return new InvocationOutcome<>(null, failure); + } catch (TimeoutException timeout) { + var failure = lifecycleFailure("Invocation deadline reached before execution completed", timeout); + lifecycleFailure.compareAndSet(null, failure); + return new InvocationOutcome<>(null, failure); + } catch (ExecutionException failure) { + return new InvocationOutcome<>(null, ExceptionHelper.unwrapInvocationFailure(failure.getCause())); + } + } + + private long cleanupDeadlineNanos() { + var localDeadline = addToNow(MAX_CLEANUP_DURATION); + if (deadlineNanos == null) { + return localDeadline; + } + return Math.min(localDeadline, deadlineNanos - RESPONSE_HEADROOM.toNanos()); + } + + private static UnrecoverableDurableExecutionException lifecycleFailure(String message, Throwable cause) { + return new UnrecoverableDurableExecutionException( + ErrorObject.builder() + .errorType(LIFECYCLE_ERROR_TYPE) + .errorMessage(message) + .build(), + true, + cause); + } + + private static Throwable combineFailures(Throwable primary, Throwable additional) { + if (primary == null) return additional; + if (additional == null || primary == additional) return primary; + if (isFatal(additional) && !isFatal(primary)) { + additional.addSuppressed(primary); + return additional; + } + primary.addSuppressed(additional); + return primary; + } + + @SuppressWarnings("removal") + static boolean isFatal(Throwable failure) { + return failure instanceof VirtualMachineError || failure instanceof ThreadDeath; + } + + static boolean isLifecycleFailure(Throwable failure) { + return failure instanceof UnrecoverableDurableExecutionException control + && LIFECYCLE_ERROR_TYPE.equals(control.getErrorObject().errorType()); + } + + private static long minDeadline(long deadline, Duration maximumWait) { + return Math.min(deadline, addToNow(maximumWait)); + } + + private static long addToNow(Duration duration) { + var now = System.nanoTime(); + var nanos = duration.toNanos(); + return now > Long.MAX_VALUE - nanos ? Long.MAX_VALUE : now + nanos; + } + + private static long remainingNanos(long deadline) { + return Math.max(0, deadline - System.nanoTime()); + } + + private static Duration remainingDuration(long deadline) { + return Duration.ofNanos(remainingNanos(deadline)); + } + + /** Cutoffs request cancellation; actual owner, continuation, and transport completion remain mandatory. */ + private Throwable finishActualInvocationCleanup(Throwable priorFailure, boolean retainRecordedFatal) { + var drain = new DrainFailure(true); + drain.run(() -> stopCheckpointContinuations(drain)); + drain.run(() -> awaitActualInvocationTasks(drain)); + drain.run(() -> closeLifecycle(drain)); + drain.run(checkpointManager::shutdown); + try { + drain.finish(); + } catch (Throwable failure) { + var recordedFatal = retainRecordedFatal ? getInvocationFatal() : null; + if (recordedFatal != null && isRecordedFatalAlias(failure, recordedFatal)) + return combineRecordedFatal(recordedFatal, priorFailure, failure); + // Keep shutdown's transport wrapper: close can run while its fatal cause already escapes the body. + return combineFailures(priorFailure, failure); + } + return priorFailure; + } + + private static boolean isRecordedFatalAlias(Throwable failure, Error fatal) { + var type = failure.getClass(); + return (type == CompletionException.class || type == ExecutionException.class) && failure.getCause() == fatal; + } + + private static Throwable combineRecordedFatal(Error fatal, Throwable priorFailure, Throwable observation) { + var pending = new ArrayList(); + if (priorFailure != null) pending.add(priorFailure); + pending.add(observation); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + for (var index = 0; index < pending.size(); index++) { + var diagnostic = pending.get(index); + if (diagnostic == fatal || !seen.add(diagnostic)) continue; + if (isRecordedFatalAlias(diagnostic, fatal)) { + // Omit the observation carrier, which points back to this fatal, but keep its diagnostics. + Collections.addAll(pending, diagnostic.getSuppressed()); + } else { + combineCompletionFailure(fatal, diagnostic); + } + } + return fatal; + } + + private void awaitActualInvocationTasks(DrainFailure drain) { + while (true) { + List> tasks; + synchronized (admissionLock) { + tasks = getActiveExecutorTasks(); + if (tasks.isEmpty()) { + lifecycleState = LifecycleState.CLOSED; + return; + } + } + for (var task : tasks) awaitTaskExit(task, drain); + } + } } diff --git a/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java b/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java index cd884796e9..13d9edc22e 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java @@ -399,6 +399,7 @@ private void clearHandlerOwner(CompletableFuture completion) { * @return the user function's result */ protected T runUserFunction(Integer attempt, Supplier userFunction) { + executionManager.checkUserFunctionExecutionAllowed(); var pluginRunner = getPluginRunner().captureUserFunctionHooks(); var startInfo = PluginInfoConverter.toUserFunctionStartInfo( operationIdentifier, @@ -409,6 +410,7 @@ protected T runUserFunction(Integer attempt, Supplier userFunction) { T result; try { executionManager.rethrowPluginFatalIfPresent(); + executionManager.checkUserFunctionExecutionAllowed(); result = userFunction.get(); executionManager.rethrowPluginFatalIfPresent(); } catch (Throwable e) { diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java index a79b8c515d..ad2d8fbed5 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java @@ -18,6 +18,7 @@ import java.util.List; import java.util.Map; import java.util.concurrent.ExecutorService; +import java.util.function.Supplier; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import software.amazon.awssdk.core.client.config.SdkAdvancedClientOption; @@ -92,6 +93,40 @@ void testBuilder_WithCustomExecutorService() { assertNotNull(config.getSerDes()); } + @Test + void testBuilder_WithInvocationExecutorFactory() { + Supplier factory = () -> mockExecutor; + + var config = DurableConfig.builder() + .withDurableExecutionClient(mockClient) + .withInvocationExecutorFactory(factory) + .build(); + + assertSame(factory, config.getInvocationExecutorFactory().orElseThrow()); + } + + @Test + void testBuilder_SharedExecutorAndInvocationFactoryAreMutuallyExclusive() { + var builder = DurableConfig.builder() + .withDurableExecutionClient(mockClient) + .withExecutorService(mockExecutor) + .withInvocationExecutorFactory(() -> mock(ExecutorService.class)); + + var exception = assertThrows(IllegalArgumentException.class, builder::build); + + assertEquals( + "ExecutorService and invocation executor factory cannot both be configured", exception.getMessage()); + } + + @Test + void testBuilder_NullInvocationExecutorFactoryThrowsException() { + var builder = DurableConfig.builder(); + + var exception = assertThrows(NullPointerException.class, () -> builder.withInvocationExecutorFactory(null)); + + assertEquals("Invocation executor factory cannot be null", exception.getMessage()); + } + @Test void testBuilder_DeserializeAfterSerializationDefaultsToTrue() { var config = @@ -174,6 +209,7 @@ void testBuilder_FluentAPI() { assertSame(builder, builder.withDurableExecutionClient(mockClient)); assertSame(builder, builder.withSerDes(mockSerDes)); assertSame(builder, builder.withExecutorService(mockExecutor)); + assertSame(builder, builder.withInvocationExecutorFactory(() -> mock(ExecutorService.class))); assertSame(builder, builder.withDeserializeAfterSerialization(false)); } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/exception/UnrecoverableDurableExecutionExceptionTest.java b/sdk/src/test/java/software/amazon/lambda/durable/exception/UnrecoverableDurableExecutionExceptionTest.java index 67c2f0fa15..b3232e38a4 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/exception/UnrecoverableDurableExecutionExceptionTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/exception/UnrecoverableDurableExecutionExceptionTest.java @@ -5,6 +5,7 @@ import static org.junit.jupiter.api.Assertions.*; import org.junit.jupiter.api.Test; +import software.amazon.awssdk.services.lambda.model.ErrorObject; class UnrecoverableDurableExecutionExceptionTest { @@ -36,4 +37,15 @@ void testIllegalDurableOperationException() { assertInstanceOf(RuntimeException.class, exception); assertInstanceOf(DurableExecutionException.class, exception); } + + @Test + void preservesCauseForRetryableInvocationFailures() { + var cause = new InterruptedException("interrupted"); + var error = ErrorObject.builder().errorMessage("cleanup failed").build(); + + var exception = new UnrecoverableDurableExecutionException(error, true, cause); + + assertSame(cause, exception.getCause()); + assertTrue(exception.isRetryable()); + } } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/AcknowledgedDeliveryFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/AcknowledgedDeliveryFailureTest.java new file mode 100644 index 0000000000..e35731bf37 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/AcknowledgedDeliveryFailureTest.java @@ -0,0 +1,265 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.concurrent.CancellationException; +import java.util.concurrent.CompletionException; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.CheckpointDurableExecutionResponse; +import software.amazon.awssdk.services.lambda.model.CheckpointUpdatedExecutionState; +import software.amazon.awssdk.services.lambda.model.ExecutionDetails; +import software.amazon.awssdk.services.lambda.model.GetDurableExecutionStateResponse; +import software.amazon.awssdk.services.lambda.model.Operation; +import software.amazon.awssdk.services.lambda.model.OperationAction; +import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.awssdk.services.lambda.model.OperationType; +import software.amazon.awssdk.services.lambda.model.OperationUpdate; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.client.DurableExecutionClient; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.DurableExecutionOutput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.plugin.InvocationStatus; + +@Timeout(15) +class AcknowledgedDeliveryFailureTest { + private static final String ARN = + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/ack/execution"; + private static final String RESULT = "x".repeat(6 * 1024 * 1024 + 100); + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void metadataFailureSurvivesCallerInterruptionDuringEnd(boolean cancelledMetadata) throws Exception { + RuntimeException failure = cancelledMetadata + ? new CancellationException("metadata request cancelled") + : new IllegalStateException("metadata store failed"); + var observed = run(failure, true); + assertSame(failure, observed.failure(), "Terminal ACK must not hide an unrelated delivery failure"); + assertNull(observed.output()); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void causelessMetadataFailureRetainsItsIdentity(boolean interruptCaller) throws Exception { + var failure = new CompletionException("metadata failure without a cause", null); + var observed = run(failure, interruptCaller); + assertSame(failure, observed.failure(), "A non-null delivery error must not become successful ACK delivery"); + assertNull(observed.output()); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void applicationExecutionExceptionRemainsTheDeliveryFailure(boolean interruptCaller) throws Exception { + var applicationFailure = new ExecutionException("application wrapper", new IllegalStateException("metadata")); + try (var fixture = new Fixture(new CompletionException(applicationFailure), applicationFailure)) { + var observed = fixture.observe(interruptCaller); + assertSame(applicationFailure, observed.failure()); + assertNull(observed.output()); + } + } + + @Test + void metadataFailureWithoutCallerInterruptionStillEscapes() throws Exception { + var failure = new IllegalStateException("metadata store failed"); + var observed = run(failure, false); + assertSame(failure, observed.failure()); + assertNull(observed.output()); + } + + @Test + void metadataCancellationWithoutCallerInterruptionRetainsIdentity() throws Exception { + var failure = new CancellationException("metadata request cancelled"); + var observed = run(failure, false); + assertSame(failure, observed.failure()); + assertNull(observed.output()); + } + + @Test + void acknowledgedResultStillWinsOverCallerInterruptionAlone() throws Exception { + var observed = run(null, true); + assertNull(observed.failure()); + assertEquals(ExecutionStatus.SUCCEEDED, observed.output().status()); + assertEquals("", observed.output().result(), "The acknowledged large payload is not returned twice"); + assertNull(observed.output().error()); + } + + private static Observed run(RuntimeException metadataFailure, boolean interruptCaller) throws Exception { + try (var fixture = new Fixture(metadataFailure)) { + return fixture.observe(interruptCaller); + } + } + + private static final class Fixture implements AutoCloseable, DurableExecutionPlugin { + private final ExecutorService users = Executors.newSingleThreadExecutor(); + private final ExecutorService callers = Executors.newSingleThreadExecutor(); + private final AtomicReference caller = new AtomicReference<>(); + private final AtomicReference owner = new AtomicReference<>(); + private final AtomicReference endInfo = new AtomicReference<>(); + private final CountDownLatch enteredEnd = new CountDownLatch(1); + private final CountDownLatch releaseEnd = new CountDownLatch(1); + private final CountDownLatch endFinished = new CountDownLatch(1); + private final AtomicBoolean endInterrupted = new AtomicBoolean(); + private final AtomicBoolean callerInterrupted = new AtomicBoolean(); + private final AtomicInteger endCalls = new AtomicInteger(); + private final AtomicReference> updates = new AtomicReference<>(); + private final Throwable expectedFailure; + private final DurableExecutionClient client; + private final DurableConfig config; + + private Fixture(RuntimeException metadataFailure) { + this(metadataFailure, metadataFailure); + } + + private Fixture(RuntimeException metadataFailure, Throwable expectedFailure) { + this.expectedFailure = expectedFailure; + client = client(metadataFailure, updates); + config = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(client) + .withCheckpointDelay(Duration.ZERO) + .withPlugins(ignoredInfo -> this) + .build(); + } + + @Override + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(owner.get(), Thread.currentThread(), "End remains on its Start owner"); + endInfo.set(info); + endCalls.incrementAndGet(); + enteredEnd.countDown(); + try { + assertTrue(releaseEnd.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException expected) { + endInterrupted.set(true); + Thread.currentThread().interrupt(); + } finally { + endFinished.countDown(); + } + } + + private Future invoke() { + return callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), null, TypeToken.get(String.class), (value, context) -> RESULT, config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + } + + private Observed observe(boolean interruptCaller) throws Exception { + var invocation = invoke(); + assertTrue(enteredEnd.await(5, TimeUnit.SECONDS), "The controlled End hook must start"); + var status = expectedFailure == null ? InvocationStatus.SUCCEEDED : InvocationStatus.RETRYING; + assertEquals(status, endInfo.get().invocationStatus()); + assertSame(expectedFailure, endInfo.get().executionError()); + if (interruptCaller) caller.get().interrupt(); + else releaseEnd.countDown(); + var observed = resultOf(invocation); + assertLifecycle(interruptCaller); + assertTransport(); + return observed; + } + + private void assertLifecycle(boolean interrupted) { + assertEquals(0, endFinished.getCount(), "The cooperative End owner exits before caller completion"); + assertEquals(1, endCalls.get()); + assertEquals(interrupted, endInterrupted.get()); + assertEquals(interrupted, callerInterrupted.get(), "Runtime caller interruption is restored"); + } + + private void assertTransport() { + verify(client, times(1)).checkpoint(any(), any(), any()); + verify(client, times(1)).getExecutionState(ARN, "accepted-token", "metadata"); + assertEquals(1, updates.get().size()); + var update = updates.get().get(0); + assertEquals(OperationType.EXECUTION, update.type()); + assertEquals(OperationAction.SUCCEED, update.action()); + assertEquals("execution", update.id()); + assertEquals('"' + RESULT + '"', update.payload()); + } + + @Override + public void close() throws InterruptedException { + releaseEnd.countDown(); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static Observed resultOf(Future invocation) throws Exception { + try { + return new Observed(invocation.get(5, TimeUnit.SECONDS), null); + } catch (ExecutionException failed) { + return new Observed(null, failed.getCause()); + } + } + + private static DurableExecutionClient client( + RuntimeException metadataFailure, AtomicReference> updates) { + var client = mock(DurableExecutionClient.class); + when(client.checkpoint(any(), any(), any())).thenAnswer(call -> { + updates.set(List.copyOf(call.getArgument(2))); + return CheckpointDurableExecutionResponse.builder() + .checkpointToken("accepted-token") + .newExecutionState(CheckpointUpdatedExecutionState.builder() + .nextMarker("metadata") + .build()) + .build(); + }); + when(client.getExecutionState(any(), any(), any())).thenAnswer(call -> { + if (metadataFailure != null) throw metadataFailure; + return GetDurableExecutionStateResponse.builder().build(); + }); + return client; + } + + private static DurableExecutionInput input() { + var execution = Operation.builder() + .id("execution") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails( + ExecutionDetails.builder().inputPayload("\"input\"").build()) + .build(); + return new DurableExecutionInput( + ARN, + "token", + CheckpointUpdatedExecutionState.builder().operations(execution).build()); + } + + private record Observed(DurableExecutionOutput output, Throwable failure) {} +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcherTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcherTest.java index 91d7d135ad..afdcbda6c2 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcherTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/ApiRequestDelayedBatcherTest.java @@ -437,4 +437,29 @@ private void await(CountDownLatch latch) { throw new AssertionError(e); } } + + @Test + void shutdownHonorsTimeoutWhileBatchActionIsBlocked() throws Exception { + var entered = new CountDownLatch(1); + var release = new CountDownLatch(1); + var batcher = new ApiRequestDelayedBatcher( + MAX_BATCH_SIZE, MAX_BATCH_BINARY_SIZE_IN_BYTES, item -> 0, items -> { + entered.countDown(); + try { + release.await(); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + } + }); + var future = batcher.submit(input, Duration.ZERO); + assertTrue(entered.await(5, TimeUnit.SECONDS)); + + try { + assertThrows(TimeoutException.class, () -> batcher.shutdown(Duration.ofMillis(50))); + assertFalse(future.isDone()); + } finally { + release.countDown(); + } + future.get(5, TimeUnit.SECONDS); + } } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/CanonicalLifecycleFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/CanonicalLifecycleFailureTest.java new file mode 100644 index 0000000000..18997b685e --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/CanonicalLifecycleFailureTest.java @@ -0,0 +1,209 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.Arrays; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.DurableExecutionOutput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(20) +class CanonicalLifecycleFailureTest { + enum Stop { + INTERRUPT, + TIMEOUT + } + + @ParameterizedTest + @CsvSource({"false,INTERRUPT", "true,INTERRUPT", "false,TIMEOUT", "true,TIMEOUT"}) + void firstLifecycleFailureRemainsCanonicalThroughCleanup(boolean plugins, Stop stop) throws Exception { + try (var fixture = new Fixture(plugins)) { + var invocation = fixture.invoke(stop == Stop.TIMEOUT ? 3000 : 60_000); + assertTrue(fixture.bodyEntered.await(5, TimeUnit.SECONDS)); + if (stop == Stop.INTERRUPT) fixture.caller.get().interrupt(); + assertTrue(fixture.ownerInterrupted.await(5, TimeUnit.SECONDS)); + var first = fixture.manager.get().getLifecycleFailure(); + assertNotNull(first); + assertEquals( + stop == Stop.INTERRUPT + ? "Invocation runtime thread was interrupted" + : "Invocation deadline reached before execution completed", + first.getMessage()); + if (stop == Stop.TIMEOUT) awaitActualCleanup(fixture.caller.get()); + fixture.releaseOwner.countDown(); + var observed = assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause(); + assertSame(first, observed, "Cleanup must return the same failure already selected for End"); + assertTrue(first.isRetryable()); + if (stop == Stop.INTERRUPT) assertInstanceOf(InterruptedException.class, first.getCause()); + else assertInstanceOf(TimeoutException.class, first.getCause()); + var laterMessage = stop == Stop.INTERRUPT + ? "Interrupted while draining invocation tasks" + : "Invocation lifecycle did not exit before the cleanup deadline"; + assertTrue( + Arrays.stream(first.getSuppressed()) + .anyMatch(later -> later instanceof UnrecoverableDurableExecutionException + && laterMessage.equals(later.getMessage())), + "The later cleanup diagnostic must remain attached to the first lifecycle failure"); + fixture.assertEnded(first, plugins ? InvocationStatus.RETRYING : null); + assertEquals(stop == Stop.INTERRUPT, fixture.callerInterrupted.get()); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void normalCompletionDoesNotSelectALifecycleFailure(boolean plugins) throws Exception { + try (var fixture = new Fixture(plugins)) { + var invocation = fixture.invoke(60_000); + assertTrue(fixture.bodyEntered.await(5, TimeUnit.SECONDS)); + fixture.releaseOwner.countDown(); + assertEquals( + ExecutionStatus.SUCCEEDED, + invocation.get(5, TimeUnit.SECONDS).status()); + assertNull(fixture.manager.get().getLifecycleFailure()); + fixture.assertEnded(null, plugins ? InvocationStatus.SUCCEEDED : null); + assertFalse(fixture.callerInterrupted.get()); + } + } + + private static void awaitActualCleanup(Thread caller) { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (Arrays.stream(caller.getStackTrace()) + .noneMatch(frame -> frame.getClassName().equals(ExecutionManager.class.getName()) + && frame.getMethodName().equals("finishActualInvocationCleanup"))) { + assertTrue( + System.nanoTime() < deadline, "The cancellation allowance did not expire while the owner was held"); + Thread.yield(); + } + } + + private static final class Fixture implements AutoCloseable { + private final ExecutorService users = Executors.newSingleThreadExecutor(); + private final ExecutorService callers = Executors.newSingleThreadExecutor(); + private final CountDownLatch bodyEntered = new CountDownLatch(1); + private final CountDownLatch ownerInterrupted = new CountDownLatch(1); + private final CountDownLatch releaseOwner = new CountDownLatch(1); + private final AtomicReference caller = new AtomicReference<>(); + private final AtomicReference owner = new AtomicReference<>(); + private final AtomicReference manager = new AtomicReference<>(); + private final AtomicReference end = new AtomicReference<>(); + private final AtomicReference endThread = new AtomicReference<>(); + private final AtomicBoolean callerInterrupted = new AtomicBoolean(); + private final AtomicBoolean bodyExited = new AtomicBoolean(); + private final AtomicInteger endCalls = new AtomicInteger(); + private final DurableConfig config; + + private Fixture(boolean plugins) { + var builder = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()); + if (plugins) + builder.withPlugins(ignored -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + end.set(info); + endThread.set(Thread.currentThread()); + endCalls.incrementAndGet(); + } + }); + config = builder.build(); + } + + private Future invoke(int remainingMillis) { + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(remainingMillis); + when(runtime.getAwsRequestId()).thenReturn("canonical-life-request"); + return callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), + runtime, + TypeToken.get(String.class), + (value, context) -> { + manager.set(((DurableContextImpl) context).getExecutionManager()); + owner.set(Thread.currentThread()); + bodyEntered.countDown(); + try { + awaitOwnerRelease(); + return "result"; + } finally { + bodyExited.set(true); + } + }, + config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + } + + private void awaitOwnerRelease() { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (true) { + try { + assertTrue(releaseOwner.await(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + return; + } catch (InterruptedException ignored) { + ownerInterrupted.countDown(); + } + } + } + + private void assertEnded(Throwable failure, InvocationStatus status) { + assertTrue(bodyExited.get()); + if (status == null) { + assertEquals(0, endCalls.get()); + return; + } + assertEquals(1, endCalls.get()); + assertSame(owner.get(), endThread.get()); + assertEquals(status, end.get().invocationStatus()); + assertSame(failure, end.get().executionError()); + } + + @Override + public void close() throws InterruptedException { + releaseOwner.countDown(); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/canonical", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("canonical") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/CheckpointManagerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/CheckpointManagerTest.java index 9ea4a5dba6..90975bd0f1 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/CheckpointManagerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/CheckpointManagerTest.java @@ -42,6 +42,8 @@ class CheckpointManagerTest { + private static final Duration SHUTDOWN_TIMEOUT = Duration.ofSeconds(5); + private DurableConfig config; private DurableExecutionClient client; private CheckpointManager batcher; @@ -271,11 +273,11 @@ void detachedPollersSettleAfterFirstObserverAborts(boolean earlierPollFailure, b } @Test - void shutdown_completesAllPendingPollersWithException() { + void shutdown_completesAllPendingPollersWithException() throws Exception { var future1 = batcher.pollForUpdate("op-1"); var future2 = batcher.pollForUpdate("op-2"); - batcher.shutdown(); + batcher.shutdown(SHUTDOWN_TIMEOUT); assertTrue(future1.isCompletedExceptionally()); assertTrue(future2.isCompletedExceptionally()); @@ -297,7 +299,7 @@ void shutdown_waitsForPendingCheckpoints() throws Exception { .type(OperationType.STEP) .build()); - batcher.shutdown(); + batcher.shutdown(SHUTDOWN_TIMEOUT); assertTrue(future.isDone()); verify(client, atLeastOnce()).checkpoint(anyString(), anyString(), anyList()); diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/ContinuationFatalBudgetTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/ContinuationFatalBudgetTest.java index 38bdb53eda..d78fab8925 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/ContinuationFatalBudgetTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/ContinuationFatalBudgetTest.java @@ -18,7 +18,7 @@ class ContinuationFatalBudgetTest { @ParameterizedTest @ValueSource(booleans = {false, true}) @Timeout(10) - void admittedContinuationMustSettleBeforeFatalClose(boolean reportWhileClosing) throws Exception { + void admittedContinuationMustExitBeforeFatalClose(boolean reportWhileClosing) throws Exception { var input = new DurableExecutionInput( "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/budget/execution", "token", @@ -40,6 +40,7 @@ void admittedContinuationMustSettleBeforeFatalClose(boolean reportWhileClosing) var caller = Executors.newSingleThreadExecutor(); var entered = new CountDownLatch(1); var release = new CountDownLatch(1); + var bodyExited = new CountDownLatch(1); var fatal = new InternalError("simulated invocation fatal"); var selected = manager.runUntilCompleteOrSuspend(new CompletableFuture()); var observed = manager.runCheckpointContinuation( @@ -50,6 +51,8 @@ void admittedContinuationMustSettleBeforeFatalClose(boolean reportWhileClosing) } catch (InterruptedException failure) { Thread.currentThread().interrupt(); throw new AssertionError(failure); + } finally { + bodyExited.countDown(); } }, worker); @@ -69,14 +72,15 @@ void admittedContinuationMustSettleBeforeFatalClose(boolean reportWhileClosing) manager.recordInvocationFatal(fatal); } assertThrows(TimeoutException.class, () -> closing.get(700, TimeUnit.MILLISECONDS)); + assertEquals(1, bodyExited.getCount(), "The continuation body is still held"); assertFalse(observed.isDone(), "The actual continuation is still owned while held"); release.countDown(); var closeFailure = closing.get(2, TimeUnit.SECONDS); if (closeFailure != null) assertSame(fatal, ExceptionHelper.unwrapAsyncFailure(closeFailure)); assertSame( fatal, ExceptionHelper.unwrapAsyncFailure(assertThrows(CompletionException.class, selected::join))); - assertTrue(observed.isDone(), "Close must await actual continuation settlement"); - release.countDown(); + assertEquals(0, bodyExited.getCount(), "Close must await actual continuation exit"); + // Observer publication follows lease release so its inline callbacks can safely call close again. observed.get(2, TimeUnit.SECONDS); } finally { release.countDown(); diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/DrainingAdmissionCancellationTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/DrainingAdmissionCancellationTest.java new file mode 100644 index 0000000000..fcedd9e88a --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/DrainingAdmissionCancellationTest.java @@ -0,0 +1,300 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Duration; +import java.time.Instant; +import java.util.Arrays; +import java.util.Map; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.function.BooleanSupplier; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.DurableExecutionOutput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.model.OperationSubType; +import software.amazon.lambda.durable.operation.BaseDurableOperation; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(20) +class DrainingAdmissionCancellationTest { + enum Timing { + AFTER_FINAL_BROADCAST, + DURING_INSERTION, + GRACEFUL + } + + @ParameterizedTest + @CsvSource({ + "AFTER_FINAL_BROADCAST,false", + "AFTER_FINAL_BROADCAST,true", + "DURING_INSERTION,false", + "DURING_INSERTION,true", + "GRACEFUL,false", + "GRACEFUL,true" + }) + void replayedWaitCannotMissCancellationButGracefulDescendantsRemainAllowed(Timing timing, boolean plugins) + throws Exception { + var childId = new OperationIdGenerator(null).nextOperationId(); + var waitId = new OperationIdGenerator(childId).nextOperationId(); + var childEntered = new CountDownLatch(1); + var releaseChild = new CountDownLatch(1); + var childExited = new CountDownLatch(1); + var interruptedChild = new CountDownLatch(1); + var beforeInsertion = new CountDownLatch(1); + var releaseInsertion = new CountDownLatch(1); + var manager = new AtomicReference(); + var caller = new AtomicReference(); + var rootOwner = new AtomicReference(); + var endOwner = new AtomicReference(); + var end = new AtomicReference(); + var ends = new AtomicInteger(); + var stepBodies = new AtomicInteger(); + var failureSeen = new AtomicReference(); + var lateWait = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var workers = Executors.newCachedThreadPool(); + var callers = Executors.newSingleThreadExecutor(); + var client = TestUtils.createMockClient(); + var config = DurableConfig.builder() + .withExecutorService(workers) + .withDurableExecutionClient(client) + .withCheckpointDelay(Duration.ZERO); + if (plugins) + config.withPlugins(info -> new DurableExecutionPlugin() { + public void onInvocationEnd(InvocationEndInfo info) { + endOwner.set(Thread.currentThread()); + end.set(info); + ends.incrementAndGet(); + } + }); + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(timing == Timing.AFTER_FINAL_BROADCAST ? 3000 : 10_000); + when(runtime.getAwsRequestId()).thenReturn("draining-admission"); + var pending = timing != Timing.GRACEFUL; + var history = CheckpointUpdatedExecutionState.builder() + .operations( + Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build(), + Operation.builder() + .id(childId) + .name("child") + .type(OperationType.CONTEXT) + .subType(OperationSubType.RUN_IN_CHILD_CONTEXT.getValue()) + .status(OperationStatus.STARTED) + .build(), + Operation.builder() + .id(waitId) + .parentId(childId) + .name("old-wait") + .type(OperationType.WAIT) + .subType(OperationSubType.WAIT.getValue()) + .status(pending ? OperationStatus.PENDING : OperationStatus.SUCCEEDED) + .waitDetails(WaitDetails.builder() + .scheduledEndTimestamp(Instant.now().plusSeconds(3600)) + .build()) + .build()) + .build(); + var input = new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/draining/exec", + "token", + history); + Future response = null; + try { + response = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input, + runtime, + TypeToken.get(String.class), + (value, root) -> { + rootOwner.set(Thread.currentThread()); + var execution = ((DurableContextImpl) root).getExecutionManager(); + manager.set(execution); + if (timing == Timing.DURING_INSERTION) + installInsertionGate( + execution, waitId, beforeInsertion, releaseInsertion, lateWait); + root.runInChildContextAsync("child", String.class, child -> { + childEntered.countDown(); + awaitIgnoringInterrupt(releaseChild, interruptedChild); + try { + var wait = child.waitAsync("old-wait", Duration.ofHours(1)); + lateWait.set((BaseDurableOperation) wait); + wait.get(); + if (timing == Timing.GRACEFUL) + child.step("new-descendant", String.class, step -> { + stepBodies.incrementAndGet(); + return "step"; + }); + } catch (Throwable failure) { + failureSeen.set(failure); + } finally { + childExited.countDown(); + } + return "child"; + }); + await(childEntered); + return "root"; + }, + config.build()); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + await(childEntered); + awaitCondition(() -> inMethod(caller.get(), "awaitTasksUntil")); + if (timing == Timing.DURING_INSERTION) { + releaseChild.countDown(); + await(beforeInsertion); + caller.get().interrupt(); + awaitCondition(() -> manager.get().getLifecycleFailure() != null + && caller.get().getState() == Thread.State.BLOCKED + && inMethod(caller.get(), "awaitTasksUntil")); + assertFalse( + inMethod(caller.get(), "signalLifecycleFailure"), "Cancellation broadcast must already finish"); + releaseInsertion.countDown(); + } else if (timing == Timing.AFTER_FINAL_BROADCAST) { + caller.get().interrupt(); + await(interruptedChild); + awaitCondition(() -> inMethod(caller.get(), "awaitActualInvocationTasks")); + releaseChild.countDown(); + } else { + assertEquals( + ExecutionManager.LifecycleState.DRAINING, manager.get().getLifecycleState()); + assertNull(manager.get().getLifecycleFailure()); + releaseChild.countDown(); + } + assertTrue( + childExited.await(700, TimeUnit.MILLISECONDS), + "A cancelled descendant must reject the old PENDING wait, not block beyond the final broadcast"); + if (pending) { + var rejected = assertInstanceOf(UnrecoverableDurableExecutionException.class, failureSeen.get()); + assertSame(manager.get().getLifecycleFailure(), rejected); + var observedResponse = response; + var callerFailure = assertThrows(ExecutionException.class, () -> responseGet(observedResponse)); + assertSame(rejected, callerFailure.getCause()); + assertTrue(callerInterrupted.get()); + assertEquals(0, stepBodies.get()); + verify(client, never()).checkpoint(any(), any(), any()); + } else { + var result = (DurableExecutionOutput) response.get(5, TimeUnit.SECONDS); + assertEquals(ExecutionStatus.SUCCEEDED, result.status()); + assertNull(failureSeen.get()); + assertEquals(1, stepBodies.get()); + assertFalse(callerInterrupted.get()); + } + assertEquals(plugins ? 1 : 0, ends.get()); + if (plugins) { + assertSame(rootOwner.get(), endOwner.get()); + assertEquals( + pending ? InvocationStatus.RETRYING : InvocationStatus.SUCCEEDED, + end.get().invocationStatus()); + } + } finally { + releaseChild.countDown(); + releaseInsertion.countDown(); + var waiting = lateWait.get(); + if (waiting != null && !waiting.getCompletionFuture().isDone()) + waiting.getCompletionFuture() + .completeExceptionally(new IllegalStateException("test teardown releases blocked wait")); + if (response != null) { + try { + response.get(5, TimeUnit.SECONDS); + } catch (ExecutionException expected) { + } + } + workers.shutdownNow(); + callers.shutdownNow(); + assertTrue(workers.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static Object responseGet(Future response) throws Exception { + return response.get(5, TimeUnit.SECONDS); + } + + @SuppressWarnings("unchecked") + private static void installInsertionGate( + ExecutionManager manager, + String target, + CountDownLatch entered, + CountDownLatch release, + AtomicReference rescue) { + try { + var field = ExecutionManager.class.getDeclaredField("registeredOperations"); + field.setAccessible(true); + var original = (Map) field.get(manager); + var gated = new ConcurrentHashMap(original) { + @Override + public BaseDurableOperation put(String id, BaseDurableOperation operation) { + if (id.equals(target)) { + rescue.set(operation); + entered.countDown(); + awaitIgnoringInterrupt(release, new CountDownLatch(0)); + } + return super.put(id, operation); + } + }; + field.set(manager, gated); + } catch (ReflectiveOperationException failure) { + throw new AssertionError(failure); + } + } + + private static boolean inMethod(Thread thread, String method) { + return thread != null + && Arrays.stream(thread.getStackTrace()) + .anyMatch(frame -> frame.getMethodName().equals(method)); + } + + private static void awaitCondition(BooleanSupplier condition) throws InterruptedException { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(6); + while (!condition.getAsBoolean()) { + assertTrue(System.nanoTime() < deadline, "Fixture did not reach the required admission/broadcast phase"); + TimeUnit.MILLISECONDS.sleep(5); + } + } + + private static void await(CountDownLatch gate) { + try { + assertTrue(gate.await(8, TimeUnit.SECONDS)); + } catch (InterruptedException failure) { + Thread.currentThread().interrupt(); + throw new AssertionError(failure); + } + } + + private static void awaitIgnoringInterrupt(CountDownLatch gate, CountDownLatch interrupted) { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(12); + while (true) { + try { + assertTrue(gate.await(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + return; + } catch (InterruptedException ignored) { + interrupted.countDown(); + } + } + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerAdmissionTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerAdmissionTest.java index 76a5845914..cdc5938c82 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerAdmissionTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerAdmissionTest.java @@ -5,24 +5,31 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.doThrow; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; +import com.amazonaws.services.lambda.runtime.Context; import java.time.Duration; import java.time.Instant; import java.util.List; +import java.util.concurrent.CompletableFuture; import java.util.concurrent.CompletionException; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; @@ -38,6 +45,7 @@ import software.amazon.awssdk.services.lambda.model.OperationUpdate; import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.client.DurableExecutionClient; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; import software.amazon.lambda.durable.model.DurableExecutionInput; import software.amazon.lambda.durable.model.OperationSubType; import software.amazon.lambda.durable.operation.BaseDurableOperation; @@ -57,11 +65,28 @@ void setUp() { .withCheckpointDelay(Duration.ofHours(1)) .withPollingStrategy(attempt -> Duration.ofHours(1)) .build()); - manager = createManager(); + manager = createManager(null); manager.registerActiveThread(null); when(client.checkpoint(any(), any(), any())).thenReturn(response()); } + private ExecutionManager createManager(Context context) { + var execution = Operation.builder() + .id("execution") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .build(); + return new ExecutionManager( + new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/test/execution", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(execution) + .build()), + config, + context); + } + @AfterEach void tearDown() throws Exception { manager.close(); @@ -145,6 +170,54 @@ void shutdownWaitsForAdmittedPollAndPreservesInterruption(boolean interrupted) t } } + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void stuckCheckpointAdmissionWaitsPastCleanupDeadlineAfterSelectingRetry(boolean interrupted) throws Exception { + manager.close(); + var context = mock(Context.class); + when(context.getRemainingTimeInMillis()).thenReturn(500); + manager = createManager(context); + var admissionEntered = new CountDownLatch(1); + var releaseAdmission = new CountDownLatch(1); + doAnswer(invocation -> { + admissionEntered.countDown(); + await(releaseAdmission); + return invocation.callRealMethod(); + }) + .when(config) + .getPollingStrategy(); + try { + var admission = callers.submit(() -> manager.pollForOperationUpdates("pending")); + await(admissionEntered); + var restoredInterrupt = new AtomicBoolean(); + var shutdown = callers.submit(() -> { + var outcome = manager.awaitInvocationOutcome(new CompletableFuture<>()); + if (interrupted) Thread.currentThread().interrupt(); + try { + return manager.finishInvocation(outcome.failure()); + } finally { + restoredInterrupt.set(Thread.interrupted()); + } + }); + callers.submit(this::awaitClosedAdmission).get(5, TimeUnit.SECONDS); + var failure = assertInstanceOf(UnrecoverableDurableExecutionException.class, manager.getLifecycleFailure()); + assertTrue(failure.isRetryable()); + assertInstanceOf(TimeoutException.class, failure.getCause()); + // The cutoff selects RETRYING and fences admission, but cannot skip an already-admitted enqueue. + assertThrows(TimeoutException.class, () -> shutdown.get(600, TimeUnit.MILLISECONDS)); + assertThrows(RejectedExecutionException.class, () -> manager.sendOperationUpdate(update())); + verify(client, never()).checkpoint(any(), any(), any()); + releaseAdmission.countDown(); + var poll = admission.get(5, TimeUnit.SECONDS); + assertSame(failure, shutdown.get(5, TimeUnit.SECONDS)); + assertEquals(interrupted, restoredInterrupt.get()); + assertTrue(poll.isCompletedExceptionally()); + verify(client, never()).checkpoint(any(), any(), any()); + } finally { + releaseAdmission.countDown(); + } + } + @Test void failedAdmissionDoesNotPreventShutdown() throws Exception { doThrow(new IllegalArgumentException("polling configuration failed")) @@ -193,7 +266,10 @@ void drainingAllowsOnlyTasksOfPreviouslyRegisteredOperations() throws Exception @ValueSource(booleans = {false, true}) void admittedContextTasksCanRegisterDescendantsWithoutLeakingAdmission(boolean coordinator) throws Exception { var executor = Executors.newSingleThreadExecutor(); + manager.close(); when(config.getExecutorService()).thenReturn(executor); + manager = createManager(); + manager.registerActiveThread(null); var parent = contextOperation("parent", coordinator ? OperationSubType.MAP : OperationSubType.RUN_IN_CHILD_CONTEXT); var descendant = step("descendant"); @@ -238,7 +314,10 @@ void inlineNestedInvocationCannotUseItsCallersDrainingPermission() throws Except }) .when(inline) .execute(any(Runnable.class)); + manager.close(); when(config.getExecutorService()).thenReturn(inline); + manager = createManager(); + manager.registerActiveThread(null); var other = createManager(); var parent = contextOperation("parent", OperationSubType.RUN_IN_CHILD_CONTEXT); var nested = contextOperation("nested", OperationSubType.RUN_IN_CHILD_CONTEXT); @@ -283,20 +362,7 @@ void alreadyRunningRootCannotAdmitOperationsAfterDrainingStarts() throws Excepti } private ExecutionManager createManager() { - var execution = Operation.builder() - .id("execution") - .type(OperationType.EXECUTION) - .status(OperationStatus.STARTED) - .build(); - return new ExecutionManager( - new DurableExecutionInput( - "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/test/execution", - "token", - CheckpointUpdatedExecutionState.builder() - .operations(execution) - .build()), - config, - null); + return createManager(null); } private static BaseDurableOperation contextOperation(String id, OperationSubType subType) { diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerLifecycleTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerLifecycleTest.java index edd1cd741a..98321b071f 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerLifecycleTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/ExecutionManagerLifecycleTest.java @@ -4,18 +4,24 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; import static org.junit.jupiter.api.Assertions.assertNotEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNotSame; import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; +import static software.amazon.lambda.durable.model.ExecutionStatus.PENDING; +import static software.amazon.lambda.durable.model.ExecutionStatus.SUCCEEDED; import com.amazonaws.services.lambda.runtime.Context; import java.time.Duration; +import java.time.Instant; import java.util.List; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; @@ -25,11 +31,14 @@ import java.util.concurrent.SynchronousQueue; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicInteger; +import java.util.stream.IntStream; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; import software.amazon.awssdk.services.lambda.model.CheckpointUpdatedExecutionState; +import software.amazon.awssdk.services.lambda.model.ExecutionDetails; import software.amazon.awssdk.services.lambda.model.Operation; import software.amazon.awssdk.services.lambda.model.OperationAction; import software.amazon.awssdk.services.lambda.model.OperationStatus; @@ -37,13 +46,15 @@ import software.amazon.awssdk.services.lambda.model.OperationUpdate; import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; import software.amazon.lambda.durable.model.DurableExecutionInput; import software.amazon.lambda.durable.operation.BaseDurableOperation; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; class ExecutionManagerLifecycleTest { private static final String EXECUTION_ID = "execution"; - private static final String EXECUTION_ARN = - "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/test/" + EXECUTION_ID; @Test void capturesRequestIdAndInvocationDeadline() { @@ -360,25 +371,282 @@ void drainingRejectsNewWorkButAllowsExistingCheckpointCleanup() throws Exception } } + @Test + void interruptedDrainStillShutsDownCheckpointsAndRestoresInterrupt() throws Exception { + var executor = Executors.newSingleThreadExecutor(); + var taskEntered = new CountDownLatch(1); + var taskInterrupted = new CountDownLatch(1); + var manager = createManager(null, executor); + manager.submitRootTask(() -> { + taskEntered.countDown(); + try { + new CountDownLatch(1).await(); + } catch (InterruptedException expected) { + taskInterrupted.countDown(); + } + return "cancelled"; + }); + assertTrue(taskEntered.await(5, TimeUnit.SECONDS)); + var checkpoint = manager.sendOperationUpdate(OperationUpdate.builder() + .id("step") + .name("step") + .type(OperationType.STEP) + .subType("Step") + .action(OperationAction.START) + .build()); + + Thread.currentThread().interrupt(); + try { + var failure = manager.finishInvocation(null); + + var lifecycleFailure = assertInstanceOf(UnrecoverableDurableExecutionException.class, failure); + assertInstanceOf(InterruptedException.class, lifecycleFailure.getCause()); + assertEquals(ExecutionManager.LifecycleState.CLOSED, manager.getLifecycleState()); + assertTrue(checkpoint.isDone(), "checkpoint shutdown should still be attempted"); + assertTrue(Thread.currentThread().isInterrupted()); + Thread.interrupted(); + assertTrue(taskInterrupted.await(5, TimeUnit.SECONDS)); + } finally { + Thread.interrupted(); + stop(executor); + } + } + + @Test + void invocationDeadlineCancelsAStuckRootAndReturnsRetryableFailure() throws Exception { + var context = mock(Context.class); + when(context.getAwsRequestId()).thenReturn("deadline-request"); + when(context.getRemainingTimeInMillis()).thenReturn(700); + var executor = Executors.newSingleThreadExecutor(); + var entered = new CountDownLatch(1); + var interrupted = new CountDownLatch(1); + var manager = createManager(context, executor); + var root = manager.submitRootTask(() -> { + entered.countDown(); + try { + new CountDownLatch(1).await(); + } catch (InterruptedException expected) { + interrupted.countDown(); + } + return "cancelled"; + }); + assertTrue(entered.await(5, TimeUnit.SECONDS)); + + try { + var outcome = manager.awaitInvocationOutcome(root); + var deadlineFailure = assertInstanceOf(UnrecoverableDurableExecutionException.class, outcome.failure()); + assertTrue(deadlineFailure.isRetryable()); + + assertEquals(deadlineFailure, manager.finishInvocation(deadlineFailure)); + assertTrue(interrupted.await(5, TimeUnit.SECONDS)); + assertEquals(ExecutionManager.LifecycleState.CLOSED, manager.getLifecycleState()); + } finally { + stop(executor); + } + } + + @Test + void responseCriticalCheckpointWaitIsBoundedByInvocationDeadline() { + var context = mock(Context.class); + when(context.getAwsRequestId()).thenReturn("checkpoint-deadline-request"); + when(context.getRemainingTimeInMillis()).thenReturn(250); + var manager = createManager(context, null); + + var failure = assertThrows( + UnrecoverableDurableExecutionException.class, + () -> manager.awaitCheckpointCompletion(new CompletableFuture<>())); + + assertTrue(failure.isRetryable()); + assertInstanceOf(TimeoutException.class, failure.getCause()); + assertEquals(failure, manager.finishInvocation(failure)); + } + + @Test + void pendingResponseAndInvocationEndWaitForRootFinally() throws Exception { + var users = Executors.newCachedThreadPool(); + var runtime = Executors.newSingleThreadExecutor(); + var cleanupEntered = new CountDownLatch(1); + var releaseCleanup = new CountDownLatch(1); + var invocationEnded = new CountDownLatch(1); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withExecutorService(users) + .withPlugins(ignoredInfo -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + invocationEnded.countDown(); + } + }) + .build(); + + try { + var response = runtime.submit(() -> DurableExecutor.execute( + input(), + context(10_000), + TypeToken.get(String.class), + (value, durableContext) -> { + try { + durableContext.wait("wait", Duration.ofSeconds(5)); + return value; + } finally { + cleanupEntered.countDown(); + await(releaseCleanup); + } + }, + config)); + assertTrue(cleanupEntered.await(5, TimeUnit.SECONDS)); + + assertThrows(TimeoutException.class, () -> response.get(100, TimeUnit.MILLISECONDS)); + assertEquals(1, invocationEnded.getCount()); + + releaseCleanup.countDown(); + assertEquals(PENDING, response.get(5, TimeUnit.SECONDS).status()); + assertTrue(invocationEnded.await(5, TimeUnit.SECONDS)); + } finally { + releaseCleanup.countDown(); + stop(users); + stop(runtime); + } + } + + @Test + void invocationExecutorFactoryIsolatesConcurrentRootsAndOwnsShutdown() throws Exception { + var executors = new CopyOnWriteArrayList(); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> { + var executor = Executors.newFixedThreadPool(2); + executors.add(executor); + return executor; + }) + .build(); + var runtime = Executors.newFixedThreadPool(2); + var rootsEntered = new CountDownLatch(2); + + try { + var responses = IntStream.range(0, 2) + .mapToObj(index -> runtime.submit(() -> DurableExecutor.execute( + input("isolated-" + index), + context(10_000), + TypeToken.get(String.class), + (value, durableContext) -> { + rootsEntered.countDown(); + await(rootsEntered); + return durableContext.step("step", String.class, step -> value); + }, + config))) + .toList(); + + for (var response : responses) { + assertEquals(SUCCEEDED, response.get(5, TimeUnit.SECONDS).status()); + } + assertEquals(2, executors.size()); + assertNotSame(executors.get(0), executors.get(1)); + assertTrue(executors.stream().allMatch(ExecutorService::isShutdown)); + assertTrue(executors.stream().allMatch(ExecutorService::isTerminated)); + } finally { + stop(runtime); + for (var executor : executors) { + stop(executor); + } + } + } + + @Test + void sharedExecutorRemainsCallerOwned() throws Exception { + var sharedExecutor = Executors.newCachedThreadPool(); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withExecutorService(sharedExecutor) + .build(); + + try { + var response = DurableExecutor.execute( + input("shared"), + context(10_000), + TypeToken.get(String.class), + (value, durableContext) -> value, + config); + + assertEquals(SUCCEEDED, response.status()); + assertFalse(sharedExecutor.isShutdown()); + } finally { + stop(sharedExecutor); + } + } + + @Test + void invocationExecutorFactoryMustReturnUsableExecutor() { + var nullConfig = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> null) + .build(); + assertThrows(NullPointerException.class, () -> new ExecutionManager(input("null-executor"), nullConfig, null)); + + var shutdownExecutor = Executors.newSingleThreadExecutor(); + shutdownExecutor.shutdown(); + var shutdownConfig = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> shutdownExecutor) + .build(); + assertThrows( + IllegalStateException.class, + () -> new ExecutionManager(input("shutdown-executor"), shutdownConfig, null)); + } + + @Test + void invocationExecutorFactoryRejectsExecutorAlreadyLeasedToAnotherInvocation() { + var executor = Executors.newFixedThreadPool(2); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> executor) + .build(); + var first = new ExecutionManager(input("first"), config, null); + + try { + var exception = assertThrows( + IllegalStateException.class, () -> new ExecutionManager(input("second"), config, null)); + assertEquals("Invocation executor factory returned an executor already in use", exception.getMessage()); + } finally { + first.close(); + } + assertTrue(executor.isTerminated()); + } + private ExecutionManager createManager(Context context, ExecutorService executor) { var config = DurableConfig.builder().withDurableExecutionClient(TestUtils.createMockClient()); if (executor != null) { config.withExecutorService(executor); } + return new ExecutionManager(input(), config.build(), context); + } + + private DurableExecutionInput input() { + return input(EXECUTION_ID); + } + + private DurableExecutionInput input(String executionId) { var execution = Operation.builder() - .id(EXECUTION_ID) + .id(executionId) .type(OperationType.EXECUTION) .status(OperationStatus.STARTED) + .startTimestamp(Instant.now()) + .executionDetails( + ExecutionDetails.builder().inputPayload("\"input\"").build()) .build(); - return new ExecutionManager( - new DurableExecutionInput( - EXECUTION_ARN, - "token", - CheckpointUpdatedExecutionState.builder() - .operations(List.of(execution)) - .build()), - config.build(), - context); + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/test/" + executionId, + "token", + CheckpointUpdatedExecutionState.builder() + .operations(List.of(execution)) + .build()); + } + + private Context context(int remainingMillis) { + var context = mock(Context.class); + when(context.getAwsRequestId()).thenReturn("request-id"); + when(context.getRemainingTimeInMillis()).thenReturn(remainingMillis); + return context; } private static void await(CountDownLatch latch) { diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/HandlerScopeHandoffTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/HandlerScopeHandoffTest.java index 3eef976a30..d874896332 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/HandlerScopeHandoffTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/HandlerScopeHandoffTest.java @@ -25,11 +25,11 @@ import software.amazon.lambda.durable.plugin.*; import software.amazon.lambda.durable.util.ExceptionHelper; -/** The old caller handoff is replaced by handler-owned completion, including exhausted runtime budgets. */ +/** The old caller handoff is replaced by handler-owned completion, after the execution cutoff. */ class HandlerScopeHandoffTest { @ParameterizedTest @ValueSource(booleans = {false, true}) - void controlWaitsForScopeOwnerAndEndDespiteExpiredHandoffBudget(boolean retry) throws Exception { + void selectedControlWaitsForScopeOwnerAndEndAfterExecutionCutoff(boolean retry) throws Exception { var scopeEntered = new CountDownLatch(1); var releaseScope = new CountDownLatch(1); var endEntered = new CountDownLatch(1); @@ -37,6 +37,7 @@ void controlWaitsForScopeOwnerAndEndDespiteExpiredHandoffBudget(boolean retry) t var owner = new AtomicReference(); var endThread = new AtomicReference(); var end = new AtomicReference(); + var executionManager = new AtomicReference(); var original = new UnrecoverableDurableExecutionException( ErrorObject.builder().errorMessage("retry").build(), true); var plugin = new ScopedPlugin( @@ -57,7 +58,7 @@ void controlWaitsForScopeOwnerAndEndDespiteExpiredHandoffBudget(boolean retry) t var workers = Executors.newSingleThreadExecutor(); var callers = Executors.newSingleThreadExecutor(); var runtime = mock(Context.class); - when(runtime.getRemainingTimeInMillis()).thenReturn(0); + when(runtime.getRemainingTimeInMillis()).thenReturn(3000); try { var config = DurableConfig.builder() .withExecutorService(workers) @@ -70,6 +71,7 @@ void controlWaitsForScopeOwnerAndEndDespiteExpiredHandoffBudget(boolean retry) t TypeToken.get(String.class), (value, context) -> { var manager = ((DurableContextImpl) context).getExecutionManager(); + executionManager.set(manager); try { if (retry) manager.terminateExecution(original); manager.suspendExecution(); @@ -80,12 +82,21 @@ void controlWaitsForScopeOwnerAndEndDespiteExpiredHandoffBudget(boolean retry) t }, config)); assertTrue(scopeEntered.await(3, TimeUnit.SECONDS)); - assertThrows(TimeoutException.class, () -> response.get(600, TimeUnit.MILLISECONDS)); + // Admission must precede the deadline. The selected control may then wait for its real scope + // owner past the execution cutoff, without consuming the separate cleanup budget. + var manager = executionManager.get(); + var untilCutoff = + manager.getRemainingInvocationTime().orElseThrow().toNanos() - TimeUnit.MILLISECONDS.toNanos(500); + assertTrue(untilCutoff > 0, "The scope must open before the execution cutoff"); + assertThrows( + TimeoutException.class, + () -> response.get(untilCutoff + TimeUnit.MILLISECONDS.toNanos(1), TimeUnit.NANOSECONDS)); + assertTrue(manager.getRemainingInvocationTime().orElseThrow().toMillis() < 500); assertEquals(1, endEntered.getCount(), "No End before the real scope owner finishes"); releaseScope.countDown(); assertTrue(endEntered.await(3, TimeUnit.SECONDS)); assertSame(owner.get(), endThread.get(), "End belongs to the handler thread"); - assertThrows(TimeoutException.class, () -> response.get(100, TimeUnit.MILLISECONDS)); + assertThrows(TimeoutException.class, () -> response.get(50, TimeUnit.MILLISECONDS)); releaseEnd.countDown(); if (retry) { var failure = assertThrows(ExecutionException.class, () -> response.get(3, TimeUnit.SECONDS)); diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/InFlightFatalAbortTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/InFlightFatalAbortTest.java index 38f86793e5..51e02f98a5 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/InFlightFatalAbortTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/InFlightFatalAbortTest.java @@ -32,7 +32,9 @@ void activeRequestObservesFatalButShutdownWaitsForBackendReturn(boolean threadDe var request = batcher.submit("START", Duration.ZERO); try { assertTrue(entered.await(3, TimeUnit.SECONDS)); - var shutdown = closing.submit(batcher::shutdown); + var shutdown = closing.submit(() -> { + batcher.shutdown(); + }); assertThrows(TimeoutException.class, () -> shutdown.get(100, TimeUnit.MILLISECONDS)); batcher.abortPending(fatal); assertFatal(request, fatal); @@ -76,7 +78,9 @@ void checkpointShutdownWaitsForAdmittedTransportAfterFatal(boolean threadDeath) .build()); try { assertTrue(entered.await(3, TimeUnit.SECONDS)); - var shutdown = closing.submit(manager::shutdown); + var shutdown = closing.submit(() -> { + manager.shutdown(); + }); assertThrows(TimeoutException.class, () -> shutdown.get(100, TimeUnit.MILLISECONDS)); signal.set(fatal); manager.abortPending(fatal); diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/IndependentDeadlineBoundaryTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/IndependentDeadlineBoundaryTest.java new file mode 100644 index 0000000000..bce33b17bd --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/IndependentDeadlineBoundaryTest.java @@ -0,0 +1,211 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.config.InvokeConfig; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.JacksonSerDes; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class IndependentDeadlineBoundaryTest { + @ParameterizedTest + @CsvSource({"false,false", "true,false", "false,true", "true,true"}) + void cancelledQueuedRootCannotHangOrRunAfterCallerReturns(boolean plugin, boolean interrupted) throws Exception { + var occupied = new CountDownLatch(1); + var release = new CountDownLatch(1); + var users = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()); + var callers = Executors.newSingleThreadExecutor(); + var caller = new AtomicReference(); + var interruptRestored = new AtomicBoolean(); + var bodies = new AtomicInteger(); + var starts = new AtomicInteger(); + var ends = new AtomicInteger(); + var client = TestUtils.createMockClient(); + var builder = DurableConfig.builder().withExecutorService(users).withDurableExecutionClient(client); + if (plugin) + builder.withPlugins(ignoredInvocation -> new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo info) { + starts.incrementAndGet(); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + } + }); + try { + users.execute(() -> { + occupied.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }); + assertTrue(occupied.await(3, TimeUnit.SECONDS)); + var config = builder.build(); + var result = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), + runtime(interrupted ? 10_000 : 1_500), + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return "unexpected"; + }, + config); + } finally { + interruptRestored.set(Thread.interrupted()); + } + }); + var limit = System.nanoTime() + TimeUnit.SECONDS.toNanos(3); + while (users.getQueue().isEmpty() && System.nanoTime() < limit) Thread.yield(); + assertEquals(1, users.getQueue().size(), "Root must actually be queued before cancellation"); + if (interrupted) caller.get().interrupt(); + var cause = assertThrows(ExecutionException.class, () -> result.get(3, TimeUnit.SECONDS)) + .getCause(); + var failure = assertInstanceOf(UnrecoverableDurableExecutionException.class, cause); + assertTrue(failure.isRetryable()); + assertEquals(interrupted, interruptRestored.get()); + release.countDown(); + users.submit(() -> {}).get(3, TimeUnit.SECONDS); + assertEquals(0, bodies.get()); + assertEquals(0, starts.get()); + assertEquals(0, ends.get()); + verify(client, never()).checkpoint(any(), any(), any()); + } finally { + release.countDown(); + callers.shutdownNow(); + users.shutdownNow(); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void caughtLateInvokeCheckpointCannotPublishFallback(boolean plugin) { + var caught = new AtomicReference(); + var serializations = new AtomicInteger(); + var fallbackSerializations = new AtomicInteger(); + var ends = new AtomicReference(); + var users = mock(ExecutorService.class); + doAnswer(call -> { + ((Runnable) call.getArgument(0)).run(); + return null; + }) + .when(users) + .execute(any()); + var client = TestUtils.createMockClient(); + var builder = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(client) + .withSerDes(new JacksonSerDes() { + @Override + public String serialize(Object value) { + if ("fallback".equals(value)) fallbackSerializations.incrementAndGet(); + return super.serialize(value); + } + }); + if (plugin) + builder.withPlugins(ignoredInvocation -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + ends.set(info); + } + }); + var payloadSerdes = new JacksonSerDes() { + @Override + public String serialize(Object value) { + serializations.incrementAndGet(); + try { + Thread.sleep(2_100); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return super.serialize(value); + } + }; + var config = builder.build(); + var returned = new AtomicReference(); + var failure = assertThrows( + UnrecoverableDurableExecutionException.class, + () -> returned.set(DurableExecutor.execute( + input(), + runtime(2_000), + TypeToken.get(String.class), + (value, context) -> { + try { + context.invoke( + "late-invoke", + "callee:1", + "payload", + String.class, + InvokeConfig.builder() + .payloadSerDes(payloadSerdes) + .build()); + } catch (Throwable error) { + caught.set(error); + } + return "fallback"; + }, + config)), + () -> "Observed output=" + returned.get() + "; caught=" + caught.get()); + assertEquals(1, serializations.get()); + assertNotNull(caught.get(), "The public handler must actually catch the late operation rejection"); + assertSame(failure, ExceptionHelper.unwrapCompletableFuture(caught.get())); + assertEquals(0, fallbackSerializations.get(), "A rejected checkpoint must not prepare a fallback success"); + assertTrue(failure.isRetryable()); + verify(client, never()).checkpoint(any(), any(), any()); + if (plugin) { + assertEquals(InvocationStatus.RETRYING, ends.get().invocationStatus()); + assertSame(failure, ends.get().executionError()); + } + } + + private static Context runtime(int millis) { + var context = mock(Context.class); + when(context.getRemainingTimeInMillis()).thenReturn(millis); + when(context.getAwsRequestId()).thenReturn("independent-request"); + return context; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/independent/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationAdmissionFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationAdmissionFailureTest.java new file mode 100644 index 0000000000..5fdf9e591d --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationAdmissionFailureTest.java @@ -0,0 +1,168 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.CheckpointUpdatedExecutionState; +import software.amazon.awssdk.services.lambda.model.ExecutionDetails; +import software.amazon.awssdk.services.lambda.model.Operation; +import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.awssdk.services.lambda.model.OperationType; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.client.DurableExecutionClient; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.DurableExecutionOutput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; + +@Timeout(15) +class InvocationAdmissionFailureTest { + @ParameterizedTest + @CsvSource({"0,false", "0,true", "1,false", "1,true", "499,false", "499,true", "500,false", "500,true"}) + void expiredRootAdmissionRetainsRetryableFailureWithoutDispatch(int remainingMillis, boolean plugins) + throws Exception { + try (var fixture = new Fixture(plugins, null)) { + var failure = + assertThrows(UnrecoverableDurableExecutionException.class, () -> fixture.invoke(remainingMillis)); + assertTrue(failure.isRetryable()); + assertEquals( + "software.amazon.lambda.durable.execution.InvocationLifecycleException", + failure.getErrorObject().errorType()); + assertEquals("Invocation deadline reached before user work", failure.getMessage()); + assertInstanceOf(TimeoutException.class, failure.getCause()); + fixture.assertNoUserWork(); + assertEquals(0, fixture.dispatches.get()); + assertFalse(fixture.users.isShutdown(), "Shared executors remain caller-owned after rejected admission"); + verifyNoInteractions(fixture.client); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void usableDeadlineStillRunsTheHandler(boolean plugins) throws Exception { + try (var fixture = new Fixture(plugins, null)) { + var response = fixture.invoke(10_000); + assertEquals(ExecutionStatus.SUCCEEDED, response.status()); + assertEquals("\"input\"", response.result()); + assertEquals(1, fixture.dispatches.get()); + assertEquals(1, fixture.bodies.get()); + assertEquals(plugins ? 1 : 0, fixture.starts.get()); + assertEquals(plugins ? 1 : 0, fixture.ends.get()); + assertFalse(fixture.users.isShutdown()); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void synchronousExecutorRejectionRetainsItsOriginalIdentity(boolean plugins) throws Exception { + var rejection = new RejectedExecutionException("executor rejected the root"); + try (var fixture = new Fixture(plugins, rejection)) { + assertSame(rejection, assertThrows(RejectedExecutionException.class, () -> fixture.invoke(10_000))); + fixture.assertNoUserWork(); + assertEquals(1, fixture.dispatches.get()); + assertFalse(fixture.users.isShutdown()); + } + } + + private static final class Fixture implements AutoCloseable { + private final AtomicInteger dispatches = new AtomicInteger(); + private final AtomicInteger bodies = new AtomicInteger(); + private final AtomicInteger factories = new AtomicInteger(); + private final AtomicInteger starts = new AtomicInteger(); + private final AtomicInteger ends = new AtomicInteger(); + private final DurableExecutionClient client = mock(DurableExecutionClient.class); + private final ThreadPoolExecutor users; + private final DurableConfig config; + + private Fixture(boolean plugins, RejectedExecutionException rejection) { + users = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()) { + @Override + public void execute(Runnable command) { + dispatches.incrementAndGet(); + if (rejection != null) throw rejection; + super.execute(command); + } + }; + var builder = DurableConfig.builder().withExecutorService(users).withDurableExecutionClient(client); + if (plugins) + builder.withPlugins(ignored -> { + factories.incrementAndGet(); + return new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo info) { + starts.incrementAndGet(); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + } + }; + }); + config = builder.build(); + } + + private DurableExecutionOutput invoke(int remainingMillis) { + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(remainingMillis); + when(runtime.getAwsRequestId()).thenReturn("admission-request"); + return DurableExecutor.execute( + input(), + runtime, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + config); + } + + private void assertNoUserWork() { + assertEquals(0, bodies.get()); + assertEquals(0, factories.get()); + assertEquals(0, starts.get()); + assertEquals(0, ends.get()); + } + + @Override + public void close() throws InterruptedException { + users.shutdownNow(); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/admission", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("admission") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationCleanupBoundaryTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationCleanupBoundaryTest.java new file mode 100644 index 0000000000..7fa150e6a2 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationCleanupBoundaryTest.java @@ -0,0 +1,515 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Duration; +import java.time.Instant; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.client.DurableExecutionClient; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.operation.BaseDurableOperation; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(15) +class InvocationCleanupBoundaryTest { + private static final String ARN = + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/cleanup/execution"; + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + @SuppressWarnings("removal") + void cleanupBudgetCannotReplaceSelectedJvmFatal(boolean threadDeath) throws Exception { + var users = Executors.newSingleThreadExecutor(); + var callers = Executors.newSingleThreadExecutor(); + var manager = manager(users, context(2_000)); + var entered = new CountDownLatch(1); + var cancelled = new CountDownLatch(1); + var release = new CountDownLatch(1); + Error fatal = threadDeath ? new ThreadDeath() : new InternalError("selected fatal"); + try { + manager.submitRootTask(() -> { + entered.countDown(); + awaitIgnoringInterrupt(release, cancelled); + return "ignored"; + }); + await(entered); + var task = manager.getActiveExecutorTasks().get(0); + var finished = callers.submit(() -> manager.finishInvocation(fatal)); + await(cancelled); + assertThrows(TimeoutException.class, () -> finished.get(3, TimeUnit.SECONDS)); + assertFalse(task.exit().isDone(), "The cleanup budget cannot stand in for the owner's actual exit"); + release.countDown(); + assertSame(fatal, finished.get(5, TimeUnit.SECONDS)); + assertTrue(task.exit().isDone()); + assertTrue(Arrays.stream(fatal.getSuppressed()).anyMatch(ExecutionManager::isLifecycleFailure)); + assertEquals(ExecutionManager.LifecycleState.CLOSED, manager.getLifecycleState()); + } finally { + release.countDown(); + stop(users); + stop(callers); + manager.close(); + } + } + + @Test + void interruptedCallerStillObservesCooperativeExitAndSettlesCheckpoints() throws Exception { + var users = Executors.newSingleThreadExecutor(); + var callers = Executors.newSingleThreadExecutor(); + var manager = manager(users, null); + var entered = new CountDownLatch(1); + var interrupted = new CountDownLatch(1); + var releaseExit = new CountDownLatch(1); + try { + manager.submitRootTask(() -> { + entered.countDown(); + try { + new CountDownLatch(1).await(); + } catch (InterruptedException expected) { + interrupted.countDown(); + } finally { + awaitIgnoringInterrupt(releaseExit); + } + return "cancelled"; + }); + await(entered); + var task = manager.getActiveExecutorTasks().get(0); + var checkpoint = manager.sendOperationUpdate(OperationUpdate.builder() + .id("queued") + .type(OperationType.STEP) + .action(OperationAction.START) + .build()); + var finished = callers.submit(() -> { + Thread.currentThread().interrupt(); + var failure = manager.finishInvocation(null); + assertTrue(Thread.interrupted(), "Cleanup restores the invocation caller's interrupt"); + return failure; + }); + await(interrupted); + assertFalse(finished.isDone(), "Interruption cannot skip the cooperative owner's actual exit"); + assertFalse(task.exit().isDone()); + releaseExit.countDown(); + var failure = + assertInstanceOf(UnrecoverableDurableExecutionException.class, finished.get(5, TimeUnit.SECONDS)); + assertInstanceOf(InterruptedException.class, failure.getCause()); + assertTrue(task.exit().isDone()); + assertTrue(checkpoint.isDone(), "Checkpoint shutdown still settles queued waiters"); + } finally { + releaseExit.countDown(); + stop(users); + stop(callers); + manager.close(); + } + } + + @Test + void recordedDeadlineFailureSurvivesCleanupWithoutAnOriginalFailure() { + var users = Executors.newSingleThreadExecutor(); + var manager = manager(users, context(700)); + try { + var recorded = manager.awaitInvocationOutcome(new CompletableFuture()) + .failure(); + assertInstanceOf(UnrecoverableDurableExecutionException.class, recorded); + assertSame(recorded, manager.finishInvocation(null)); + } finally { + users.shutdownNow(); + manager.close(); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void terminalAcknowledgmentStopsAdditionalMetadataPagesAtCutoff(boolean admitFirstPage) throws Exception { + for (boolean terminal : List.of(false, true)) { + var client = mock(DurableExecutionClient.class); + var expired = new AtomicBoolean(); + var published = new AtomicReference>(); + var inline = Operation.builder() + .id("inline") + .type(OperationType.STEP) + .status(OperationStatus.SUCCEEDED) + .build(); + var paged = Operation.builder() + .id("paged") + .type(OperationType.STEP) + .status(OperationStatus.SUCCEEDED) + .build(); + var cutoff = cutoff(); + when(client.checkpoint(any(), any(), any())).thenAnswer(call -> { + expired.set(!admitFirstPage); + return CheckpointDurableExecutionResponse.builder() + .checkpointToken("accepted-token") + .newExecutionState(CheckpointUpdatedExecutionState.builder() + .operations(inline) + .nextMarker("first") + .build()) + .build(); + }); + when(client.getExecutionState(any(), any(), any())).thenAnswer(call -> { + expired.set(true); + return GetDurableExecutionStateResponse.builder() + .operations(paged) + .nextMarker("must-not-fetch") + .build(); + }); + var config = DurableConfig.builder() + .withDurableExecutionClient(client) + .withCheckpointDelay(Duration.ZERO) + .build(); + var checkpoints = + new CheckpointManager(config, ARN, "token", published::set, () -> true, () -> {}, () -> {}, () -> { + if (expired.get()) throw cutoff; + }); + try { + var future = checkpoints.checkpoint(OperationUpdate.builder() + .id("update") + .type(terminal ? OperationType.EXECUTION : OperationType.STEP) + .action(OperationAction.SUCCEED) + .build()); + if (terminal) { + future.get(5, TimeUnit.SECONDS); + assertEquals(admitFirstPage ? List.of(inline, paged) : List.of(inline), published.get()); + assertTrue(checkpoints.isTerminalExecutionAcknowledged()); + } else { + assertSame( + cutoff, + assertThrows(ExecutionException.class, () -> future.get(5, TimeUnit.SECONDS)) + .getCause()); + assertNull(published.get()); + } + verify(client, times(admitFirstPage ? 1 : 0)).getExecutionState(any(), any(), any()); + } finally { + checkpoints.shutdown(Duration.ofSeconds(1)); + } + } + } + + @Test + void terminalAcknowledgmentDoesNotHideUnrelatedMetadataFailure() throws Exception { + var client = mock(DurableExecutionClient.class); + var problem = new IllegalStateException("metadata store failed"); + when(client.checkpoint(any(), any(), any())) + .thenReturn(CheckpointDurableExecutionResponse.builder() + .checkpointToken("accepted-token") + .newExecutionState(CheckpointUpdatedExecutionState.builder() + .nextMarker("first") + .build()) + .build()); + when(client.getExecutionState(any(), any(), any())).thenThrow(problem); + var checkpoints = new CheckpointManager( + DurableConfig.builder() + .withDurableExecutionClient(client) + .withCheckpointDelay(Duration.ZERO) + .build(), + ARN, + "token", + ignored -> {}); + try { + var future = checkpoints.checkpoint(OperationUpdate.builder() + .id("execution") + .type(OperationType.EXECUTION) + .action(OperationAction.SUCCEED) + .build()); + assertSame( + problem, + assertThrows(ExecutionException.class, () -> future.get(5, TimeUnit.SECONDS)) + .getCause()); + assertTrue(checkpoints.isTerminalExecutionAcknowledged()); + } finally { + checkpoints.shutdown(Duration.ofSeconds(1)); + } + } + + @Test + void ownerEndFinishesBeforeCallerReturnAndRestoresScopeBeforeWorkerReuse() throws Exception { + var users = Executors.newSingleThreadExecutor(); + var callers = Executors.newSingleThreadExecutor(); + var scope = new ThreadLocal(); + var prior = new ThreadLocal(); + var owner = new AtomicReference(); + var ended = new AtomicInteger(); + var endStatus = new AtomicReference(); + var entered = new CountDownLatch(1); + var release = new CountDownLatch(1); + var end = new CountDownLatch(1); + var lateBodies = new AtomicInteger(); + var cancelled = new CountDownLatch(1); + var plugin = new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + prior.set(scope.get()); + scope.set("invocation"); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(owner.get(), Thread.currentThread()); + assertEquals("invocation", scope.get()); + scope.set(prior.get()); + prior.remove(); + endStatus.set(info.invocationStatus()); + ended.incrementAndGet(); + end.countDown(); + } + }; + try { + users.submit(() -> scope.set("host")).get(5, TimeUnit.SECONDS); + var config = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withPlugins(ignoredInvocation -> plugin) + .build(); + var invocation = callers.submit(() -> DurableExecutor.execute( + input(), + context(2_000), + TypeToken.get(String.class), + (value, ctx) -> { + entered.countDown(); + awaitIgnoringInterrupt(release, cancelled); + assertThrows( + UnrecoverableDurableExecutionException.class, + () -> ctx.step("late", String.class, step -> { + lateBodies.incrementAndGet(); + return "late"; + })); + return "late-result"; + }, + config)); + await(entered); + await(cancelled); + assertThrows(TimeoutException.class, () -> invocation.get(3, TimeUnit.SECONDS)); + assertEquals(0, ended.get(), "The caller cannot run End on a different thread to force cleanup"); + release.countDown(); + assertInstanceOf( + UnrecoverableDurableExecutionException.class, + assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause()); + assertEquals(0, end.getCount(), "The owner's End must finish before the SDK caller returns"); + assertEquals(InvocationStatus.RETRYING, endStatus.get()); + assertEquals(1, ended.get()); + assertEquals(0, lateBodies.get()); + assertSame(owner.get(), users.submit(Thread::currentThread).get(5, TimeUnit.SECONDS)); + assertEquals("host", users.submit(scope::get).get(5, TimeUnit.SECONDS)); + } finally { + release.countDown(); + stop(users); + stop(callers); + } + } + + @Test + void terminalAckWaitsForAdmittedPublicationThenPropagatesEndFatal() throws Exception { + var users = Executors.newSingleThreadExecutor(); + var callers = Executors.newSingleThreadExecutor(); + var releaseChange = new CountDownLatch(1); + var changeEntered = new CountDownLatch(1); + var fatalDelivered = new CountDownLatch(1); + var fatal = new InternalError("fatal from End after terminal ACK"); + var client = mock(DurableExecutionClient.class); + when(client.checkpoint(any(), any(), any())) + .thenReturn(CheckpointDurableExecutionResponse.builder() + .checkpointToken("accepted") + .newExecutionState(CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("execution") + .type(OperationType.EXECUTION) + .status(OperationStatus.SUCCEEDED) + .build()) + .build()) + .build()); + var plugin = new DurableExecutionPlugin() { + @Override + public void onOperationChange(OperationChangeInfo info) { + changeEntered.countDown(); + awaitIgnoringInterrupt(releaseChange); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + fatalDelivered.countDown(); + throw fatal; + } + }; + try { + var config = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(client) + .withPlugins(ignoredInvocation -> plugin) + .withCheckpointDelay(Duration.ZERO) + .build(); + var invocation = callers.submit(() -> DurableExecutor.execute( + input(), + context(2_000), + TypeToken.get(String.class), + (value, ctx) -> "x".repeat(6 * 1024 * 1024 + 100), + config)); + await(changeEntered); + assertThrows(TimeoutException.class, () -> invocation.get(3, TimeUnit.SECONDS)); + assertEquals(1, fatalDelivered.getCount(), "End cannot overtake admitted checkpoint publication"); + releaseChange.countDown(); + var thrown = assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)); + assertSame( + fatal, + thrown.getCause(), + "An accepted terminal result cannot hide a fatal already delivered by End"); + assertEquals(0, fatalDelivered.getCount()); + } finally { + releaseChange.countDown(); + stop(users); + stop(callers); + } + } + + @Test + void finalEmptyTaskSnapshotClosesAdmissionBeforeAQueuedSubmitterCanEnter() throws Exception { + var users = Executors.newSingleThreadExecutor(); + var callers = Executors.newFixedThreadPool(2); + var emptySnapshot = new CountDownLatch(1); + var releaseSnapshot = new CountDownLatch(1); + var submitted = new CountDownLatch(1); + var submitter = new AtomicReference(); + var intercept = new AtomicBoolean(true); + var config = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .build(); + var manager = new ExecutionManager(input(), config, null) { + @Override + List> getActiveExecutorTasks() { + var tasks = super.getActiveExecutorTasks(); + if (tasks.isEmpty() && intercept.compareAndSet(true, false)) { + emptySnapshot.countDown(); + await(releaseSnapshot); + } + return tasks; + } + }; + var operation = mock(BaseDurableOperation.class); + when(operation.getOperationId()).thenReturn("already-registered"); + when(operation.getType()).thenReturn(OperationType.STEP); + manager.registerOperation(operation); + var bodies = new AtomicInteger(); + try { + var closing = callers.submit(manager::close); + await(emptySnapshot); + var late = callers.submit(() -> { + submitter.set(Thread.currentThread()); + submitted.countDown(); + assertThrows( + RejectedExecutionException.class, + () -> manager.submitOperationTask(operation, bodies::incrementAndGet)); + }); + await(submitted); + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(2); + var observed = submitter.get().getState(); + while (observed != Thread.State.BLOCKED && System.nanoTime() < deadline) { + Thread.onSpinWait(); + observed = submitter.get().getState(); + } + // Keep the observed contention, rather than race a second thread-state snapshot. + assertEquals(Thread.State.BLOCKED, observed); + releaseSnapshot.countDown(); + closing.get(5, TimeUnit.SECONDS); + late.get(5, TimeUnit.SECONDS); + assertEquals(0, bodies.get()); + } finally { + releaseSnapshot.countDown(); + stop(users); + stop(callers); + manager.close(); + } + } + + private static ExecutionManager manager(ExecutorService users, Context context) { + var result = new ExecutionManager( + input(), + DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withCheckpointDelay(Duration.ofHours(1)) + .build(), + context); + result.registerActiveThread(null); + return result; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + ARN, + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("execution") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.now()) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } + + private static Context context(int millis) { + var context = mock(Context.class); + when(context.getAwsRequestId()).thenReturn("cleanup-request"); + when(context.getRemainingTimeInMillis()).thenReturn(millis); + return context; + } + + private static UnrecoverableDurableExecutionException cutoff() { + return new UnrecoverableDurableExecutionException( + ErrorObject.builder() + .errorType("software.amazon.lambda.durable.execution.InvocationLifecycleException") + .errorMessage("controlled deadline cutoff") + .build(), + true, + new TimeoutException()); + } + + private static void await(CountDownLatch latch) { + try { + assertTrue(latch.await(5, TimeUnit.SECONDS), "Fixture gate must be released"); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new AssertionError(interrupted); + } + } + + private static void awaitIgnoringInterrupt(CountDownLatch latch) { + awaitIgnoringInterrupt(latch, new CountDownLatch(0)); + } + + private static void awaitIgnoringInterrupt(CountDownLatch latch, CountDownLatch interrupted) { + while (latch.getCount() > 0) { + try { + latch.await(); + } catch (InterruptedException ignored) { + interrupted.countDown(); + // This bounded fixture is released in finally; it models an uncooperative customer callback. + } + } + } + + private static void stop(ExecutorService executor) throws Exception { + executor.shutdownNow(); + assertTrue(executor.awaitTermination(5, TimeUnit.SECONDS)); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationDeadlineReviewRegressionTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationDeadlineReviewRegressionTest.java new file mode 100644 index 0000000000..6769e45cc8 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationDeadlineReviewRegressionTest.java @@ -0,0 +1,265 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; +import static software.amazon.lambda.durable.TypeToken.get; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Duration; +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationStatus; +import software.amazon.lambda.durable.serde.JacksonSerDes; + +@Timeout(20) +class InvocationDeadlineReviewRegressionTest { + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void caughtDeadlineAdmissionCannotPublishSuccessfulFallback(boolean plugins) { + var caught = new AtomicReference(); + var bodyEntered = new AtomicBoolean(); + var effects = new AtomicInteger(); + var fallbackSerializations = new AtomicInteger(); + var end = new AtomicReference(); + var config = config( + inlineExecutor(), + new JacksonSerDes() { + @Override + public String serialize(Object value) { + if ("fallback".equals(value)) fallbackSerializations.incrementAndGet(); + return super.serialize(value); + } + }, + plugins, + end); + var failure = assertThrows( + UnrecoverableDurableExecutionException.class, + () -> DurableExecutor.execute( + input(), + runtime(2_000), + get(String.class), + (value, context) -> { + bodyEntered.set(true); + try { + TimeUnit.MILLISECONDS.sleep(1_600); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new AssertionError(interrupted); + } + try { + context.step("too-late", String.class, step -> { + effects.incrementAndGet(); + return "side-effect"; + }); + } catch (UnrecoverableDurableExecutionException interrupted) { + caught.set(interrupted); + } + return "fallback"; + }, + config)); + assertTrue(bodyEntered.get(), "The handler must reach the caught-admission path"); + assertSame(caught.get(), failure, "Swallowing a deadline rejection must not erase its selected failure"); + assertTrue(failure.isRetryable()); + assertEquals(0, effects.get()); + assertEquals(0, fallbackSerializations.get(), "An expired invocation must not prepare a fallback success"); + if (plugins) { + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(failure, end.get().executionError()); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void inlineSuccessBeforeTheCutoffIsStillAllowed(boolean plugins) { + var end = new AtomicReference(); + var config = config(inlineExecutor(), new JacksonSerDes(), plugins, end); + var output = DurableExecutor.execute( + input(), runtime(10_000), get(String.class), (value, context) -> "result", config); + assertEquals(ExecutionStatus.SUCCEEDED, output.status()); + assertEquals("\"result\"", output.result()); + if (plugins) assertEquals(InvocationStatus.SUCCEEDED, end.get().invocationStatus()); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void anAlreadySelectedSuspensionSurvivesLaterCaughtAdmission(boolean plugins) { + var end = new AtomicReference(); + var effects = new AtomicInteger(); + var config = config(inlineExecutor(), new JacksonSerDes(), plugins, end); + var output = DurableExecutor.execute( + input(), + runtime(2_000), + get(String.class), + (value, context) -> { + try { + context.wait("pause", Duration.ofSeconds(5)); + throw new AssertionError("The wait should select suspension"); + } catch (SuspendExecutionException suspended) { + try { + TimeUnit.MILLISECONDS.sleep(1_600); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new AssertionError(interrupted); + } + try { + context.step("too-late", String.class, step -> { + effects.incrementAndGet(); + return "side-effect"; + }); + } catch (RuntimeException stopped) { + // A rejected operation must not replace the suspension already selected by the durable + // wait. + } + return "fallback"; + } + }, + config); + assertEquals(ExecutionStatus.PENDING, output.status()); + assertEquals(0, effects.get()); + if (plugins) assertEquals(InvocationStatus.PENDING, end.get().invocationStatus()); + } + + @ParameterizedTest + @CsvSource({"false,true", "true,true", "false,false", "true,false"}) + void cancellationDuringSerializationRetainsFatalPriority(boolean plugins, boolean fatal) throws Exception { + var serializationEntered = new CountDownLatch(1); + var serializerInterrupted = new CountDownLatch(1); + var release = new CountDownLatch(1); + var uncaught = new CountDownLatch(1); + var uncaughtFailure = new AtomicReference(); + var runtimeThread = new AtomicReference(); + var end = new AtomicReference(); + Throwable serializationFailure = fatal + ? new InternalError("serializer fatal") + : new IllegalStateException("ordinary serializer failure"); + var workers = Executors.newSingleThreadExecutor(action -> { + var worker = new Thread(action, "serialization-owner"); + worker.setUncaughtExceptionHandler((thread, error) -> { + uncaughtFailure.set(error); + uncaught.countDown(); + }); + return worker; + }); + var callers = Executors.newSingleThreadExecutor(); + var serializer = new JacksonSerDes() { + @Override + public String serialize(Object value) { + if ("result".equals(value)) { + serializationEntered.countDown(); + try { + release.await(); + } catch (InterruptedException interrupted) { + serializerInterrupted.countDown(); + Thread.currentThread().interrupt(); + if (serializationFailure instanceof Error error) throw error; + throw (RuntimeException) serializationFailure; + } + } + return super.serialize(value); + } + }; + try { + var config = config(workers, serializer, plugins, end); + var invocation = callers.submit(() -> { + runtimeThread.set(Thread.currentThread()); + return DurableExecutor.execute( + input(), runtime(10_000), get(String.class), (value, context) -> "result", config); + }); + assertTrue(serializationEntered.await(5, TimeUnit.SECONDS)); + runtimeThread.get().interrupt(); + assertTrue(serializerInterrupted.await(5, TimeUnit.SECONDS), "Cleanup must physically interrupt the owner"); + var actual = assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause(); + if (fatal) { + assertSame(serializationFailure, actual, "An available JVM fatal must beat lifecycle cancellation"); + assertTrue(uncaught.await(5, TimeUnit.SECONDS), "The actual worker must not absorb a JVM fatal"); + assertSame(serializationFailure, uncaughtFailure.get()); + } else { + assertInstanceOf(UnrecoverableDurableExecutionException.class, actual); + assertNotSame( + serializationFailure, actual, "An ordinary late error must not replace caller interruption"); + } + if (plugins) { + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(serializationFailure, end.get().executionError()); + } + } finally { + release.countDown(); + callers.shutdownNow(); + workers.shutdownNow(); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(workers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static DurableConfig config( + ExecutorService executor, + JacksonSerDes serializer, + boolean plugins, + AtomicReference end) { + var builder = DurableConfig.builder() + .withExecutorService(executor) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withCheckpointDelay(Duration.ZERO) + .withSerDes(serializer); + if (plugins) + builder.withPlugins(ignoredInfo -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + end.set(info); + } + }); + return builder.build(); + } + + private static ExecutorService inlineExecutor() { + var executor = mock(ExecutorService.class); + doAnswer(call -> { + ((Runnable) call.getArgument(0)).run(); + return null; + }) + .when(executor) + .execute(any()); + return executor; + } + + private static Context runtime(int millis) { + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(millis); + when(runtime.getAwsRequestId()).thenReturn("runtime-request"); + return runtime; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/review/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationPreparationBoundaryTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationPreparationBoundaryTest.java new file mode 100644 index 0000000000..3602bcb9b5 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/InvocationPreparationBoundaryTest.java @@ -0,0 +1,240 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.JacksonSerDes; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(20) +class InvocationPreparationBoundaryTest { + @SuppressWarnings("removal") + @ParameterizedTest + @CsvSource({ + "false,true,false,false", "true,true,false,false", + "false,true,true,false", "true,true,true,false", + "false,false,false,false", "true,false,false,false", + "false,false,true,false", "true,false,true,false", + "false,true,false,true", "true,true,false,true" + }) + void capturedInputFatalSurvivesAdmissionCutoff(boolean plugins, boolean expired, boolean death, boolean wrapped) { + Error fatal = death ? new ThreadDeath() : new InternalError("input deserializer fatal"); + Throwable inputFailure = wrapped ? new CompletionException(fatal) : fatal; + var reads = new AtomicInteger(); + var bodies = new AtomicInteger(); + var starts = new AtomicInteger(); + var ends = new AtomicInteger(); + var end = new AtomicReference(); + var executor = mock(ExecutorService.class); + doAnswer(call -> { + ((Runnable) call.getArgument(0)).run(); + return null; + }) + .when(executor) + .execute(any()); + var serializer = new JacksonSerDes() { + @Override + public T deserialize(String data, TypeToken type) { + reads.incrementAndGet(); + if (expired) { + try { + assertFalse(new CountDownLatch(1).await(1600, TimeUnit.MILLISECONDS)); + } catch (InterruptedException interrupted) { + throw new AssertionError(interrupted); + } + } + ExceptionHelper.sneakyThrow(inputFailure); + return null; + } + }; + var client = TestUtils.createMockClient(); + var builder = DurableConfig.builder() + .withExecutorService(executor) + .withDurableExecutionClient(client) + .withSerDes(serializer); + if (plugins) + builder.withPlugins(info -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + starts.incrementAndGet(); + assertNull(info.executionInput()); + } + + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + end.set(info); + } + }); + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), + runtime(expired ? 2000 : 10_000), + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + builder.build())); + assertSame(fatal, observed, "A captured input JVM fatal must outrank a later admission rejection"); + assertEquals(1, reads.get()); + assertEquals(0, bodies.get()); + assertEquals(plugins && !expired ? 1 : 0, starts.get(), "Expired input cannot start new plugin work"); + assertEquals(starts.get(), ends.get()); + if (end.get() != null) { + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(fatal, end.get().executionError()); + } + verify(client, never()).checkpoint(any(), any(), any()); + } + + @ParameterizedTest + @CsvSource({"false,false", "false,true", "true,false", "true,true"}) + void terminalEndReflectsCancellationDuringNormalSerialization(boolean handlerFails, boolean cancel) + throws Exception { + var serializationEntered = new CountDownLatch(1); + var serializationInterrupted = new CountDownLatch(1); + var releaseSerialization = new CountDownLatch(1); + var endCount = new AtomicInteger(); + var end = new AtomicReference(); + var owner = new AtomicReference(); + var caller = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var applicationFailure = new IllegalStateException("handler failed"); + var workers = Executors.newSingleThreadExecutor(); + var callers = Executors.newSingleThreadExecutor(); + var serializer = new JacksonSerDes() { + @Override + public String serialize(Object value) { + if ("result".equals(value) || value == applicationFailure) { + assertSame(owner.get(), Thread.currentThread()); + serializationEntered.countDown(); + awaitIgnoringInterruptions(releaseSerialization, serializationInterrupted); + } + return super.serialize(value); + } + }; + var client = TestUtils.createMockClient(); + var config = DurableConfig.builder() + .withExecutorService(workers) + .withDurableExecutionClient(client) + .withSerDes(serializer) + .withPlugins(info -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(owner.get(), Thread.currentThread()); + endCount.incrementAndGet(); + end.set(info); + } + }) + .build(); + try { + var response = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), + runtime(10_000), + TypeToken.get(String.class), + (value, context) -> { + if (handlerFails) throw applicationFailure; + return "result"; + }, + config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertTrue(serializationEntered.await(5, TimeUnit.SECONDS)); + if (cancel) { + caller.get().interrupt(); + assertTrue(serializationInterrupted.await(5, TimeUnit.SECONDS)); + assertThrows(TimeoutException.class, () -> response.get(100, TimeUnit.MILLISECONDS)); + assertEquals(0, endCount.get(), "End must await the actual serializer owner"); + } + releaseSerialization.countDown(); + if (cancel) { + var failure = assertInstanceOf( + UnrecoverableDurableExecutionException.class, + assertThrows(ExecutionException.class, () -> response.get(5, TimeUnit.SECONDS)) + .getCause()); + assertTrue(failure.isRetryable()); + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(failure, end.get().executionError()); + assertNull(end.get().executionResult()); + } else { + assertEquals( + handlerFails ? ExecutionStatus.FAILED : ExecutionStatus.SUCCEEDED, + response.get(5, TimeUnit.SECONDS).status()); + assertEquals( + handlerFails ? InvocationStatus.FAILED : InvocationStatus.SUCCEEDED, + end.get().invocationStatus()); + assertSame(handlerFails ? applicationFailure : null, end.get().executionError()); + } + assertEquals(1, endCount.get()); + assertEquals(cancel, callerInterrupted.get()); + verify(client, never()).checkpoint(any(), any(), any()); + } finally { + releaseSerialization.countDown(); + workers.shutdownNow(); + callers.shutdownNow(); + assertTrue(workers.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static void awaitIgnoringInterruptions(CountDownLatch release, CountDownLatch interrupted) { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (true) { + try { + assertTrue(release.await(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + return; + } catch (InterruptedException ignored) { + interrupted.countDown(); + } + } + } + + private static Context runtime(int millis) { + var context = mock(Context.class); + when(context.getRemainingTimeInMillis()).thenReturn(millis); + when(context.getAwsRequestId()).thenReturn("preparation-request"); + return context; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/review/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/LmiLifecycleRegressionTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/LmiLifecycleRegressionTest.java index 6d53fc0184..5b5d816df9 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/execution/LmiLifecycleRegressionTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/LmiLifecycleRegressionTest.java @@ -10,17 +10,15 @@ import com.amazonaws.services.lambda.runtime.Context; import java.time.Duration; import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; -import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; import java.util.stream.IntStream; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.condition.EnabledIfSystemProperty; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.ValueSource; import software.amazon.awssdk.services.lambda.model.CheckpointUpdatedExecutionState; import software.amazon.awssdk.services.lambda.model.ExecutionDetails; import software.amazon.awssdk.services.lambda.model.Operation; @@ -102,14 +100,20 @@ void deadlineMustCancelAnInvocationOwnedTaskAndBoundCleanup() throws Exception { } } - @ParameterizedTest - @ValueSource(booleans = {false, true}) - void sharedFixedExecutorMustProgress(boolean nested) throws Exception { + @Test + void invocationExecutorFactoryIsolatesConcurrentRoots() throws Exception { var entered = new CountDownLatch(2); - var users = (ThreadPoolExecutor) Executors.newFixedThreadPool(2); + var invocationExecutors = new CopyOnWriteArrayList(); var runtime = Executors.newFixedThreadPool(2); try { - var config = config(users); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> { + var executor = Executors.newFixedThreadPool(2); + invocationExecutors.add(executor); + return executor; + }) + .build(); var responses = IntStream.range(0, 2) .mapToObj(index -> runtime.submit(() -> DurableExecutor.execute( input("fixed-" + index), @@ -118,16 +122,6 @@ void sharedFixedExecutorMustProgress(boolean nested) throws Exception { (value, ctx) -> { entered.countDown(); await(entered); - if (nested) { - return ctx.runInChildContext( - "child", - String.class, - child -> child.runInChildContext( - "grandchild", - String.class, - grandchild -> - grandchild.step("step", String.class, step -> value))); - } return ctx.step("step", String.class, step -> value); }, config))) @@ -142,10 +136,12 @@ void sharedFixedExecutorMustProgress(boolean nested) throws Exception { .status(), "Blocking orchestration must not starve the work it awaits"); } + assertEquals(2, invocationExecutors.size()); + assertTrue(invocationExecutors.stream().allMatch(ExecutorService::isTerminated)); } finally { - users.setMaximumPoolSize(16); - users.setCorePoolSize(16); // Escape only after the assertion, so the test process cannot hang. - stop(users, runtime); + invocationExecutors.forEach(ExecutorService::shutdownNow); + runtime.shutdownNow(); + assertTrue(runtime.awaitTermination(10, TimeUnit.SECONDS)); } } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/MdcFinalizationAdjacentTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/MdcFinalizationAdjacentTest.java new file mode 100644 index 0000000000..b2fdc34c3c --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/MdcFinalizationAdjacentTest.java @@ -0,0 +1,196 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Proxy; +import java.time.Instant; +import java.util.Arrays; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.slf4j.MDC; +import org.slf4j.spi.MDCAdapter; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestContext; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(15) +class MdcFinalizationAdjacentTest { + @ParameterizedTest + @CsvSource({"true,false", "false,true", "true,true"}) + void restorationFatalRetainsTaskAndOwnedDiagnosticsOnce(boolean fallbackClear, boolean retrying) throws Exception { + var original = MDC.getMDCAdapter(); + var before = MDC.getCopyOfContextMap(); + var first = new InternalError("MDC restoration fatal"); + var cleanup = new InternalError("owned shutdown fatal"); + var ordinaryRestore = new IllegalStateException("ordinary restore failure"); + var retry = new UnrecoverableDurableExecutionException( + ErrorObject.builder() + .errorType("test.retry") + .errorMessage("retryable handler outcome") + .build(), + true); + var owner = new AtomicReference(); + var workerError = new AtomicReference(); + var escaped = new CountDownLatch(1); + var outerInjected = new AtomicBoolean(); + var fallbackInjected = new AtomicBoolean(); + var ends = new AtomicInteger(); + var end = new AtomicReference(); + var shutdowns = new AtomicInteger(); + var users = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(), task -> { + var thread = new Thread(task, "adjacent-mdc-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerError.set(failure); + escaped.countDown(); + }); + return thread; + }) { + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + super.shutdown(); + throw cleanup; + } + }; + var callers = Executors.newSingleThreadExecutor(); + var proxy = (MDCAdapter) Proxy.newProxyInstance( + MDCAdapter.class.getClassLoader(), new Class[] {MDCAdapter.class}, (unused, method, arguments) -> { + if (Thread.currentThread() == owner.get()) { + if ((method.getName().equals("clear") + || method.getName().equals("setContextMap")) + && outerRestore() + && outerInjected.compareAndSet(false, true)) + throw fallbackClear ? ordinaryRestore : first; + if (fallbackClear + && method.getName().equals("clear") + && inMethod("clearMdcAfterOrdinaryFailure") + && fallbackInjected.compareAndSet(false, true)) throw first; + } + try { + return method.invoke(original, arguments); + } catch (InvocationTargetException failure) { + throw failure.getCause(); + } + }); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> users) + .withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(owner.get(), Thread.currentThread()); + end.set(info); + ends.incrementAndGet(); + } + }) + .build(); + try { + replaceAdapter(proxy); + var response = callers.submit(() -> DurableExecutor.execute( + input(), + new TestContext("mdc-adjacent"), + TypeToken.get(String.class), + (value, context) -> { + if (retrying) throw retry; + return "result"; + }, + config)); + var observed = assertThrows(ExecutionException.class, () -> response.get(5, TimeUnit.SECONDS)) + .getCause(); + assertSame(first, observed); + assertTrue(escaped.await(5, TimeUnit.SECONDS)); + assertSame(first, workerError.get()); + assertTrue(outerInjected.get()); + assertEquals(fallbackClear, fallbackInjected.get()); + assertEquals(1, ends.get()); + assertEquals( + retrying ? InvocationStatus.RETRYING : InvocationStatus.SUCCEEDED, + end.get().invocationStatus()); + assertSame( + retrying ? retry : null, + end.get().executionError(), + "Later MDC failure must not rewrite or re-fire End"); + assertEquals( + 1, + Arrays.stream(first.getSuppressed()) + .filter(error -> error == cleanup) + .count(), + "Retain later owned fatal once"); + if (retrying) + assertEquals( + 1, + Arrays.stream(first.getSuppressed()) + .filter(error -> error == retry) + .count(), + "The pre-existing retry diagnostic must not be duplicated during finalization"); + if (fallbackClear) + assertEquals( + 1, + Arrays.stream(first.getSuppressed()) + .filter(error -> error == ordinaryRestore) + .count()); + assertEquals(1, shutdowns.get()); + } finally { + try { + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + } finally { + replaceAdapter(original); + if (before == null) MDC.clear(); + else MDC.setContextMap(before); + } + } + } + + private static boolean outerRestore() { + return Arrays.stream(Thread.currentThread().getStackTrace()) + .anyMatch(frame -> frame.getClassName().equals(DurableExecutor.class.getName()) + && frame.getMethodName().startsWith("lambda$restoreMdcOnClose")); + } + + private static boolean inMethod(String method) { + return Arrays.stream(Thread.currentThread().getStackTrace()) + .anyMatch(frame -> frame.getClassName().equals(DurableExecutor.class.getName()) + && frame.getMethodName().equals(method)); + } + + private static void replaceAdapter(MDCAdapter adapter) throws Exception { + var method = MDC.class.getDeclaredMethod("setMDCAdapter", MDCAdapter.class); + method.setAccessible(true); + method.invoke(null, adapter); + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/test", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("test") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedCallbackFailureAggregationTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedCallbackFailureAggregationTest.java new file mode 100644 index 0000000000..456a636776 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedCallbackFailureAggregationTest.java @@ -0,0 +1,303 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.Set; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestContext; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.JacksonSerDes; +import software.amazon.lambda.durable.serde.SerDes; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedCallbackFailureAggregationTest { + enum Stage { + SHUTDOWN, + AWAIT, + STATE, + FORCED, + INITIAL, + SETUP + } + + enum Wrapper { + RAW, + COMPLETION, + EXECUTION, + CUSTOM + } + + record Scenario( + Stage stage, + Wrapper wrapper, + boolean recorded, + boolean repeatAtState, + boolean wrappedState, + boolean death) {} + + static Stream cases() { + var triple = Stream.of(Wrapper.values()) + .flatMap(wrapper -> Stream.of(false, true) + .flatMap(wrappedState -> Stream.of(false, true) + .map(death -> new Scenario(Stage.SHUTDOWN, wrapper, true, true, wrappedState, death)))); + var wrappers = Stream.of(Stage.values()) + .filter(stage -> stage != Stage.SETUP) + .flatMap(stage -> Stream.of(Wrapper.COMPLETION, Wrapper.EXECUTION) + .flatMap(wrapper -> (stage == Stage.INITIAL ? Stream.of(false) : Stream.of(false, true)) + .map(recorded -> new Scenario(stage, wrapper, recorded, false, false, false)))); + return Stream.concat( + Stream.concat(triple, wrappers), + Stream.of( + new Scenario(Stage.SHUTDOWN, Wrapper.RAW, false, true, false, false), + new Scenario(Stage.INITIAL, Wrapper.CUSTOM, false, false, false, false), + new Scenario(Stage.SETUP, Wrapper.COMPLETION, true, true, true, false))); + } + + @SuppressWarnings("removal") + @ParameterizedTest + @MethodSource("cases") + void callbackFailuresKeepTheFirstFatalAndAllNonAliasDiagnostics(Scenario scenario) throws Exception { + Error first = scenario.death() ? new ThreadDeath() : new InternalError("serializer fatal"); + var later = new InternalError("executor callback fatal"); + var diagnostic = new IllegalArgumentException("wrapper cleanup diagnostic"); + var nestedDiagnostic = new IllegalStateException("nested wrapper diagnostic"); + var queryDiagnostic = new IllegalArgumentException("query wrapper diagnostic"); + var ordinaryShutdown = new IllegalStateException("graceful shutdown failed before delegation"); + var causeReads = new AtomicInteger(); + Throwable callback = switch (scenario.wrapper()) { + case RAW -> later; + case COMPLETION -> new CompletionException(later); + case EXECUTION -> new ExecutionException(later); + case CUSTOM -> + new CompletionException(later) { + @Override + public synchronized Throwable getCause() { + assertEquals(1, causeReads.incrementAndGet(), "Read a custom wrapper cause only once"); + return super.getCause(); + } + }; + }; + if (callback != later) { + callback.addSuppressed(diagnostic); + var alias = new CompletionException(later); + alias.addSuppressed(nestedDiagnostic); + callback.addSuppressed(alias); + if (scenario.recorded()) callback.addSuppressed(first); + } + Throwable repeated = scenario.wrappedState() ? new CompletionException(first) : first; + if (repeated != first) repeated.addSuppressed(queryDiagnostic); + var started = new AtomicBoolean(); + var inject = new AtomicBoolean(true); + var firstAwait = new AtomicBoolean(true); + var firstInitial = new AtomicBoolean(true); + var shutdowns = new AtomicInteger(); + var forced = new AtomicInteger(); + var stateFailures = new AtomicInteger(); + var bodies = new AtomicInteger(); + var serializations = new AtomicInteger(); + var manager = new AtomicReference(); + var fatalAtShutdown = new AtomicReference(); + var ends = new AtomicInteger(); + var end = new AtomicReference(); + var workerFatal = new AtomicReference(); + var workerEscaped = new CountDownLatch(1); + var pool = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(), task -> { + var thread = new Thread(task, "callback-failure-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerFatal.set(failure); + workerEscaped.countDown(); + }); + return thread; + }) { + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + started.set(true); + if (manager.get() != null) + fatalAtShutdown.set(manager.get().getInvocationFatal()); + if (inject.get() && scenario.stage() == Stage.FORCED) throw ordinaryShutdown; + super.shutdown(); + if (inject.get() && (scenario.stage() == Stage.SHUTDOWN || scenario.stage() == Stage.SETUP)) + ExceptionHelper.sneakyThrow(callback); + } + + @Override + public List shutdownNow() { + var tasks = super.shutdownNow(); + if (inject.get()) { + forced.incrementAndGet(); + if (scenario.stage() == Stage.FORCED) ExceptionHelper.sneakyThrow(callback); + } + return tasks; + } + + @Override + public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException { + if (inject.get() && scenario.stage() == Stage.AWAIT && firstAwait.compareAndSet(true, false)) + ExceptionHelper.sneakyThrow(callback); + return super.awaitTermination(timeout, unit); + } + + @Override + public boolean isShutdown() { + if (inject.get()) { + if (scenario.stage() == Stage.INITIAL && firstInitial.compareAndSet(true, false)) { + stateFailures.incrementAndGet(); + ExceptionHelper.sneakyThrow(callback); + } + if (started.get() && (scenario.repeatAtState() || scenario.stage() == Stage.STATE)) { + stateFailures.incrementAndGet(); + ExceptionHelper.sneakyThrow(scenario.repeatAtState() ? repeated : callback); + } + } + return super.isShutdown(); + } + }; + pool.prestartAllCoreThreads(); + var serde = new JacksonSerDes(); + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> pool) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withSerDes(new SerDes() { + public String serialize(Object value) { + serializations.incrementAndGet(); + if (scenario.recorded()) throw first; + return serde.serialize(value); + } + + public T deserialize(String value, TypeToken type) { + return serde.deserialize(value, type); + } + }) + .withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + end.set(info); + } + }) + .build(); + var requestReads = new AtomicInteger(); + var runtime = scenario.stage() == Stage.SETUP + ? new TestContext("setup-callback") { + @Override + public String getAwsRequestId() { + if (requestReads.incrementAndGet() == 2) throw first; + return super.getAwsRequestId(); + } + } + : null; + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), + runtime, + TypeToken.get(String.class), + (value, context) -> { + manager.set(((DurableContextImpl) context).getExecutionManager()); + bodies.incrementAndGet(); + return "output"; + }, + config)); + assertSame(scenario.recorded() ? first : later, observed); + if (scenario.recorded() && scenario.stage() != Stage.SETUP) { + assertTrue(workerEscaped.await(3, TimeUnit.SECONDS)); + assertSame(first, workerFatal.get()); + assertSame(first, fatalAtShutdown.get()); + } + assertTrue(contains(observed, later), "Retain the later executor callback fatal"); + if (scenario.wrapper() != Wrapper.RAW) { + assertTrue(contains(observed, diagnostic), "Retain discarded wrapper diagnostics"); + assertTrue(contains(observed, nestedDiagnostic), "Retain diagnostics on wrapper aliases"); + } + if (scenario.wrappedState()) assertTrue(contains(observed, queryDiagnostic)); + if (scenario.stage() == Stage.FORCED) assertTrue(contains(observed, ordinaryShutdown)); + if (scenario.repeatAtState()) { + assertEquals(1, stateFailures.get()); + assertTrue(contains(observed, first)); + } + assertEquals(1, shutdowns.get()); + if (scenario.stage() == Stage.INITIAL || scenario.stage() == Stage.SETUP) { + assertEquals(0, bodies.get()); + assertEquals(0, serializations.get()); + assertEquals(0, ends.get()); + } else { + assertEquals(1, bodies.get()); + assertEquals(1, serializations.get()); + assertEquals(1, ends.get()); + assertEquals( + scenario.recorded() ? InvocationStatus.RETRYING : InvocationStatus.SUCCEEDED, + end.get().invocationStatus()); + assertSame(scenario.recorded() ? first : null, end.get().executionError()); + } + if (scenario.wrapper() == Wrapper.CUSTOM) assertEquals(1, causeReads.get()); + if (scenario.stage() == Stage.SETUP) { + assertEquals(2, requestReads.get()); + assertNull(fatalAtShutdown.get(), "Setup fatal is carried as originalFailure, without plugin state"); + } + assertAcyclic(observed, Collections.newSetFromMap(new IdentityHashMap())); + } finally { + inject.set(false); + pool.shutdownNow(); + assertTrue(pool.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static boolean contains(Throwable root, Throwable target) { + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var next = pending.remove(); + if (next == target) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) pending.add(next.getCause()); + Collections.addAll(pending, next.getSuppressed()); + } + return false; + } + + private static void assertAcyclic(Throwable failure, Set path) { + assertTrue(path.add(failure), "Executor callback diagnostics must not point back to the first fatal"); + if (failure.getCause() != null) assertAcyclic(failure.getCause(), path); + for (var diagnostic : failure.getSuppressed()) assertAcyclic(diagnostic, path); + path.remove(failure); + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/callback-fatal", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("callback-fatal") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedCompletionAdjacentTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedCompletionAdjacentTest.java new file mode 100644 index 0000000000..77b56e4c19 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedCompletionAdjacentTest.java @@ -0,0 +1,231 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Duration; +import java.time.Instant; +import java.util.*; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.client.DurableExecutionClient; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.JacksonSerDes; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedCompletionAdjacentTest { + @Test + void acknowledgedDeliveryFailureIsRetainedByLaterOwnedFatal() throws Exception { + run(true, true); + } + + @Test + void acknowledgedDeliveryFailureStillOutranksOrdinaryOwnedFailure() throws Exception { + run(true, false); + } + + @Test + void callerCutoffRetainsLaterSerializerFailureAndOwnedFailure() throws Exception { + run(false, false); + } + + @Test + void callerCutoffDoesNotDuplicateTheSerializerUsedAsShutdownFailure() throws Exception { + run(false, false, true); + } + + private static void run(boolean acknowledged, boolean cleanupFatal) throws Exception { + run(acknowledged, cleanupFatal, false); + } + + private static void run(boolean acknowledged, boolean cleanupFatal, boolean sameError) throws Exception { + var delivery = new IllegalStateException("late delivery failure"); + Throwable shutdown = sameError + ? delivery + : cleanupFatal + ? new InternalError("later owned shutdown fatal") + : new IllegalStateException("later owned shutdown failure"); + var injected = new AtomicBoolean(true); + var shutdownCalls = new AtomicInteger(); + var users = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()) { + @Override + public void shutdown() { + shutdownCalls.incrementAndGet(); + super.shutdown(); + if (injected.get()) ExceptionHelper.sneakyThrow(shutdown); + } + }; + var callers = Executors.newSingleThreadExecutor(); + var caller = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var serializationEntered = new CountDownLatch(1); + var ownerInterrupted = new CountDownLatch(1); + var releaseSerialization = new CountDownLatch(1); + var owner = new AtomicReference(); + var ends = new AtomicInteger(); + var end = new AtomicReference(); + var updates = new AtomicReference>(); + DurableExecutionClient client; + if (acknowledged) { + client = mock(DurableExecutionClient.class); + when(client.checkpoint(any(), any(), any())).thenAnswer(call -> { + updates.set(List.copyOf(call.getArgument(2))); + return CheckpointDurableExecutionResponse.builder() + .checkpointToken("accepted-token") + .newExecutionState(CheckpointUpdatedExecutionState.builder() + .nextMarker("metadata") + .build()) + .build(); + }); + when(client.getExecutionState(any(), any(), any())).thenThrow(delivery); + } else { + client = TestUtils.createMockClient(); + } + var result = acknowledged ? "x".repeat(6 * 1024 * 1024 + 100) : "result"; + var serializer = new JacksonSerDes() { + @Override + public String serialize(Object value) { + if (!acknowledged && result.equals(value)) { + serializationEntered.countDown(); + boolean interrupted = false; + try { + while (true) { + try { + assertTrue(releaseSerialization.await(5, TimeUnit.SECONDS)); + break; + } catch (InterruptedException observed) { + interrupted = true; + ownerInterrupted.countDown(); + } + } + } finally { + if (interrupted) Thread.currentThread().interrupt(); + } + throw delivery; + } + return super.serialize(value); + } + }; + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> users) + .withCheckpointDelay(Duration.ZERO) + .withDurableExecutionClient(client) + .withSerDes(serializer) + .withPlugins(ignored -> new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(owner.get(), Thread.currentThread()); + ends.incrementAndGet(); + end.set(info); + } + }) + .build(); + try { + var invocation = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), runtime(), TypeToken.get(String.class), (value, context) -> result, config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + if (!acknowledged) { + assertTrue(serializationEntered.await(5, TimeUnit.SECONDS)); + caller.get().interrupt(); + assertTrue(ownerInterrupted.await(5, TimeUnit.SECONDS)); + assertFalse(invocation.isDone(), "Caller must still wait for the actual serializer owner"); + releaseSerialization.countDown(); + } + var observed = assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause(); + assertAcyclic(observed, Collections.newSetFromMap(new IdentityHashMap<>())); + if (cleanupFatal) { + assertSame(shutdown, observed); + assertEquals(1, countIdentity(observed, delivery)); + } else if (acknowledged) { + assertSame(delivery, observed, "ACK must not erase the unrelated metadata failure"); + } else { + assertTrue( + ExecutionManager.isLifecycleFailure(observed), "The caller's recorded cutoff remains primary"); + assertEquals(1, countIdentity(observed, delivery)); + assertEquals(1, countIdentity(observed, shutdown)); + } + assertEquals(!acknowledged, callerInterrupted.get()); + assertEquals(1, ends.get()); + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(delivery, end.get().executionError()); + assertEquals(1, shutdownCalls.get()); + if (acknowledged) { + verify(client, times(1)).checkpoint(any(), any(), any()); + verify(client, times(1)).getExecutionState(any(), eq("accepted-token"), eq("metadata")); + assertEquals(1, updates.get().size()); + assertEquals(OperationAction.SUCCEED, updates.get().get(0).action()); + assertEquals('"' + result + '"', updates.get().get(0).payload()); + } + } finally { + releaseSerialization.countDown(); + injected.set(false); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static int countIdentity(Throwable node, Throwable expected) { + if (node == null) return 0; + int count = node == expected ? 1 : 0; + count += countIdentity(node.getCause(), expected); + for (var suppressed : node.getSuppressed()) count += countIdentity(suppressed, expected); + return count; + } + + private static void assertAcyclic(Throwable node, Set path) { + if (node == null) return; + assertTrue(path.add(node), "SDK aggregation must not create a cycle"); + assertAcyclic(node.getCause(), path); + for (var suppressed : node.getSuppressed()) assertAcyclic(suppressed, path); + path.remove(node); + } + + private static Context runtime() { + var context = mock(Context.class); + when(context.getRemainingTimeInMillis()).thenReturn(10_000); + when(context.getAwsRequestId()).thenReturn("completion-adjacent-request"); + return context; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/adjacent/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorAdmissionFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorAdmissionFailureTest.java new file mode 100644 index 0000000000..9b9a54563a --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorAdmissionFailureTest.java @@ -0,0 +1,102 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(10) +class OwnedExecutorAdmissionFailureTest { + @ParameterizedTest + @CsvSource({"0,false", "0,true", "500,false", "500,true"}) + void cutoffBeforeDispatchFinalizesOwnedExecutorOnce(int remainingMillis, boolean plugins) throws Exception { + var dispatches = new AtomicInteger(); + var shutdowns = new AtomicInteger(); + var bodies = new AtomicInteger(); + var starts = new AtomicInteger(); + var ends = new AtomicInteger(); + var owned = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void execute(Runnable command) { + dispatches.incrementAndGet(); + super.execute(command); + } + + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + super.shutdown(); + } + }; + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(remainingMillis); + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned); + if (plugins) + builder.withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + starts.incrementAndGet(); + } + + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + } + }); + var input = new DurableExecutionInput( + "arn:execution/test", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("test") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + try { + var failure = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input, + runtime, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + builder.build())); + assertAll( + () -> assertInstanceOf(UnrecoverableDurableExecutionException.class, failure), + () -> assertEquals(0, dispatches.get()), + () -> assertEquals(0, bodies.get()), + () -> assertEquals(0, starts.get()), + () -> assertEquals(0, ends.get()), + () -> assertEquals( + 1, shutdowns.get(), "Synchronous failure must finalize owned resources only once"), + () -> assertTrue(owned.isTerminated())); + assertTrue(((UnrecoverableDurableExecutionException) failure).isRetryable()); + } finally { + owned.shutdownNow(); + assertTrue(owned.awaitTermination(3, TimeUnit.SECONDS)); + } + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorDeliveryFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorDeliveryFailureTest.java new file mode 100644 index 0000000000..d5c1cbae6e --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorDeliveryFailureTest.java @@ -0,0 +1,156 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.SerDes; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedExecutorDeliveryFailureTest { + enum Cleanup { + NONE, + ORDINARY, + FATAL, + SAME_ERROR + } + + static Stream cases() { + return Stream.of(Cleanup.values()) + .flatMap(cleanup -> Stream.of(false, true).map(plugins -> Arguments.of(cleanup, plugins))); + } + + @ParameterizedTest + @MethodSource("cases") + void ownedCleanupRetainsSerializerFailureWithoutChangingWinner(Cleanup cleanup, boolean plugins) throws Exception { + var delivery = new IllegalStateException("output serialization failed"); + var ordinary = new IllegalArgumentException("owned shutdown failed"); + var fatal = new InternalError("owned shutdown fatal"); + Throwable shutdownFailure = switch (cleanup) { + case NONE -> null; + case ORDINARY -> ordinary; + case FATAL -> fatal; + case SAME_ERROR -> delivery; + }; + var bodies = new AtomicInteger(); + var serializations = new AtomicInteger(); + var shutdowns = new AtomicInteger(); + var ends = new AtomicInteger(); + var end = new AtomicReference(); + var pool = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + super.shutdown(); + if (shutdownFailure != null) ExceptionHelper.sneakyThrow(shutdownFailure); + } + }; + pool.prestartAllCoreThreads(); + var builder = DurableConfig.builder() + .withInvocationExecutorFactory(() -> pool) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withSerDes(new SerDes() { + public String serialize(Object value) { + serializations.incrementAndGet(); + throw delivery; + } + + @SuppressWarnings("unchecked") + public T deserialize(String value, TypeToken type) { + return (T) "input"; + } + }); + if (plugins) + builder.withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + end.set(info); + } + }); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return "output"; + }, + builder.build())); + if (cleanup == Cleanup.NONE) assertSame(delivery, observed); + else if (cleanup == Cleanup.FATAL) assertSame(fatal, observed); + else { + var winner = assertInstanceOf(UnrecoverableDurableExecutionException.class, observed); + assertEquals("Invocation executor shutdown failed", winner.getMessage()); + assertSame(shutdownFailure, winner.getCause()); + } + assertTrue(contains(observed, delivery), "The completion-only serializer diagnostic must be retained"); + assertEquals(1, bodies.get()); + assertEquals(1, serializations.get()); + assertEquals(1, shutdowns.get()); + assertTrue(pool.isTerminated()); + assertEquals(plugins ? 1 : 0, ends.get()); + if (plugins) { + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(delivery, end.get().executionError()); + } + if (cleanup == Cleanup.SAME_ERROR) + assertEquals(0, observed.getSuppressed().length, "Do not duplicate the known lifecycle cause"); + } finally { + pool.shutdownNow(); + assertTrue(pool.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static boolean contains(Throwable root, Throwable expected) { + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var next = pending.remove(); + if (next == expected) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) pending.add(next.getCause()); + Collections.addAll(pending, next.getSuppressed()); + } + return false; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/delivery", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("delivery") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorFailureOrderingTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorFailureOrderingTest.java new file mode 100644 index 0000000000..3f9448e891 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorFailureOrderingTest.java @@ -0,0 +1,217 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedExecutorFailureOrderingTest { + enum Stage { + SHUTDOWN, + AWAIT + } + + enum Failures { + FATAL_THEN_ORDINARY, + DEATH_THEN_FATAL, + ORDINARY_THEN_FATAL, + COMPLETION_FATAL, + EXECUTION_FATAL + } + + static Stream failureOrders() { + return Stream.of(Stage.values()) + .flatMap(stage -> Stream.of(Failures.values()) + .flatMap(order -> Stream.of(false, true).map(prior -> Arguments.of(stage, order, prior)))); + } + + @SuppressWarnings("removal") + @ParameterizedTest + @MethodSource("failureOrders") + void stateQueryCannotReplaceEarlierFailures(Stage stage, Failures order, boolean priorHandlerFatal) + throws Exception { + var handlerFatal = new InternalError("earlier handler fatal"); + Error shutdownFatal = + order == Failures.DEATH_THEN_FATAL ? new ThreadDeath() : new InternalError("shutdown fatal"); + var ordinaryShutdown = new IllegalStateException("shutdown failed"); + Throwable shutdownFailure = switch (order) { + case ORDINARY_THEN_FATAL -> ordinaryShutdown; + case COMPLETION_FATAL -> new CompletionException(shutdownFatal); + case EXECUTION_FATAL -> new ExecutionException(shutdownFatal); + default -> shutdownFatal; + }; + Throwable queryFailure = order == Failures.DEATH_THEN_FATAL || order == Failures.ORDINARY_THEN_FATAL + ? new InternalError("later state-query fatal") + : new IllegalArgumentException("later state-query failure"); + var expected = + priorHandlerFatal ? handlerFatal : order == Failures.ORDINARY_THEN_FATAL ? queryFailure : shutdownFatal; + var inject = new AtomicBoolean(true); + var shutdownStarted = new AtomicBoolean(); + var bodyExited = new AtomicBoolean(); + var endBeforeShutdown = new AtomicBoolean(); + var shutdownCalls = new AtomicInteger(); + var stateCalls = new AtomicInteger(); + var events = new CopyOnWriteArrayList(); + var owner = new AtomicReference(); + var endThread = new AtomicReference(); + var end = new AtomicReference(); + var ends = new AtomicInteger(); + var workerFailure = new AtomicReference(); + var workerEscaped = new CountDownLatch(1); + var users = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(), action -> { + var thread = new Thread(action, "ordered-shutdown-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerFailure.set(failure); + workerEscaped.countDown(); + }); + return thread; + }) { + @Override + public void shutdown() { + if (inject.get()) { + events.add("shutdown"); + shutdownCalls.incrementAndGet(); + endBeforeShutdown.set(bodyExited.get() && ends.get() == 1); + } + shutdownStarted.set(true); + super.shutdown(); + if (inject.get() && stage == Stage.SHUTDOWN) ExceptionHelper.sneakyThrow(shutdownFailure); + } + + @Override + public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException { + if (inject.get() && stage == Stage.AWAIT) { + events.add("await"); + ExceptionHelper.sneakyThrow(shutdownFailure); + } + return super.awaitTermination(timeout, unit); + } + + @Override + public boolean isShutdown() { + var stopped = super.isShutdown(); + if (inject.get() && shutdownStarted.get()) { + events.add("state"); + stateCalls.incrementAndGet(); + ExceptionHelper.sneakyThrow(queryFailure); + } + return stopped; + } + }; + var callers = Executors.newSingleThreadExecutor(); + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + public void onInvocationEnd(InvocationEndInfo info) { + endThread.set(Thread.currentThread()); + end.set(info); + ends.incrementAndGet(); + events.add("end"); + } + }) + .build(); + try { + var result = callers.submit(() -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + try { + if (priorHandlerFatal) throw handlerFatal; + return "result"; + } finally { + bodyExited.set(true); + } + }, + config)); + var observed = assertThrows(ExecutionException.class, () -> result.get(5, TimeUnit.SECONDS)) + .getCause(); + assertSame(expected, observed, "The first recognized fatal must survive the later state query"); + assertTrue( + contains(observed, order == Failures.ORDINARY_THEN_FATAL ? ordinaryShutdown : shutdownFatal), + "The shutdown attempt's original failure remains reachable"); + assertTrue(contains(observed, queryFailure), "The later state-query diagnostic remains reachable"); + assertEquals( + stage == Stage.SHUTDOWN + ? List.of("end", "shutdown", "state") + : List.of("end", "shutdown", "await", "state"), + events); + assertTrue(endBeforeShutdown.get(), "Owned cleanup follows actual body and End completion"); + assertEquals(1, shutdownCalls.get(), "Automatic close cannot repeat finalized shutdown callbacks"); + assertEquals(1, stateCalls.get()); + assertEquals(1, ends.get()); + assertSame(owner.get(), endThread.get()); + assertEquals( + priorHandlerFatal ? InvocationStatus.RETRYING : InvocationStatus.SUCCEEDED, + end.get().invocationStatus()); + assertSame(priorHandlerFatal ? handlerFatal : null, end.get().executionError()); + if (priorHandlerFatal) { + assertTrue(workerEscaped.await(3, TimeUnit.SECONDS)); + assertSame(handlerFatal, workerFailure.get()); + } else assertNull(workerFailure.get()); + } finally { + inject.set(false); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static boolean contains(Throwable root, Throwable expected) { + var seen = Collections.newSetFromMap(new IdentityHashMap()); + var pending = new ArrayDeque(); + pending.add(root); + while (!pending.isEmpty()) { + var failure = pending.removeFirst(); + if (failure == expected) return true; + if (!seen.add(failure)) continue; + if (failure.getCause() != null) pending.add(failure.getCause()); + Collections.addAll(pending, failure.getSuppressed()); + } + return false; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/ordering/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorForcedShutdownTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorForcedShutdownTest.java new file mode 100644 index 0000000000..a44027ad6b --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorForcedShutdownTest.java @@ -0,0 +1,323 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(20) +class OwnedExecutorForcedShutdownTest { + enum First { + ORDINARY, + FATAL, + WRAPPED_FATAL + } + + enum Later { + NONE, + FORCED_FATAL, + STATE_FATAL, + FORCED_ORDINARY_STATE_FATAL + } + + static Stream cases() { + var normal = Stream.of(First.values()) + .flatMap(first -> Stream.of(Later.values()) + .flatMap(later -> + Stream.of(false, true).map(prior -> Arguments.of(first, later, prior, false)))); + return Stream.concat( + normal, + Stream.of( + Arguments.of(First.ORDINARY, Later.NONE, false, true), + Arguments.of(First.ORDINARY, Later.FORCED_FATAL, false, true))); + } + + @ParameterizedTest + @MethodSource("cases") + void aThrowingGracefulShutdownStillStopsItsOwnedWorkers( + First first, Later later, boolean handlerFatal, boolean acknowledged) throws Exception { + var earlierHandlerFatal = new InternalError("earlier handler fatal"); + var shutdownFatal = new InternalError("graceful shutdown fatal"); + var ordinaryShutdown = new IllegalStateException("graceful shutdown failed before delegation"); + Throwable shutdownFailure = switch (first) { + case ORDINARY -> ordinaryShutdown; + case FATAL -> shutdownFatal; + case WRAPPED_FATAL -> new CompletionException(shutdownFatal); + }; + Throwable forcedFailure = later == Later.FORCED_FATAL + ? new InternalError("forced stop fatal") + : later == Later.FORCED_ORDINARY_STATE_FATAL + ? new IllegalArgumentException("forced stop diagnostic") + : null; + Throwable stateFailure = later == Later.STATE_FATAL || later == Later.FORCED_ORDINARY_STATE_FATAL + ? new InternalError("state query fatal") + : null; + Throwable expectedFatal = handlerFatal + ? earlierHandlerFatal + : first != First.ORDINARY ? shutdownFatal : later == Later.FORCED_FATAL ? forcedFailure : stateFailure; + var end = new AtomicReference(); + var ends = new AtomicInteger(); + var owner = new AtomicReference(); + var endOwner = new AtomicReference(); + var caller = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var manager = new AtomicReference(); + var endEntered = new CountDownLatch(1); + var releaseEnd = new CountDownLatch(acknowledged ? 1 : 0); + var workerFailure = new AtomicReference(); + var workerEscaped = new CountDownLatch(1); + var users = new FailingPool(shutdownFailure, forcedFailure, stateFailure, workerFailure, workerEscaped); + assertEquals(1, users.prestartAllCoreThreads()); + var callers = Executors.newSingleThreadExecutor(); + var client = TestUtils.createMockClient(); + var value = acknowledged ? "x".repeat(6 * 1024 * 1024 + 100) : "result"; + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> users) + .withDurableExecutionClient(client) + .withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + public void onInvocationEnd(InvocationEndInfo info) { + endOwner.set(Thread.currentThread()); + end.set(info); + ends.incrementAndGet(); + endEntered.countDown(); + try { + assertTrue(releaseEnd.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + } + } + }) + .build(); + try { + var invocation = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (input, context) -> { + manager.set(((DurableContextImpl) context).getExecutionManager()); + if (handlerFatal) throw earlierHandlerFatal; + return value; + }, + config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertTrue(endEntered.await(5, TimeUnit.SECONDS)); + if (acknowledged) caller.get().interrupt(); + Throwable observed; + if (acknowledged && expectedFatal == null) { + var output = invocation.get(5, TimeUnit.SECONDS); + assertEquals(ExecutionStatus.SUCCEEDED, output.status()); + assertEquals("", output.result()); + observed = manager.get().getLifecycleFailure(); + } else { + observed = assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause(); + if (expectedFatal != null) assertSame(expectedFatal, observed); + else { + var failure = assertInstanceOf(UnrecoverableDurableExecutionException.class, observed); + assertEquals("Invocation executor shutdown failed", failure.getMessage()); + assertSame(ordinaryShutdown, failure.getCause()); + } + } + assertTrue(users.isTerminated(), "A graceful callback failure must not retain live owned workers"); + assertEquals(1, users.gracefulCalls.get()); + assertEquals(1, users.forcedCalls.get(), "Normal finalization forces cleanup once, without close retry"); + assertTrue(contains(observed, first == First.ORDINARY ? ordinaryShutdown : shutdownFatal)); + if (forcedFailure != null) assertTrue(contains(observed, forcedFailure)); + if (stateFailure != null) assertTrue(contains(observed, stateFailure)); + assertEquals(acknowledged, callerInterrupted.get()); + assertEquals(1, ends.get()); + assertSame(owner.get(), endOwner.get()); + assertEquals( + handlerFatal ? InvocationStatus.RETRYING : InvocationStatus.SUCCEEDED, + end.get().invocationStatus()); + assertSame(handlerFatal ? earlierHandlerFatal : null, end.get().executionError()); + if (handlerFatal) { + assertTrue(workerEscaped.await(3, TimeUnit.SECONDS)); + assertSame(earlierHandlerFatal, workerFailure.get()); + } else assertNull(workerFailure.get()); + if (acknowledged) + verify(client, times(1)) + .checkpoint( + any(), + any(), + argThat(updates -> updates.size() == 1 + && updates.get(0).type() == OperationType.EXECUTION + && updates.get(0).action() == OperationAction.SUCCEED)); + if (stateFailure == null) { + users.inject.set(false); + var rejected = assertThrows( + IllegalStateException.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (input, context) -> "must not run", + config)); + assertEquals( + "Invocation executor factory returned a shutdown executor", + rejected.getMessage(), + "The stopped executor's normal lease is released"); + } + } finally { + releaseEnd.countDown(); + users.dispose(); + callers.shutdownNow(); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + @Test + void sharedExecutorsDoNotEnterOwnedShutdownRecovery() throws Exception { + var users = new FailingPool( + new IllegalStateException("must not shut down shared executor"), + null, + null, + new AtomicReference<>(), + new CountDownLatch(1)); + users.prestartAllCoreThreads(); + try { + var config = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .build(); + assertEquals( + ExecutionStatus.SUCCEEDED, + DurableExecutor.execute( + input(), null, TypeToken.get(String.class), (value, context) -> value, config) + .status()); + assertEquals(0, users.gracefulCalls.get()); + assertEquals(0, users.forcedCalls.get()); + assertFalse(users.isShutdown()); + } finally { + users.dispose(); + } + } + + private static final class FailingPool extends ThreadPoolExecutor { + private final Throwable first; + private final Throwable forced; + private final Throwable state; + private final AtomicBoolean inject = new AtomicBoolean(true); + private final AtomicBoolean firstCall = new AtomicBoolean(true); + private final AtomicBoolean shutdownAttempted = new AtomicBoolean(); + private final AtomicInteger gracefulCalls = new AtomicInteger(); + private final AtomicInteger forcedCalls = new AtomicInteger(); + + private FailingPool( + Throwable first, + Throwable forced, + Throwable state, + AtomicReference workerFailure, + CountDownLatch escaped) { + super(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(), command -> { + var thread = new Thread(command, "forced-shutdown-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerFailure.set(failure); + escaped.countDown(); + }); + return thread; + }); + this.first = first; + this.forced = forced; + this.state = state; + } + + @Override + public void shutdown() { + if (inject.get()) { + gracefulCalls.incrementAndGet(); + shutdownAttempted.set(true); + if (firstCall.compareAndSet(true, false)) ExceptionHelper.sneakyThrow(first); + } + super.shutdown(); + } + + @Override + public List shutdownNow() { + var result = super.shutdownNow(); + if (inject.get()) { + forcedCalls.incrementAndGet(); + if (forced != null) ExceptionHelper.sneakyThrow(forced); + } + return result; + } + + @Override + public boolean isShutdown() { + if (inject.get() && shutdownAttempted.get() && state != null) ExceptionHelper.sneakyThrow(state); + return super.isShutdown(); + } + + private void dispose() throws InterruptedException { + inject.set(false); + super.shutdownNow(); + assertTrue(super.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static boolean contains(Throwable root, Throwable target) { + if (root == null) return false; + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var next = pending.removeFirst(); + if (next == target) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) pending.add(next.getCause()); + Collections.addAll(pending, next.getSuppressed()); + } + return false; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/forced/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorInitializationFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorInitializationFailureTest.java new file mode 100644 index 0000000000..db4b4e0e4b --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorInitializationFailureTest.java @@ -0,0 +1,263 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedExecutorInitializationFailureTest { + enum Kind { + ORDINARY, + FATAL, + WRAPPED_FATAL + } + + enum Cleanup { + NONE, + ORDINARY, + FATAL + } + + static Stream failures() { + return Stream.of(Kind.values()) + .flatMap(kind -> Stream.of(Cleanup.values()).map(cleanup -> Arguments.of(kind, cleanup))); + } + + @ParameterizedTest + @MethodSource("failures") + void failedStateValidationDisposesOnlyTheClaimedPool(Kind kind, Cleanup cleanup) throws Exception { + var initialFatal = new InternalError("initial state query fatal"); + Throwable initial = switch (kind) { + case ORDINARY -> new IllegalArgumentException("initial state query failed"); + case FATAL -> initialFatal; + case WRAPPED_FATAL -> new CompletionException(initialFatal); + }; + Throwable cleanupFailure = switch (cleanup) { + case NONE -> null; + case ORDINARY -> new IllegalStateException("graceful shutdown failed"); + case FATAL -> new InternalError("later shutdown fatal"); + }; + var expected = kind != Kind.ORDINARY ? initialFatal : cleanup == Cleanup.FATAL ? cleanupFailure : initial; + var bodies = new AtomicInteger(); + var starts = new AtomicInteger(); + var ends = new AtomicInteger(); + var pool = new QueryFailingPool(initial, cleanupFailure); + assertEquals(1, pool.prestartAllCoreThreads()); + var config = config(pool, starts, ends); + try { + var observed = assertThrows(Throwable.class, () -> invoke(config, bodies)); + assertSame(expected, observed, "Preserve the original failure unless cleanup introduces the first fatal"); + assertTrue(pool.isTerminated(), "Prestarted workers must be disposed before failed setup returns"); + assertTrue(pool.shutdownCalls.get() > 0 || pool.forcedCalls.get() > 0); + assertEquals(1, pool.queryCalls.get(), "Cleanup does not need to repeat the failed state query"); + assertEquals(0, bodies.get()); + assertEquals(0, starts.get()); + assertEquals(0, ends.get()); + if (cleanupFailure != null) { + assertTrue(contains(observed, cleanupFailure), "Retain later cleanup diagnostics"); + assertTrue(contains(observed, kind == Kind.ORDINARY ? initial : initialFatal)); + assertTrue(pool.forcedCalls.get() > 0, "Failed graceful shutdown still gets a best-effort forced stop"); + } + pool.inject.set(false); + var rejected = assertThrows(IllegalStateException.class, () -> invoke(config, bodies)); + assertEquals( + "Invocation executor factory returned a shutdown executor", + rejected.getMessage(), + "Failed initialization must release its lease, so closed-state validation remains authoritative"); + assertEquals(0, bodies.get()); + } finally { + pool.dispose(); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void initializationCleanupPreservesCallerInterrupt(boolean interrupted) throws Exception { + var initial = new IllegalArgumentException("initial query failed"); + var pool = new QueryFailingPool(initial, null); + pool.prestartAllCoreThreads(); + try { + if (interrupted) Thread.currentThread().interrupt(); + var observed = assertThrows( + Throwable.class, + () -> invoke(config(pool, new AtomicInteger(), new AtomicInteger()), new AtomicInteger())); + assertSame(initial, observed); + assertEquals(interrupted, Thread.interrupted()); + assertTrue(pool.isTerminated()); + } finally { + Thread.interrupted(); + pool.dispose(); + } + } + + @Test + void anAlreadyLeasedPoolIsNeitherQueriedNorStopped() throws Exception { + var pool = new QueryFailingPool(new IllegalArgumentException("foreign state query must not run"), null); + pool.inject.set(false); + pool.prestartAllCoreThreads(); + var entered = new CountDownLatch(1); + var release = new CountDownLatch(1); + var callers = Executors.newSingleThreadExecutor(); + var config = config(pool, new AtomicInteger(), new AtomicInteger()); + try { + var first = callers.submit(() -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + entered.countDown(); + try { + assertTrue(release.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException interrupted) { + throw new AssertionError(interrupted); + } + return value; + }, + config)); + assertTrue(entered.await(5, TimeUnit.SECONDS)); + var queriesBeforeSecond = pool.queryCalls.get(); + pool.inject.set(true); + var secondBodies = new AtomicInteger(); + var rejected = assertThrows(IllegalStateException.class, () -> invoke(config, secondBodies)); + assertEquals("Invocation executor factory returned an executor already in use", rejected.getMessage()); + assertEquals(queriesBeforeSecond, pool.queryCalls.get(), "Do not call foreign executor state callbacks"); + assertEquals(0, pool.shutdownCalls.get()); + assertEquals(0, pool.forcedCalls.get()); + assertEquals(0, secondBodies.get()); + assertFalse(first.isDone()); + pool.inject.set(false); + release.countDown(); + assertEquals( + ExecutionStatus.SUCCEEDED, first.get(5, TimeUnit.SECONDS).status()); + } finally { + pool.inject.set(false); + release.countDown(); + callers.shutdownNow(); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + pool.dispose(); + } + } + + private static DurableConfig config(QueryFailingPool pool, AtomicInteger starts, AtomicInteger ends) { + return DurableConfig.builder() + .withInvocationExecutorFactory(() -> pool) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + starts.incrementAndGet(); + } + + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + } + }) + .build(); + } + + private static Object invoke(DurableConfig config, AtomicInteger bodies) { + return DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + config); + } + + private static final class QueryFailingPool extends ThreadPoolExecutor { + private final Throwable initial; + private final Throwable cleanup; + private final AtomicBoolean inject = new AtomicBoolean(true); + private final AtomicInteger queryCalls = new AtomicInteger(); + private final AtomicInteger shutdownCalls = new AtomicInteger(); + private final AtomicInteger forcedCalls = new AtomicInteger(); + + private QueryFailingPool(Throwable initial, Throwable cleanup) { + super(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()); + this.initial = initial; + this.cleanup = cleanup; + } + + @Override + public boolean isShutdown() { + queryCalls.incrementAndGet(); + if (inject.get()) ExceptionHelper.sneakyThrow(initial); + return super.isShutdown(); + } + + @Override + public void shutdown() { + shutdownCalls.incrementAndGet(); + if (inject.get() && cleanup != null) ExceptionHelper.sneakyThrow(cleanup); + super.shutdown(); + } + + @Override + public List shutdownNow() { + forcedCalls.incrementAndGet(); + return super.shutdownNow(); + } + + private void dispose() throws InterruptedException { + inject.set(false); + super.shutdownNow(); + assertTrue(super.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static boolean contains(Throwable root, Throwable target) { + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var next = pending.removeFirst(); + if (next == target) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) pending.add(next.getCause()); + Collections.addAll(pending, next.getSuppressed()); + } + return false; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/init", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("init") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorLifecycleIdentityTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorLifecycleIdentityTest.java new file mode 100644 index 0000000000..3ae04d556b --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorLifecycleIdentityTest.java @@ -0,0 +1,141 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.Arrays; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(15) +class OwnedExecutorLifecycleIdentityTest { + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void ownedTerminationTimeoutDoesNotReplacePreviouslySelectedCutoff(boolean plugins) throws Exception { + var releaseWorkerTail = new CountDownLatch(1); + var workerTailEntered = new CountDownLatch(1); + var first = new AtomicReference(); + var end = new AtomicReference(); + var stepCalls = new AtomicInteger(); + var owned = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void execute(Runnable command) { + var commandReturned = new CountDownLatch(1); + super.execute(() -> { + try { + command.run(); + } finally { + commandReturned.countDown(); + } + }); + // Execute may be synchronous, as with CallerRunsPolicy. Runtime observation starts after the owner. + awaitIgnoringInterrupt(commandReturned); + } + + @Override + protected void afterExecute(Runnable command, Throwable failure) { + super.afterExecute(command, failure); + workerTailEntered.countDown(); + // Hold a real executor termination boundary after the tracked SDK task has actually exited. + awaitIgnoringInterrupt(releaseWorkerTail); + } + }; + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(2000); + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned); + if (plugins) + builder.withPlugins(info -> new DurableExecutionPlugin() { + public void onInvocationEnd(InvocationEndInfo info) { + end.set(info); + } + }); + var input = new DurableExecutionInput( + "arn:execution/test", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("test") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + try { + var observed = assertThrows( + UnrecoverableDurableExecutionException.class, + () -> DurableExecutor.execute( + input, + runtime, + TypeToken.get(String.class), + (value, context) -> { + try { + assertFalse(new CountDownLatch(1).await(1600, TimeUnit.MILLISECONDS)); + } catch (InterruptedException interrupted) { + throw new AssertionError(interrupted); + } + try { + context.step("expired", String.class, step -> { + stepCalls.incrementAndGet(); + return "late"; + }); + } catch (UnrecoverableDurableExecutionException selected) { + first.set(selected); + } + return "fallback"; + }, + builder.build())); + assertNotNull(first.get()); + assertEquals( + "Invocation deadline reached before user work", first.get().getMessage()); + assertSame( + first.get(), + observed, + "Owned termination is a later lifecycle diagnostic, not a replacement outcome"); + assertTrue(Arrays.stream(first.get().getSuppressed()) + .anyMatch(later -> later instanceof UnrecoverableDurableExecutionException + && later.getMessage() + .equals("Invocation executor did not terminate before cleanup deadline"))); + assertTrue(workerTailEntered.await(2, TimeUnit.SECONDS)); + assertFalse(owned.isTerminated(), "The fixture must exercise a real termination timeout"); + assertEquals(0, stepCalls.get()); + if (plugins) { + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(first.get(), end.get().executionError()); + } + } finally { + releaseWorkerTail.countDown(); + owned.shutdownNow(); + assertTrue(owned.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static void awaitIgnoringInterrupt(CountDownLatch gate) { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(8); + while (true) { + try { + assertTrue(gate.await(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + return; + } catch (InterruptedException ignored) { + } + } + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorMdcFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorMdcFailureTest.java new file mode 100644 index 0000000000..4a0e5893e3 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorMdcFailureTest.java @@ -0,0 +1,203 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.slf4j.MDC; +import org.slf4j.helpers.BasicMDCAdapter; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedExecutorMdcFailureTest { + enum Cleanup { + NONE, + ORDINARY, + FATAL, + WRAPPED_FATAL, + SAME_FATAL, + WRAPPED_SAME_FATAL + } + + static Stream cases() { + return Stream.concat( + Stream.of(false, true) + .flatMap(death -> + Stream.of(Cleanup.values()).map(cleanup -> Arguments.of(death, cleanup, false))), + Stream.of(false, true).map(death -> Arguments.of(death, Cleanup.NONE, true))); + } + + @SuppressWarnings("removal") + @ParameterizedTest + @MethodSource("cases") + void asynchronousMdcFatalRetainsLaterOwnedCleanup(boolean death, Cleanup cleanup, boolean shared) throws Exception { + Error first = death ? new ThreadDeath() : new InternalError("MDC restoration fatal"); + var laterFatal = new InternalError("later owned shutdown fatal"); + var ordinary = new IllegalStateException("later owned shutdown diagnostic"); + Throwable cleanupFailure = switch (cleanup) { + case NONE -> null; + case ORDINARY -> ordinary; + case FATAL -> laterFatal; + case WRAPPED_FATAL -> new CompletionException(laterFatal); + case SAME_FATAL -> first; + case WRAPPED_SAME_FATAL -> new CompletionException(first); + }; + var end = new AtomicReference(); + var owner = new AtomicReference(); + var ends = new AtomicInteger(); + var restored = new AtomicInteger(); + var shutdowns = new AtomicInteger(); + var manager = new AtomicReference(); + var fatalSeenByShutdown = new AtomicReference(); + var workerFailure = new AtomicReference(); + var workerEscaped = new CountDownLatch(1); + var originalAdapter = MDC.getMDCAdapter(); + var adapterField = MDC.class.getDeclaredField("MDC_ADAPTER"); + adapterField.setAccessible(true); + adapterField.set(null, new BasicMDCAdapter() { + @Override + public void setContextMap(Map values) { + if (Thread.currentThread() == owner.get() + && end.get() != null + && values.equals(Map.of("seed", "before-invocation"))) { + restored.incrementAndGet(); + throw first; + } + super.setContextMap(values); + } + }); + var pool = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(), task -> { + var thread = new Thread( + () -> { + MDC.put("seed", "before-invocation"); + task.run(); + }, + "mdc-restoration-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerFailure.set(failure); + workerEscaped.countDown(); + }); + return thread; + }) { + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + fatalSeenByShutdown.set(manager.get().getInvocationFatal()); + super.shutdown(); + if (cleanupFailure != null) ExceptionHelper.sneakyThrow(cleanupFailure); + } + }; + pool.prestartAllCoreThreads(); + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + end.set(info); + } + }); + if (shared) builder.withExecutorService(pool); + else builder.withInvocationExecutorFactory(() -> pool); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + manager.set(((DurableContextImpl) context).getExecutionManager()); + return value; + }, + builder.build())); + assertSame(first, observed, "The first root completion fatal must reach the caller unchanged"); + assertTrue(workerEscaped.await(3, TimeUnit.SECONDS)); + assertSame(first, workerFailure.get()); + assertEquals(1, restored.get(), "Only final handler MDC restoration injects the fatal"); + assertEquals(1, ends.get()); + assertEquals(InvocationStatus.SUCCEEDED, end.get().invocationStatus()); + assertNull(end.get().executionError(), "End must keep its already-selected successful body outcome"); + assertEquals(shared ? 0 : 1, shutdowns.get()); + if (shared) assertFalse(pool.isShutdown()); + else assertTrue(pool.isTerminated()); + assertNull(fatalSeenByShutdown.get(), "This regression reaches cleanup through the root future alone"); + if (cleanupFailure != null) { + var expected = cleanup == Cleanup.ORDINARY + ? ordinary + : cleanup == Cleanup.SAME_FATAL || cleanup == Cleanup.WRAPPED_SAME_FATAL ? first : laterFatal; + assertTrue(contains(observed, expected), "Retain the later owned cleanup diagnostic"); + } + assertNoCycle(observed, Collections.newSetFromMap(new IdentityHashMap())); + } finally { + try { + pool.shutdownNow(); + assertTrue(pool.awaitTermination(3, TimeUnit.SECONDS)); + } finally { + adapterField.set(null, originalAdapter); + } + } + } + + private static boolean contains(Throwable root, Throwable expected) { + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var failure = pending.remove(); + if (failure == expected) return true; + if (!seen.add(failure)) continue; + if (failure.getCause() != null) pending.add(failure.getCause()); + Collections.addAll(pending, failure.getSuppressed()); + } + return false; + } + + private static void assertNoCycle(Throwable failure, Set path) { + assertTrue(path.add(failure), "Diagnostics must not contain a cause/suppression cycle"); + if (failure.getCause() != null) assertNoCycle(failure.getCause(), path); + for (var suppressed : failure.getSuppressed()) assertNoCycle(suppressed, path); + path.remove(failure); + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/mdc", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("mdc") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorSetupFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorSetupFailureTest.java new file mode 100644 index 0000000000..276510c743 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorSetupFailureTest.java @@ -0,0 +1,452 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Arrays; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestContext; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.util.ExceptionHelper; + +class OwnedExecutorSetupFailureTest { + @SuppressWarnings("removal") + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void headerFatalRetainsIdentityWhenOwnedCloseReportsTheSameFatal(boolean death) throws Exception { + Error failure = death ? new ThreadDeath() : new OutOfMemoryError("simulated reused VM fatal"); + var shutdownCalls = new AtomicInteger(); + var bodies = new AtomicInteger(); + var owned = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void shutdown() { + shutdownCalls.incrementAndGet(); + super.shutdown(); + throw failure; + } + }; + var runtime = mock(RuntimeContext.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(10_000); + when(runtime.getXrayTraceId()).thenThrow(failure); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned) + .withPlugins(ignored -> new DurableExecutionPlugin() {}) + .build(); + var input = new DurableExecutionInput( + "arn:execution/test", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("test") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input, + runtime, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + config)); + assertSame(failure, observed, "Implicit close must not self-suppress a fatal raised before dispatch"); + assertEquals(0, bodies.get()); + assertEquals(1, shutdownCalls.get()); + assertTrue(owned.isTerminated()); + } finally { + owned.shutdownNow(); + assertTrue(owned.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + enum Setup { + ORDINARY, + FATAL, + WRAPPED_FATAL, + SAME_FATAL, + NESTED_WRAPPED_FATAL, + CHANGING_FATAL_CAUSE, + THROWING_FATAL_CAUSE, + APPLICATION_WRAPPER, + WRAPPED_ORDINARY, + NESTED_WRAPPED_ORDINARY, + CHANGING_ORDINARY_CAUSE, + COMPLETED_APPLICATION_WRAPPER, + WRAPPED_ORDINARY_FATAL_DIAGNOSTIC + } + + enum Cleanup { + NONE, + ORDINARY, + FATAL, + THREAD_DEATH, + WRAPPED_FATAL + } + + @SuppressWarnings("removal") + @ParameterizedTest + @CsvSource({ + "ORDINARY,FATAL,false", "ORDINARY,FATAL,true", + "ORDINARY,THREAD_DEATH,false", "ORDINARY,THREAD_DEATH,true", + "ORDINARY,WRAPPED_FATAL,false", "ORDINARY,WRAPPED_FATAL,true", + "WRAPPED_FATAL,FATAL,false", "WRAPPED_FATAL,FATAL,true", + "FATAL,FATAL,false", "FATAL,FATAL,true", + "SAME_FATAL,FATAL,false", "SAME_FATAL,FATAL,true", + "ORDINARY,NONE,false", "ORDINARY,NONE,true", + "ORDINARY,ORDINARY,false", "ORDINARY,ORDINARY,true", + "APPLICATION_WRAPPER,NONE,false", "APPLICATION_WRAPPER,NONE,true", + "APPLICATION_WRAPPER,FATAL,false", "APPLICATION_WRAPPER,FATAL,true", + "WRAPPED_FATAL,NONE,false", "WRAPPED_FATAL,NONE,true", + "NESTED_WRAPPED_FATAL,NONE,false", "NESTED_WRAPPED_FATAL,NONE,true", + "NESTED_WRAPPED_FATAL,FATAL,false", "NESTED_WRAPPED_FATAL,FATAL,true", + "CHANGING_FATAL_CAUSE,NONE,false", "CHANGING_FATAL_CAUSE,NONE,true", + "THROWING_FATAL_CAUSE,NONE,false", "THROWING_FATAL_CAUSE,NONE,true", + "WRAPPED_ORDINARY,NONE,false", "WRAPPED_ORDINARY,NONE,true", + "WRAPPED_ORDINARY,ORDINARY,false", "WRAPPED_ORDINARY,ORDINARY,true", + "WRAPPED_ORDINARY,FATAL,false", "WRAPPED_ORDINARY,FATAL,true", + "NESTED_WRAPPED_ORDINARY,NONE,false", "NESTED_WRAPPED_ORDINARY,NONE,true", + "NESTED_WRAPPED_ORDINARY,FATAL,false", "NESTED_WRAPPED_ORDINARY,FATAL,true", + "CHANGING_ORDINARY_CAUSE,NONE,false", "CHANGING_ORDINARY_CAUSE,NONE,true", + "COMPLETED_APPLICATION_WRAPPER,NONE,false", "COMPLETED_APPLICATION_WRAPPER,NONE,true", + "COMPLETED_APPLICATION_WRAPPER,FATAL,false", "COMPLETED_APPLICATION_WRAPPER,FATAL,true", + "WRAPPED_ORDINARY_FATAL_DIAGNOSTIC,NONE,false", "WRAPPED_ORDINARY_FATAL_DIAGNOSTIC,NONE,true", + "WRAPPED_ORDINARY_FATAL_DIAGNOSTIC,FATAL,false", "WRAPPED_ORDINARY_FATAL_DIAGNOSTIC,FATAL,true" + }) + void postManagerSetupFailureIsExplicitlyFinalized(Setup setup, Cleanup cleanup, boolean plugins) throws Exception { + var setupFatal = new InternalError("first setup fatal"); + var setupOrdinary = new IllegalStateException("nonfatal setup cause"); + var applicationCause = new CompletionException(new IllegalStateException("application cause")); + var innerApplicationDiagnostic = new IllegalArgumentException("retained cause diagnostic"); + applicationCause.addSuppressed(innerApplicationDiagnostic); + var applicationWrapper = new ExecutionException("application setup failure", applicationCause); + var applicationDiagnostic = new IllegalStateException("retained application diagnostic"); + applicationWrapper.addSuppressed(applicationDiagnostic); + Throwable wrapperDiagnostic = setup == Setup.WRAPPED_ORDINARY_FATAL_DIAGNOSTIC + ? new InternalError("suppressed wrapper diagnostic") + : new IllegalArgumentException("setup wrapper diagnostic"); + var nestedDiagnostic = new IllegalStateException("nested setup wrapper diagnostic"); + var aliasDiagnostic = new IllegalArgumentException("alias wrapper diagnostic"); + var causeReads = new AtomicInteger(); + Throwable setupFailure = switch (setup) { + case ORDINARY -> new IllegalStateException("second request ID read failed"); + case FATAL, SAME_FATAL -> setupFatal; + case WRAPPED_FATAL -> new CompletionException(setupFatal); + case NESTED_WRAPPED_FATAL -> { + var inner = new ExecutionException(setupFatal); + inner.addSuppressed(nestedDiagnostic); + var alias = new CompletionException(setupFatal); + alias.addSuppressed(aliasDiagnostic); + inner.addSuppressed(alias); + yield new CompletionException(inner); + } + case CHANGING_FATAL_CAUSE -> + new CompletionException("changing cause", null) { + @Override + public synchronized Throwable getCause() { + return causeReads.incrementAndGet() == 1 + ? setupFatal + : new IllegalStateException("second read"); + } + }; + case THROWING_FATAL_CAUSE -> + new CompletionException("fatal cause accessor", null) { + @Override + public synchronized Throwable getCause() { + causeReads.incrementAndGet(); + throw setupFatal; + } + }; + case APPLICATION_WRAPPER -> applicationWrapper; + case COMPLETED_APPLICATION_WRAPPER -> new CompletionException(applicationWrapper); + case WRAPPED_ORDINARY, WRAPPED_ORDINARY_FATAL_DIAGNOSTIC -> new CompletionException(setupOrdinary); + case NESTED_WRAPPED_ORDINARY -> { + var inner = new CompletionException(setupOrdinary); + inner.addSuppressed(nestedDiagnostic); + var alias = new CompletionException(setupOrdinary); + alias.addSuppressed(aliasDiagnostic); + inner.addSuppressed(alias); + yield new CompletionException(inner); + } + case CHANGING_ORDINARY_CAUSE -> + new CompletionException("changing nonfatal cause", null) { + @Override + public synchronized Throwable getCause() { + return causeReads.incrementAndGet() == 1 + ? setupOrdinary + : new IllegalStateException("second read"); + } + }; + }; + var wrappedFatal = setup == Setup.WRAPPED_FATAL + || setup == Setup.NESTED_WRAPPED_FATAL + || setup == Setup.CHANGING_FATAL_CAUSE + || setup == Setup.THROWING_FATAL_CAUSE; + var wrappedOrdinary = setup == Setup.WRAPPED_ORDINARY + || setup == Setup.WRAPPED_ORDINARY_FATAL_DIAGNOSTIC + || setup == Setup.NESTED_WRAPPED_ORDINARY + || setup == Setup.CHANGING_ORDINARY_CAUSE + || setup == Setup.COMPLETED_APPLICATION_WRAPPER; + var setupIsFatal = setup == Setup.FATAL || setup == Setup.SAME_FATAL || wrappedFatal; + var selectedSetup = setupIsFatal + ? setupFatal + : setup == Setup.COMPLETED_APPLICATION_WRAPPER + ? applicationWrapper + : wrappedOrdinary ? setupOrdinary : setupFailure; + if (wrappedFatal || wrappedOrdinary) { + setupFailure.addSuppressed(wrapperDiagnostic); + // Raw and observed-wrapper aliases must not create a cycle when their diagnostics transfer. + setupFailure.addSuppressed(selectedSetup); + if (setup == Setup.NESTED_WRAPPED_FATAL || setup == Setup.NESTED_WRAPPED_ORDINARY) { + setupFailure.getCause().addSuppressed(setupFailure); + setupFailure.addSuppressed(nestedDiagnostic); + } + } + Error cleanupFatal = setup == Setup.SAME_FATAL + ? setupFatal + : cleanup == Cleanup.THREAD_DEATH ? new ThreadDeath() : new InternalError("owned cleanup fatal"); + Throwable cleanupFailure = switch (cleanup) { + case NONE -> null; + case ORDINARY -> new IllegalArgumentException("ordinary owned cleanup failure"); + case FATAL, THREAD_DEATH -> cleanupFatal; + case WRAPPED_FATAL -> new CompletionException(cleanupFatal); + }; + var requests = new AtomicInteger(); + var starts = new AtomicInteger(); + var ends = new AtomicInteger(); + var bodies = new AtomicInteger(); + var factoryCalls = new AtomicInteger(); + var shutdowns = new AtomicInteger(); + var owned = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + super.shutdown(); + if (cleanupFailure != null) ExceptionHelper.sneakyThrow(cleanupFailure); + } + }; + assertEquals(1, owned.prestartAllCoreThreads()); + var runtime = new TestContext("setup-request") { + @Override + public String getAwsRequestId() { + if (requests.incrementAndGet() == 2) ExceptionHelper.sneakyThrow(setupFailure); + return super.getAwsRequestId(); + } + }; + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> { + factoryCalls.incrementAndGet(); + return owned; + }); + if (plugins) + builder.withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + starts.incrementAndGet(); + } + + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + } + }); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + setupInput(), + runtime, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + builder.build())); + if (setupIsFatal) { + assertSame(setupFatal, observed, "The first setup JVM fatal must remain primary"); + } else if (cleanup == Cleanup.FATAL + || cleanup == Cleanup.THREAD_DEATH + || cleanup == Cleanup.WRAPPED_FATAL) { + assertSame( + cleanupFatal, observed, "A cleanup JVM fatal must outrank an earlier nonfatal setup failure"); + } else if (cleanup == Cleanup.NONE) { + assertSame(selectedSetup, observed); + } + assertTrue(contains(observed, selectedSetup), "Retain setup diagnostics"); + if (cleanupFailure != null) + assertTrue( + contains(observed, cleanup == Cleanup.ORDINARY ? cleanupFailure : cleanupFatal), + "Retain later cleanup diagnostics"); + if (setup == Setup.APPLICATION_WRAPPER || setup == Setup.COMPLETED_APPLICATION_WRAPPER) { + assertSame(applicationCause, applicationWrapper.getCause()); + assertArrayEquals(new Throwable[] {innerApplicationDiagnostic}, applicationCause.getSuppressed()); + assertTrue(contains(observed, applicationDiagnostic)); + assertTrue(contains(observed, innerApplicationDiagnostic)); + assertEquals(setup == Setup.APPLICATION_WRAPPER ? 1 : 2, applicationWrapper.getSuppressed().length); + assertSame(applicationDiagnostic, applicationWrapper.getSuppressed()[0]); + assertFalse( + Arrays.asList(applicationWrapper.getSuppressed()).contains(innerApplicationDiagnostic), + "Do not promote diagnostics from the still-reachable application cause"); + } + if (wrappedFatal || wrappedOrdinary) { + assertTrue(contains(observed, wrapperDiagnostic), "The final winner must retain setup diagnostics"); + assertEquals( + 1, + Arrays.stream(selectedSetup.getSuppressed()) + .filter(e -> e == wrapperDiagnostic) + .count()); + assertFalse( + contains(observed, setupFailure), + "Do not attach a wrapper that points back to the selected cause"); + if (setup == Setup.NESTED_WRAPPED_FATAL || setup == Setup.NESTED_WRAPPED_ORDINARY) { + assertEquals( + 1, + Arrays.stream(selectedSetup.getSuppressed()) + .filter(e -> e == nestedDiagnostic) + .count()); + assertEquals( + 1, + Arrays.stream(selectedSetup.getSuppressed()) + .filter(e -> e == aliasDiagnostic) + .count()); + } + if (setup == Setup.CHANGING_FATAL_CAUSE + || setup == Setup.THROWING_FATAL_CAUSE + || setup == Setup.CHANGING_ORDINARY_CAUSE) + assertEquals(1, causeReads.get(), "Inspect each recognized setup wrapper cause only once"); + } + if (setup == Setup.WRAPPED_ORDINARY_FATAL_DIAGNOSTIC) + assertFalse( + contains(wrapperDiagnostic, selectedSetup), + "Diagnostic transfer must not reverse priority or create a cycle"); + assertEquals(2, requests.get(), "The failure must occur after the manager claimed its executor"); + assertEquals(1, factoryCalls.get()); + assertEquals(1, shutdowns.get(), "Automatic close must not repeat explicit finalization"); + assertEquals(0, bodies.get()); + assertEquals(0, starts.get()); + assertEquals(0, ends.get()); + assertTrue(owned.awaitTermination(3, TimeUnit.SECONDS)); + } finally { + owned.shutdownNow(); + assertTrue(owned.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void nonfatalPostManagerSetupStillLeavesSharedExecutorCallerOwned(boolean plugins) throws Exception { + var failure = new IllegalStateException("second request ID read failed"); + var requests = new AtomicInteger(); + var bodies = new AtomicInteger(); + var hooks = new AtomicInteger(); + var shared = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()); + shared.prestartAllCoreThreads(); + var runtime = new TestContext("setup-request") { + @Override + public String getAwsRequestId() { + if (requests.incrementAndGet() == 2) throw failure; + return super.getAwsRequestId(); + } + }; + var builder = DurableConfig.builder() + .withExecutorService(shared) + .withDurableExecutionClient(TestUtils.createMockClient()); + if (plugins) + builder.withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationStart(InvocationInfo info) { + hooks.incrementAndGet(); + } + + public void onInvocationEnd(InvocationEndInfo info) { + hooks.incrementAndGet(); + } + }); + try { + assertSame( + failure, + assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + setupInput(), + runtime, + TypeToken.get(String.class), + (value, context) -> { + bodies.incrementAndGet(); + return value; + }, + builder.build()))); + assertEquals(2, requests.get()); + assertEquals(0, bodies.get()); + assertEquals(0, hooks.get()); + assertFalse(shared.isShutdown()); + } finally { + shared.shutdownNow(); + assertTrue(shared.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static DurableExecutionInput setupInput() { + return new DurableExecutionInput( + "arn:execution/setup", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("setup") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } + + private static boolean contains(Throwable root, Throwable expected) { + var queue = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + queue.add(root); + while (!queue.isEmpty()) { + var next = queue.remove(); + if (next == expected) return true; + if (!seen.add(next)) continue; + var cause = next.getCause(); + if (cause != null) queue.add(cause); + queue.addAll(Arrays.asList(next.getSuppressed())); + } + return false; + } + + private abstract static class RuntimeContext implements Context { + @Override + public String getXrayTraceId() { + return null; + } + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorShutdownFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorShutdownFailureTest.java new file mode 100644 index 0000000000..2103cf2787 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorShutdownFailureTest.java @@ -0,0 +1,262 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.JacksonSerDes; + +@Timeout(15) +class OwnedExecutorShutdownFailureTest { + enum ShutdownStage { + SHUTDOWN, + INTERRUPTED_FALLBACK, + STATE_QUERY + } + + @SuppressWarnings("removal") + @ParameterizedTest + @CsvSource({ + "false, SHUTDOWN, false", "true, SHUTDOWN, false", + "false, INTERRUPTED_FALLBACK, false", "true, INTERRUPTED_FALLBACK, false", + "false, STATE_QUERY, false", "true, STATE_QUERY, false", + "false, SHUTDOWN, true", "true, SHUTDOWN, true", + "false, INTERRUPTED_FALLBACK, true", "true, INTERRUPTED_FALLBACK, true", + "false, STATE_QUERY, true", "true, STATE_QUERY, true" + }) + void serializationFatalRemainsPrimaryWhenOwnedShutdownIntroducesAnotherFatal( + boolean deathFirst, ShutdownStage stage, boolean inline) throws Exception { + Error first = deathFirst ? new ThreadDeath() : new InternalError("first serializer fatal"); + Error second = deathFirst ? new InternalError("later owned shutdown fatal") : new ThreadDeath(); + var workerFailure = new AtomicReference(); + var escaped = new CountDownLatch(1); + var ends = new AtomicInteger(); + var shutdownCalls = new AtomicInteger(); + var injectFailure = new AtomicBoolean(true); + var callerInterrupted = new AtomicBoolean(); + var owner = new AtomicReference(); + var users = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(), action -> { + var thread = new Thread(action, "owned-fatal-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerFailure.set(failure); + escaped.countDown(); + }); + return thread; + }) { + @Override + public void execute(Runnable command) { + if (inline) command.run(); + else super.execute(command); + } + + @Override + public void shutdown() { + shutdownCalls.incrementAndGet(); + super.shutdown(); + if (injectFailure.get() && stage == ShutdownStage.SHUTDOWN) throw second; + } + + @Override + public List shutdownNow() { + var queued = super.shutdownNow(); + if (injectFailure.get() && stage == ShutdownStage.INTERRUPTED_FALLBACK) throw second; + return queued; + } + + @Override + public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException { + if (injectFailure.get() && stage == ShutdownStage.INTERRUPTED_FALLBACK) { + Thread.currentThread().interrupt(); + new CountDownLatch(1) + .await(); // Observe the interruption even if an inline pool terminated. + } + return super.awaitTermination(timeout, unit); + } + + @Override + public boolean isShutdown() { + var stopped = super.isShutdown(); + if (injectFailure.get() && stage == ShutdownStage.STATE_QUERY && stopped) throw second; + return stopped; + } + }; + var callers = Executors.newSingleThreadExecutor(); + var serializer = new JacksonSerDes() { + @Override + public String serialize(Object value) { + if ("result".equals(value)) throw first; + return super.serialize(value); + } + }; + try { + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .withSerDes(serializer) + .withPlugins(ignoredInvocation -> new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo info) { + owner.set(Thread.currentThread()); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(owner.get(), Thread.currentThread()); + assertEquals(InvocationStatus.RETRYING, info.invocationStatus()); + assertSame(first, info.executionError()); + ends.incrementAndGet(); + } + }) + .build(); + var invocation = callers.submit(() -> { + try { + return DurableExecutor.execute( + input(), + runtime(10_000), + TypeToken.get(String.class), + (value, context) -> "result", + config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertSame( + first, + assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause()); + if (inline) { + assertNull(workerFailure.get(), "The inline fatal belongs to the invocation caller"); + } else { + assertTrue(escaped.await(3, TimeUnit.SECONDS)); + assertSame(first, workerFailure.get()); + } + assertEquals(1, ends.get()); + assertEquals(1, shutdownCalls.get(), "Implicit close must not retry an already-finalized invocation"); + assertEquals(stage == ShutdownStage.INTERRUPTED_FALLBACK, callerInterrupted.get()); + assertTrue(Arrays.asList(first.getSuppressed()).contains(second)); + } finally { + injectFailure.set(false); + callers.shutdownNow(); + users.shutdownNow(); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + @Test + void ownedShutdownFatalOutranksAcknowledgedSuccessAndCallerInterruption() throws Exception { + var fatal = new InternalError("owned shutdown fatal after ACK"); + var injectFailure = new AtomicBoolean(true); + var shutdownCalls = new AtomicInteger(); + var users = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()) { + @Override + public void shutdown() { + shutdownCalls.incrementAndGet(); + super.shutdown(); + if (injectFailure.get()) throw fatal; + } + }; + var callers = Executors.newSingleThreadExecutor(); + var caller = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var endInfo = new AtomicReference(); + var endEntered = new CountDownLatch(1); + var releaseEnd = new CountDownLatch(1); + var client = TestUtils.createMockClient(); + var result = "x".repeat(6 * 1024 * 1024 + 100); // Force the explicit EXECUTION SUCCEED checkpoint. + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> users) + .withDurableExecutionClient(client) + .withPlugins(ignored -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + endInfo.set(info); + endEntered.countDown(); + try { + assertTrue(releaseEnd.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + } + } + }) + .build(); + try { + var invocation = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), runtime(10_000), TypeToken.get(String.class), (value, context) -> result, config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertTrue(endEntered.await(5, TimeUnit.SECONDS)); + assertEquals(InvocationStatus.SUCCEEDED, endInfo.get().invocationStatus()); + assertNull(endInfo.get().executionError()); + caller.get().interrupt(); + assertSame( + fatal, + assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause()); + assertTrue(callerInterrupted.get()); + assertEquals(1, shutdownCalls.get()); + verify(client, times(1)) + .checkpoint( + any(), + any(), + argThat(updates -> updates.size() == 1 + && updates.get(0).type() == OperationType.EXECUTION + && updates.get(0).action() == OperationAction.SUCCEED + && updates.get(0).payload().equals('"' + result + '"'))); + } finally { + releaseEnd.countDown(); + injectFailure.set(false); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static Context runtime(int millis) { + var context = mock(Context.class); + when(context.getRemainingTimeInMillis()).thenReturn(millis); + when(context.getAwsRequestId()).thenReturn("independent-request"); + return context; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/independent/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorTerminationLeaseTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorTerminationLeaseTest.java new file mode 100644 index 0000000000..fd555989a7 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorTerminationLeaseTest.java @@ -0,0 +1,271 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedExecutorTerminationLeaseTest { + enum Path { + GRACEFUL, + SECOND_WAIT, + FORCED + } + + static Stream confirmedCases() { + return Stream.of(Path.values()) + .flatMap(path -> Stream.of(false, true) + .flatMap(queryFatal -> + Stream.of(false, true).map(priorFatal -> Arguments.of(path, queryFatal, priorFatal)))); + } + + @ParameterizedTest + @MethodSource("confirmedCases") + void confirmedTerminationReleasesLeaseEvenWhenLaterStateQueryFails( + Path path, boolean queryFatal, boolean priorFatal) throws Exception { + var firstFatal = new InternalError("first handler fatal"); + Throwable queryFailure = queryFatal + ? new InternalError("state query fatal") + : new IllegalStateException("state query diagnostic"); + var shutdownFailure = new IllegalArgumentException("graceful shutdown failed before delegation"); + var queryEnabled = new AtomicBoolean(true); + var confirmed = new AtomicBoolean(); + var awaits = new AtomicInteger(); + var forced = new AtomicInteger(); + var queriesAfterTermination = new AtomicInteger(); + var tailEntered = new CountDownLatch(1); + var releaseTail = new CountDownLatch(path == Path.SECOND_WAIT ? 1 : 0); + var workerFatal = new AtomicReference(); + var workerEscaped = new CountDownLatch(1); + var pool = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(), task -> { + var thread = new Thread(task, "termination-lease-owner"); + thread.setUncaughtExceptionHandler((ignored, failure) -> { + workerFatal.set(failure); + workerEscaped.countDown(); + }); + return thread; + }) { + @Override + public void shutdown() { + if (path == Path.FORCED && queryEnabled.get()) throw shutdownFailure; + super.shutdown(); + } + + @Override + public List shutdownNow() { + forced.incrementAndGet(); + return super.shutdownNow(); + } + + @Override + protected void afterExecute(Runnable task, Throwable failure) { + tailEntered.countDown(); + try { + assertTrue(releaseTail.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + } + } + + @Override + public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException { + if (path == Path.SECOND_WAIT && awaits.incrementAndGet() == 1) { + assertTrue(tailEntered.await(3, TimeUnit.SECONDS)); + var stopped = super.awaitTermination(0, TimeUnit.NANOSECONDS); + assertFalse(stopped, "The actual worker tail is still held at the first wait"); + releaseTail.countDown(); + return stopped; + } + var stopped = super.awaitTermination(timeout, unit); + if (stopped) confirmed.set(true); + return stopped; + } + + @Override + public boolean isShutdown() { + if (queryEnabled.get() && confirmed.get()) { + queriesAfterTermination.incrementAndGet(); + ExceptionHelper.sneakyThrow(queryFailure); + } + return super.isShutdown(); + } + }; + assertEquals(1, pool.prestartAllCoreThreads()); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> pool) + .build(); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + if (priorFatal) throw firstFatal; + return value; + }, + config)); + if (priorFatal) assertSame(firstFatal, observed); + else if (queryFatal || path != Path.FORCED) assertSame(queryFailure, observed); + else + assertSame( + shutdownFailure, + assertInstanceOf(UnrecoverableDurableExecutionException.class, observed) + .getCause()); + assertTrue(contains(observed, queryFailure), "The state-query diagnostic must remain visible"); + if (path == Path.FORCED) assertTrue(contains(observed, shutdownFailure)); + assertTrue(confirmed.get(), "Termination must have been confirmed by a real pool wait"); + assertTrue(pool.isTerminated()); + assertEquals(1, queriesAfterTermination.get(), "The later diagnostic query still runs"); + assertEquals(path == Path.GRACEFUL ? 0 : 1, forced.get()); + if (priorFatal) { + assertTrue(workerEscaped.await(3, TimeUnit.SECONDS)); + assertSame(firstFatal, workerFatal.get()); + } + + queryEnabled.set(false); + var rejected = assertThrows( + IllegalStateException.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> fail("must not dispatch"), + config)); + assertEquals( + "Invocation executor factory returned a shutdown executor", + rejected.getMessage(), + "Released lease allows closed-state validation instead of the stale already-in-use rejection"); + } finally { + queryEnabled.set(false); + releaseTail.countDown(); + pool.shutdownNow(); + assertTrue(pool.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void unconfirmedOrActiveExecutorKeepsItsLease(boolean stillActive) throws Exception { + var injected = new AtomicBoolean(true); + var attempted = new AtomicBoolean(); + var confirms = new AtomicInteger(); + var queries = new AtomicInteger(); + var awaitFailure = new IllegalArgumentException("termination not observed"); + var queryFailure = new IllegalStateException("state unknown"); + var pool = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void shutdown() { + attempted.set(true); + if (!injected.get() || !stillActive) super.shutdown(); + } + + @Override + public List shutdownNow() { + return injected.get() && stillActive ? List.of() : super.shutdownNow(); + } + + @Override + public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException { + if (injected.get() && !stillActive) throw awaitFailure; + var stopped = super.awaitTermination(injected.get() ? 0 : timeout, unit); + if (stopped) confirms.incrementAndGet(); + return stopped; + } + + @Override + public boolean isShutdown() { + if (injected.get() && attempted.get()) { + queries.incrementAndGet(); + throw queryFailure; + } + return super.isShutdown(); + } + }; + pool.prestartAllCoreThreads(); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> pool) + .build(); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), null, TypeToken.get(String.class), (value, context) -> value, config)); + assertTrue(contains(observed, queryFailure)); + if (!stillActive) assertTrue(contains(observed, awaitFailure)); + assertEquals(0, confirms.get()); + if (stillActive) assertFalse(pool.isTerminated()); + var queryCount = queries.get(); + var rejected = assertThrows( + IllegalStateException.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> fail("must not dispatch"), + config)); + assertEquals("Invocation executor factory returned an executor already in use", rejected.getMessage()); + assertEquals(queryCount, queries.get(), "An existing lease must be rejected before any state callback"); + } finally { + injected.set(false); + pool.shutdownNow(); + assertTrue(pool.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static boolean contains(Throwable root, Throwable target) { + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var next = pending.remove(); + if (next == target) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) pending.add(next.getCause()); + Collections.addAll(pending, next.getSuppressed()); + } + return false; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/lease", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("lease") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorWrappedFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorWrappedFailureTest.java new file mode 100644 index 0000000000..a8fadaafe9 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedExecutorWrappedFailureTest.java @@ -0,0 +1,130 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class OwnedExecutorWrappedFailureTest { + @ParameterizedTest + @CsvSource({"false, false, true", "true, false, true", "false, true, true", "true, true, true", "true, false, false" + }) + void wrappedOwnedShutdownFailureRetainsIdentity(boolean executionWrapper, boolean acknowledged, boolean fatalCause) + throws Exception { + Throwable cause = fatalCause + ? new InternalError("wrapped owned shutdown fatal") + : new IllegalStateException("ordinary application failure"); + Throwable wrapped = executionWrapper ? new ExecutionException(cause) : new CompletionException(cause); + var injectFailure = new AtomicBoolean(true); + var shutdownCalls = new AtomicInteger(); + var users = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()) { + @Override + public void shutdown() { + shutdownCalls.incrementAndGet(); + super.shutdown(); + if (injectFailure.get()) ExceptionHelper.sneakyThrow(wrapped); + } + }; + var callers = Executors.newSingleThreadExecutor(); + var caller = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var endInfo = new AtomicReference(); + var endEntered = new CountDownLatch(1); + var releaseEnd = new CountDownLatch(acknowledged ? 1 : 0); + var client = TestUtils.createMockClient(); + var result = acknowledged ? "x".repeat(6 * 1024 * 1024 + 100) : "result"; + var config = DurableConfig.builder() + .withInvocationExecutorFactory(() -> users) + .withDurableExecutionClient(client) + .withPlugins(ignored -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + endInfo.set(info); + endEntered.countDown(); + try { + assertTrue(releaseEnd.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + } + } + }) + .build(); + try { + var invocation = callers.submit(() -> { + caller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), runtime(10_000), TypeToken.get(String.class), (value, context) -> result, config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertTrue(endEntered.await(5, TimeUnit.SECONDS)); + assertEquals(InvocationStatus.SUCCEEDED, endInfo.get().invocationStatus()); + assertNull(endInfo.get().executionError()); + if (acknowledged) caller.get().interrupt(); + assertSame( + fatalCause ? cause : wrapped, + assertThrows(ExecutionException.class, () -> invocation.get(5, TimeUnit.SECONDS)) + .getCause()); + assertEquals(acknowledged, callerInterrupted.get()); + assertEquals(1, shutdownCalls.get()); + if (acknowledged) + verify(client, times(1)) + .checkpoint( + any(), + any(), + argThat(updates -> updates.size() == 1 + && updates.get(0).type() == OperationType.EXECUTION + && updates.get(0).action() == OperationAction.SUCCEED + && updates.get(0).payload().equals('"' + result + '"'))); + } finally { + releaseEnd.countDown(); + injectFailure.set(false); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(3, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static Context runtime(int millis) { + var context = mock(Context.class); + when(context.getRemainingTimeInMillis()).thenReturn(millis); + when(context.getAwsRequestId()).thenReturn("independent-request"); + return context; + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/independent/exec", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("exec") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedInvocationExecutorLifecycleTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedInvocationExecutorLifecycleTest.java new file mode 100644 index 0000000000..5a511a1f4c --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnedInvocationExecutorLifecycleTest.java @@ -0,0 +1,286 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotSame; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.List; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.CheckpointUpdatedExecutionState; +import software.amazon.awssdk.services.lambda.model.ExecutionDetails; +import software.amazon.awssdk.services.lambda.model.Operation; +import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.awssdk.services.lambda.model.OperationType; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; + +class OwnedInvocationExecutorLifecycleTest { + @Test + void callerShutsDownOwnedExecutorAfterHandlerEndReturns() throws Exception { + var ownedThreads = ConcurrentHashMap.newKeySet(); + var owned = Executors.newFixedThreadPool(2, action -> { + var thread = new Thread(action, "invocation-owned"); + ownedThreads.add(thread); + return thread; + }); + var runtime = Executors.newSingleThreadExecutor(); + var startThread = new AtomicReference(); + var stepThread = new AtomicReference(); + var endThread = new AtomicReference(); + var callerThread = new AtomicReference(); + var endEntered = new CountDownLatch(1); + var releaseEnd = new CountDownLatch(1); + var plugin = new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo info) { + startThread.set(Thread.currentThread()); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + endThread.set(Thread.currentThread()); + endEntered.countDown(); + await(releaseEnd); + } + }; + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned) + .withPlugins(ignoredInfo -> plugin) + .build(); + try { + var response = runtime.submit(() -> { + callerThread.set(Thread.currentThread()); + return DurableExecutor.execute( + input(), + context(), + TypeToken.get(String.class), + (value, durable) -> durable.step("step", String.class, step -> { + stepThread.set(Thread.currentThread()); + return value; + }), + config); + }); + assertTrue(endEntered.await(5, TimeUnit.SECONDS)); + assertSame(startThread.get(), endThread.get()); + assertTrue(ownedThreads.contains(startThread.get()), "Plugin-enabled roots must use the owned executor"); + assertTrue(ownedThreads.contains(stepThread.get()), "Operation wrappers must use the owned executor"); + assertNotSame(callerThread.get(), endThread.get()); + assertFalse(owned.isShutdown(), "The handler-owned End hook must run before executor shutdown"); + assertFalse(response.isDone(), "The runtime caller must wait for End"); + releaseEnd.countDown(); + assertEquals( + ExecutionStatus.SUCCEEDED, response.get(5, TimeUnit.SECONDS).status()); + assertTrue(owned.isTerminated(), "Executor shutdown must not wait for its own root worker"); + } finally { + releaseEnd.countDown(); + stop(runtime); + stop(owned); + } + } + + @Test + void interruptedCallerStillWaitsForEndBeforeShuttingDownOwnedExecutor() throws Exception { + var owned = Executors.newFixedThreadPool(2); + var runtime = Executors.newSingleThreadExecutor(); + var callerThread = new AtomicReference(); + var callerInterrupted = new AtomicBoolean(); + var endEntered = new CountDownLatch(1); + var endInterrupted = new CountDownLatch(1); + var releaseEnd = new CountDownLatch(1); + var config = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned) + .withPlugins(ignoredInfo -> new DurableExecutionPlugin() { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + endEntered.countDown(); + awaitIgnoringInterruptions(releaseEnd, endInterrupted); + } + }) + .build(); + try { + var response = runtime.submit(() -> { + callerThread.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input(), context(), TypeToken.get(String.class), (value, durable) -> value, config); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + assertTrue(endEntered.await(5, TimeUnit.SECONDS)); + callerThread.get().interrupt(); + assertTrue(endInterrupted.await(5, TimeUnit.SECONDS)); + assertThrows(TimeoutException.class, () -> response.get(1200, TimeUnit.MILLISECONDS)); + assertFalse(owned.isShutdown(), "Cancellation cannot move executor shutdown ahead of actual End exit"); + releaseEnd.countDown(); + var failure = assertInstanceOf( + UnrecoverableDurableExecutionException.class, + assertThrows(ExecutionException.class, () -> response.get(5, TimeUnit.SECONDS)) + .getCause()); + assertTrue(failure.isRetryable()); + assertInstanceOf(InterruptedException.class, failure.getCause()); + assertTrue(callerInterrupted.get()); + assertTrue(owned.isTerminated()); + } finally { + releaseEnd.countDown(); + stop(runtime); + stop(owned); + } + } + + private static void awaitIgnoringInterruptions(CountDownLatch release, CountDownLatch interrupted) { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (true) { + try { + assertTrue(release.await(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + return; + } catch (InterruptedException ignored) { + interrupted.countDown(); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void rootStepAndNestedChildUseTheInvocationSelectedExecutor(boolean pluginsEnabled) throws Exception { + var ownedThreads = ConcurrentHashMap.newKeySet(); + var observedThreads = new CopyOnWriteArrayList(); + // Root, child and nested step can wait on each other; give this scoped routing test enough capacity. + var owned = Executors.newFixedThreadPool(4, action -> { + var thread = new Thread(action, "routing-owned"); + ownedThreads.add(thread); + return thread; + }); + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned); + if (pluginsEnabled) builder.withPlugins(ignoredInfo -> new DurableExecutionPlugin() {}); + try { + var response = DurableExecutor.execute( + input(), + context(), + TypeToken.get(String.class), + (value, durable) -> { + observedThreads.add(Thread.currentThread()); + durable.step("root-step", String.class, step -> { + observedThreads.add(Thread.currentThread()); + return value; + }); + return durable.runInChildContext("child", String.class, child -> { + observedThreads.add(Thread.currentThread()); + return child.step("nested-step", String.class, step -> { + observedThreads.add(Thread.currentThread()); + return value; + }); + }); + }, + builder.build()); + assertEquals(ExecutionStatus.SUCCEEDED, response.status()); + assertEquals("\"input\"", response.result()); + assertEquals(4, observedThreads.size()); + assertTrue( + ownedThreads.containsAll(observedThreads), + "Root, step, child and nested operation code must all use the invocation-selected executor"); + assertTrue(owned.isTerminated()); + } finally { + stop(owned); + } + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void rejectedRootStillShutsDownInvocationOwnedExecutor(boolean pluginsEnabled) throws Exception { + var owned = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>()) { + @Override + public void execute(Runnable command) { + throw new RejectedExecutionException("test root submission rejection"); + } + }; + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> owned); + if (pluginsEnabled) builder.withPlugins(ignoredInfo -> new DurableExecutionPlugin() {}); + var config = builder.build(); + try { + assertThrows( + RuntimeException.class, + () -> DurableExecutor.execute( + input(), context(), TypeToken.get(String.class), (value, durable) -> value, config)); + assertTrue(owned.isTerminated()); + } finally { + stop(owned); + } + } + + private static DurableExecutionInput input() { + var operation = Operation.builder() + .id("execution") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.now()) + .executionDetails( + ExecutionDetails.builder().inputPayload("\"input\"").build()) + .build(); + return new DurableExecutionInput( + "arn:aws:lambda:us-east-1:123456789012:function:test/durable-execution/test/execution", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(List.of(operation)) + .build()); + } + + private static Context context() { + var context = mock(Context.class); + when(context.getAwsRequestId()).thenReturn("owned-executor-request"); + when(context.getRemainingTimeInMillis()).thenReturn(20_000); + return context; + } + + private static void await(CountDownLatch latch) { + try { + if (!latch.await(5, TimeUnit.SECONDS)) throw new IllegalStateException("Test gate timed out"); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new IllegalStateException(interrupted); + } + } + + private static void stop(ExecutorService executor) throws InterruptedException { + executor.shutdownNow(); + assertTrue(executor.awaitTermination(5, TimeUnit.SECONDS)); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnerDrainLifecycleFailureTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnerDrainLifecycleFailureTest.java new file mode 100644 index 0000000000..2582eb24d0 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/OwnerDrainLifecycleFailureTest.java @@ -0,0 +1,319 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import com.amazonaws.services.lambda.runtime.Context; +import java.time.Instant; +import java.util.*; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.function.BooleanSupplier; +import java.util.function.Supplier; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.model.ExecutionStatus; +import software.amazon.lambda.durable.plugin.*; + +@Timeout(20) +class OwnerDrainLifecycleFailureTest { + enum Stop { + CALLER, + EXPIRY, + QUEUED, + QUEUED_OWNER, + OWNER, + NORMAL, + ALIAS, + TRANSPORT, + FATAL + } + + @ParameterizedTest + @CsvSource({ + "CALLER,true", + "CALLER,false", + "EXPIRY,true", + "QUEUED,true", + "QUEUED_OWNER,true", + "OWNER,true", + "NORMAL,true", + "ALIAS,true", + "TRANSPORT,true", + "FATAL,true" + }) + void ownerDrainKeepsTheRightFailureAndActualExitBoundary(Stop stop, boolean plugins) throws Exception { + var rootOwner = new AtomicReference(); + var runtimeCaller = new AtomicReference(); + var manager = new AtomicReference(); + var end = new AtomicReference(); + var endCalls = new AtomicInteger(); + var stepBodies = new AtomicInteger(); + var stepExited = new AtomicBoolean(); + var stepEntered = new CountDownLatch(1); + var stepInterrupted = new CountDownLatch(1); + var releaseStep = new CountDownLatch(1); + var rootReturning = new CountDownLatch(1); + var ignoredRootInterrupt = new CountDownLatch(1); + var uncaught = new AtomicReference(); + var fatalEscaped = new CountDownLatch(1); + var callerInterrupted = new AtomicBoolean(); + var suppressRootInterrupt = new AtomicBoolean(stop == Stop.QUEUED || stop == Stop.TRANSPORT); + var queued = stop == Stop.QUEUED || stop == Stop.QUEUED_OWNER; + var transport = new IllegalStateException( + "transport cleanup interrupted", new InterruptedException("transport interruption")); + var fatal = new InternalError("fatal transport cleanup"); + ThreadFactory threadFactory = command -> { + var thread = new Thread(command, "owner-drain-worker") { + @Override + public void interrupt() { + if (suppressRootInterrupt.get() && this == rootOwner.get()) { + ignoredRootInterrupt.countDown(); + return; + } + super.interrupt(); + } + }; + thread.setUncaughtExceptionHandler((owner, failure) -> { + uncaught.set(failure); + fatalEscaped.countDown(); + }); + return thread; + }; + var users = queued + ? new ThreadPoolExecutor( + 1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(), threadFactory) + : new ThreadPoolExecutor( + 0, Integer.MAX_VALUE, 60, TimeUnit.SECONDS, new SynchronousQueue(), threadFactory); + var callers = Executors.newSingleThreadExecutor(); + var builder = DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()); + if (plugins) + builder.withPlugins(info -> new DurableExecutionPlugin() { + public void onInvocationEnd(InvocationEndInfo info) { + assertSame(rootOwner.get(), Thread.currentThread()); + if (!queued) assertTrue(stepExited.get(), "End cannot precede actual step exit"); + end.set(info); + endCalls.incrementAndGet(); + } + }); + var runtime = mock(Context.class); + when(runtime.getRemainingTimeInMillis()).thenReturn(stop == Stop.EXPIRY ? 2500 : 20_000); + var input = new DurableExecutionInput( + "arn:execution/test", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("test") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + try { + var response = callers.submit(() -> { + runtimeCaller.set(Thread.currentThread()); + try { + return DurableExecutor.execute( + input, + runtime, + TypeToken.get(String.class), + (value, root) -> { + rootOwner.set(Thread.currentThread()); + var execution = ((DurableContextImpl) root).getExecutionManager(); + manager.set(execution); + if (stop == Stop.ALIAS || stop == Stop.TRANSPORT || stop == Stop.FATAL) + installShutdownFault( + execution, + rootOwner, + () -> stop == Stop.ALIAS + ? new CompletionException(execution.getLifecycleFailure()) + : stop == Stop.FATAL ? fatal : transport); + root.stepAsync("held", String.class, step -> { + stepBodies.incrementAndGet(); + stepEntered.countDown(); + try { + awaitIgnoringInterrupt(releaseStep, stepInterrupted); + return "step"; + } finally { + stepExited.set(true); + } + }); + if (!queued) await(stepEntered); + rootReturning.countDown(); + return "root-result"; + }, + builder.build()); + } finally { + callerInterrupted.set(Thread.interrupted()); + } + }); + await(rootReturning); + awaitCondition(() -> inMethod(rootOwner.get(), "awaitUserHandler")); + if (stop == Stop.NORMAL) { + assertEquals(0, endCalls.get()); + releaseStep.countDown(); + assertEquals( + ExecutionStatus.SUCCEEDED, + response.get(5, TimeUnit.SECONDS).status()); + assertEquals(InvocationStatus.SUCCEEDED, end.get().invocationStatus()); + assertNull(end.get().executionError()); + } else { + if (stop == Stop.OWNER) { + rootOwner.get().interrupt(); + awaitCondition( + () -> !rootOwner.get().isInterrupted() && inMethod(rootOwner.get(), "awaitUserHandler")); + } else if (stop == Stop.QUEUED_OWNER) { + var stepTask = manager.get().getActiveExecutorTasks().stream() + .filter(task -> task.role() == ExecutorTaskHandle.Role.STEP) + .findFirst() + .orElseThrow(); + assertTrue(stepTask.cancel(false)); + } else { + if (stop != Stop.EXPIRY) runtimeCaller.get().interrupt(); + if (stop == Stop.QUEUED) await(ignoredRootInterrupt); + else await(stepInterrupted); + } + if (!queued) { + assertThrows(TimeoutException.class, () -> response.get(100, TimeUnit.MILLISECONDS)); + assertEquals(0, endCalls.get()); + releaseStep.countDown(); + } + var observed = assertThrows(ExecutionException.class, () -> response.get(5, TimeUnit.SECONDS)) + .getCause(); + var lifecycle = manager.get().getLifecycleFailure(); + if (stop == Stop.FATAL) { + assertSame(fatal, observed); + assertSame(fatal, end.get().executionError()); + await(fatalEscaped); + assertSame(fatal, uncaught.get()); + } else if (stop == Stop.TRANSPORT) { + assertSame(lifecycle, observed); + assertSame( + transport, + end.get().executionError(), + "An unrelated transport failure is not a synthetic owner cancellation"); + } else if (stop == Stop.OWNER || stop == Stop.QUEUED_OWNER) { + assertNull(lifecycle); + assertEquals(IllegalStateException.class, observed.getClass()); + assertSame(observed, end.get().executionError()); + assertTrue( + stop == Stop.OWNER + ? observed.getCause() instanceof InterruptedException + : observed.getCause() instanceof CancellationException); + } else { + assertNotNull(lifecycle); + assertSame(lifecycle, observed); + if (plugins) + assertSame( + lifecycle, end.get().executionError(), "End must retain the original lifecycle cause"); + var diagnostic = Arrays.stream(lifecycle.getSuppressed()) + .filter(failure -> failure.getClass() == IllegalStateException.class + && (failure.getCause() instanceof InterruptedException + || failure.getCause() instanceof CancellationException)) + .toList(); + assertEquals(1, diagnostic.size(), "Keep the original synthetic drain error exactly once"); + assertFalse( + references(diagnostic.get(0), lifecycle), + "The diagnostic must not reference its canonical lifecycle owner"); + } + if (plugins) assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + } + assertEquals(plugins ? 1 : 0, endCalls.get()); + assertEquals(queued ? 0 : 1, stepBodies.get()); + assertEquals( + stop != Stop.NORMAL && stop != Stop.OWNER && stop != Stop.QUEUED_OWNER && stop != Stop.EXPIRY, + callerInterrupted.get()); + } finally { + releaseStep.countDown(); + suppressRootInterrupt.set(false); + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static void installShutdownFault( + ExecutionManager manager, AtomicReference owner, Supplier fault) { + try { + var field = ExecutionManager.class.getDeclaredField("checkpointManager"); + field.setAccessible(true); + var checkpoint = spy((CheckpointManager) field.get(manager)); + doAnswer(call -> { + if (Thread.currentThread() != owner.get()) return call.callRealMethod(); + try { + call.callRealMethod(); + } finally { + throw fault.get(); + } + }) + .when(checkpoint) + .shutdown(); + field.set(manager, checkpoint); + } catch (ReflectiveOperationException failure) { + throw new AssertionError(failure); + } + } + + private static boolean references(Throwable root, Throwable target) { + var pending = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + pending.add(root); + while (!pending.isEmpty()) { + var next = pending.remove(); + if (next == target) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) pending.add(next.getCause()); + pending.addAll(Arrays.asList(next.getSuppressed())); + } + return false; + } + + private static boolean inMethod(Thread thread, String name) { + return thread != null + && Arrays.stream(thread.getStackTrace()) + .anyMatch(frame -> frame.getMethodName().equals(name)); + } + + private static void awaitCondition(BooleanSupplier condition) throws InterruptedException { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (!condition.getAsBoolean()) { + assertTrue(System.nanoTime() < deadline); + TimeUnit.MILLISECONDS.sleep(5); + } + } + + private static void await(CountDownLatch gate) { + try { + assertTrue(gate.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException failure) { + throw new AssertionError(failure); + } + } + + private static void awaitIgnoringInterrupt(CountDownLatch gate, CountDownLatch interrupted) { + var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (true) { + try { + assertTrue(gate.await(Math.max(0, deadline - System.nanoTime()), TimeUnit.NANOSECONDS)); + return; + } catch (InterruptedException ignored) { + interrupted.countDown(); + } + } + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/RecordedFatalCleanupTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/RecordedFatalCleanupTest.java new file mode 100644 index 0000000000..f2d475855f --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/RecordedFatalCleanupTest.java @@ -0,0 +1,220 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Collections; +import java.util.IdentityHashMap; +import java.util.Set; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import java.util.stream.Stream; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.TypeToken; +import software.amazon.lambda.durable.context.DurableContextImpl; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.plugin.*; +import software.amazon.lambda.durable.serde.SerDes; +import software.amazon.lambda.durable.util.ExceptionHelper; + +@Timeout(15) +class RecordedFatalCleanupTest { + enum Cleanup { + NONE, + ORDINARY, + FATAL, + WRAPPED_FATAL, + SAME_FATAL, + WRAPPED_SAME_FATAL, + WRAPPER_DIAGNOSTICS + } + + static Stream cases() { + return Stream.of(Cleanup.values()) + .flatMap(cleanup -> Stream.of(false, true) + .flatMap( + plugins -> Stream.of(false, true).map(death -> Arguments.of(cleanup, plugins, death)))); + } + + @SuppressWarnings("removal") + @ParameterizedTest + @MethodSource("cases") + void recordedSerializerFatalOwnsCleanupDiagnosticsWithoutCycles(Cleanup cleanup, boolean plugins, boolean death) + throws Exception { + Error first = death ? new ThreadDeath() : new InternalError("fatal output serialization"); + var ordinary = new IllegalStateException("ordinary owned shutdown failure"); + var laterFatal = new InternalError("later owned shutdown fatal"); + var wrapperDiagnostic = new IllegalArgumentException("transport-wrapper diagnostic"); + var nestedDiagnostic = new IllegalArgumentException("nested observation diagnostic"); + Throwable shutdownFailure = switch (cleanup) { + case NONE -> null; + case ORDINARY, WRAPPER_DIAGNOSTICS -> ordinary; + case FATAL -> laterFatal; + case WRAPPED_FATAL -> new CompletionException(laterFatal); + case SAME_FATAL -> first; + case WRAPPED_SAME_FATAL -> new CompletionException(first); + }; + var bodies = new AtomicInteger(); + var serialized = new AtomicInteger(); + var shutdowns = new AtomicInteger(); + var ends = new AtomicInteger(); + var end = new AtomicReference(); + var manager = new AtomicReference(); + var fatalAtShutdown = new AtomicReference(); + var workerFatal = new AtomicReference(); + var workerEscaped = new CountDownLatch(1); + var pool = + new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(), task -> { + var worker = new Thread(task, "recorded-fatal-owner"); + worker.setUncaughtExceptionHandler((ignored, failure) -> { + workerFatal.set(failure); + workerEscaped.countDown(); + }); + return worker; + }) { + @Override + public void shutdown() { + shutdowns.incrementAndGet(); + fatalAtShutdown.set(manager.get().getInvocationFatal()); + super.shutdown(); + if (shutdownFailure != null) ExceptionHelper.sneakyThrow(shutdownFailure); + } + }; + pool.prestartAllCoreThreads(); + var builder = DurableConfig.builder() + .withDurableExecutionClient(TestUtils.createMockClient()) + .withInvocationExecutorFactory(() -> pool) + .withSerDes(new SerDes() { + public String serialize(Object value) { + serialized.incrementAndGet(); + throw first; + } + + @SuppressWarnings("unchecked") + public T deserialize(String value, TypeToken type) { + return (T) "input"; + } + }); + if (plugins) + builder.withPlugins(ignored -> new DurableExecutionPlugin() { + public void onInvocationEnd(InvocationEndInfo info) { + ends.incrementAndGet(); + end.set(info); + } + }); + try { + var observed = assertThrows( + Throwable.class, + () -> DurableExecutor.execute( + input(), + null, + TypeToken.get(String.class), + (value, context) -> { + manager.set(((DurableContextImpl) context).getExecutionManager()); + if (cleanup == Cleanup.WRAPPER_DIAGNOSTICS) + assertDoesNotThrow(() -> decorateShutdownObservation( + manager.get(), first, wrapperDiagnostic, nestedDiagnostic)); + bodies.incrementAndGet(); + return "output"; + }, + builder.build())); + assertSame(first, observed, "The first recorded fatal must remain the caller outcome"); + assertTrue(workerEscaped.await(3, TimeUnit.SECONDS)); + assertSame(first, workerFatal.get()); + assertSame(first, fatalAtShutdown.get(), "This fatal was recorded before owned executor cleanup"); + assertEquals(1, bodies.get()); + assertEquals(1, serialized.get()); + assertEquals(1, shutdowns.get()); + assertTrue(pool.isTerminated()); + assertEquals(plugins ? 1 : 0, ends.get()); + if (plugins) { + assertEquals(InvocationStatus.RETRYING, end.get().invocationStatus()); + assertSame(first, end.get().executionError()); + } + if (cleanup == Cleanup.ORDINARY || cleanup == Cleanup.WRAPPER_DIAGNOSTICS) + assertTrue(contains(first, ordinary), "Retain the ordinary shutdown diagnostic on the recorded fatal"); + if (cleanup == Cleanup.WRAPPER_DIAGNOSTICS) { + assertTrue(contains(first, wrapperDiagnostic)); + assertTrue(contains(first, nestedDiagnostic)); + } + if (cleanup == Cleanup.FATAL || cleanup == Cleanup.WRAPPED_FATAL) + assertTrue(contains(first, laterFatal), "Retain the later cleanup fatal"); + assertAcyclic(first, Collections.newSetFromMap(new IdentityHashMap())); + } finally { + pool.shutdownNow(); + assertTrue(pool.awaitTermination(3, TimeUnit.SECONDS)); + } + } + + private static void decorateShutdownObservation( + ExecutionManager manager, Error fatal, Throwable diagnostic, Throwable nestedDiagnostic) + throws ReflectiveOperationException { + var field = ExecutionManager.class.getDeclaredField("checkpointManager"); + field.setAccessible(true); + var checkpointManager = spy((CheckpointManager) field.get(manager)); + doAnswer(call -> { + try { + call.callRealMethod(); + return null; + } catch (CompletionException observation) { + assertSame(fatal, observation.getCause(), "Decorate the actual shutdown observation"); + observation.addSuppressed(diagnostic); + var nested = new CompletionException(fatal); + nested.addSuppressed(nestedDiagnostic); + observation.addSuppressed(nested); + throw observation; + } + }) + .when(checkpointManager) + .shutdown(); + field.set(manager, checkpointManager); + } + + private static boolean contains(Throwable root, Throwable expected) { + var queue = new ArrayDeque(); + var seen = Collections.newSetFromMap(new IdentityHashMap()); + queue.add(root); + while (!queue.isEmpty()) { + var next = queue.remove(); + if (next == expected) return true; + if (!seen.add(next)) continue; + if (next.getCause() != null) queue.add(next.getCause()); + Collections.addAll(queue, next.getSuppressed()); + } + return false; + } + + private static void assertAcyclic(Throwable failure, Set path) { + assertTrue(path.add(failure), "Cleanup diagnostics must not point back to the selected fatal"); + if (failure.getCause() != null) assertAcyclic(failure.getCause(), path); + for (var additional : failure.getSuppressed()) assertAcyclic(additional, path); + path.remove(failure); + } + + private static DurableExecutionInput input() { + return new DurableExecutionInput( + "arn:execution/recorded-fatal", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("recorded-fatal") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .executionDetails(ExecutionDetails.builder() + .inputPayload("\"input\"") + .build()) + .build()) + .build()); + } +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/execution/TaskAdmissionFenceTest.java b/sdk/src/test/java/software/amazon/lambda/durable/execution/TaskAdmissionFenceTest.java new file mode 100644 index 0000000000..caa545c930 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/execution/TaskAdmissionFenceTest.java @@ -0,0 +1,126 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.execution; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.*; + +import java.time.Instant; +import java.util.concurrent.*; +import java.util.concurrent.atomic.*; +import org.junit.jupiter.api.Timeout; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import software.amazon.awssdk.services.lambda.model.*; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.TestUtils; +import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; +import software.amazon.lambda.durable.model.DurableExecutionInput; +import software.amazon.lambda.durable.operation.BaseDurableOperation; + +@Timeout(15) +class TaskAdmissionFenceTest { + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void registeredOperationCannotDispatchAfterLifecycleFence(boolean duringRegistration) throws Exception { + var actionCalls = new AtomicInteger(); + var dispatches = new AtomicInteger(); + var lookups = new AtomicInteger(); + var submitter = new AtomicReference(); + var lookupEntered = new CountDownLatch(1); + var releaseLookup = new CountDownLatch(1); + var users = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, new LinkedBlockingQueue()) { + @Override + public void execute(Runnable action) { + dispatches.incrementAndGet(); + super.execute(action); + } + }; + var callers = Executors.newSingleThreadExecutor(); + var input = new DurableExecutionInput( + "arn:execution/test", + "token", + CheckpointUpdatedExecutionState.builder() + .operations(Operation.builder() + .id("test") + .type(OperationType.EXECUTION) + .status(OperationStatus.STARTED) + .startTimestamp(Instant.EPOCH) + .build()) + .build()); + var manager = new ExecutionManager( + input, + DurableConfig.builder() + .withExecutorService(users) + .withDurableExecutionClient(TestUtils.createMockClient()) + .build(), + null); + var operation = mock(BaseDurableOperation.class); + when(operation.getOperationId()).thenAnswer(ignored -> { + if (duringRegistration && Thread.currentThread() == submitter.get() && lookups.incrementAndGet() == 2) { + lookupEntered.countDown(); + try { + assertTrue(releaseLookup.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException failure) { + throw new AssertionError(failure); + } + } + return "registered-step"; + }); + when(operation.getType()).thenReturn(OperationType.STEP); + when(operation.getCompletionFuture()).thenReturn(new CompletableFuture<>()); + manager.registerOperation(operation); + manager.beginDraining(); + Future submission = null; + UnrecoverableDurableExecutionException fence = null; + try { + if (!duringRegistration) fence = selectInterruption(manager); + submission = callers.submit(() -> { + submitter.set(Thread.currentThread()); + manager.submitOperationTask(operation, actionCalls::incrementAndGet) + .join(); + }); + if (duringRegistration) { + assertTrue(lookupEntered.await(5, TimeUnit.SECONDS)); + fence = selectInterruption(manager); + releaseLookup.countDown(); + } + var observed = submission; + var failure = assertThrows(ExecutionException.class, () -> observed.get(5, TimeUnit.SECONDS)); + assertSame(fence, failure.getCause()); + assertEquals(0, dispatches.get(), "A task fenced during registration must not be offered to the executor"); + assertEquals(0, actionCalls.get()); + assertTrue( + manager.getActiveExecutorTasks().isEmpty(), + "Rejected registration must release actual-exit tracking"); + } finally { + releaseLookup.countDown(); + if (submission != null) { + try { + submission.get(5, TimeUnit.SECONDS); + } catch (ExecutionException expected) { + } + } + try { + manager.close(); + } catch (UnrecoverableDurableExecutionException expected) { + } finally { + Thread.interrupted(); + } + users.shutdownNow(); + callers.shutdownNow(); + assertTrue(users.awaitTermination(5, TimeUnit.SECONDS)); + assertTrue(callers.awaitTermination(5, TimeUnit.SECONDS)); + } + } + + private static UnrecoverableDurableExecutionException selectInterruption(ExecutionManager manager) { + Thread.currentThread().interrupt(); + var selected = + manager.awaitInvocationOutcome(new CompletableFuture()).failure(); + assertFalse( + Thread.currentThread().isInterrupted(), + "The blocking observation consumes the interrupt before cleanup"); + return assertInstanceOf(UnrecoverableDurableExecutionException.class, selected); + } +}