[Feature]: Add LMI cloud end-to-end tests for concurrency, lifecycle and OpenTelemetry #727

What would you like?

Add a dedicated end-to-end cloud test suite for the Java Durable Execution SDK on Lambda Managed Instances (LMI), including PerExecutionEnvironmentMaxConcurrency > 1, OpenTelemetry invocation isolation, and invocation-local X-Ray propagation.

This is the cloud-validation follow-up to #726. That issue contains local reproductions of unbounded invocation cleanup, a PENDING response preceding root-handler cleanup, and starvation when concurrent handlers share a fixed executor. Real LMI tests are needed to validate the runtime behavior, the eventual fixes, and future regressions.

Observability validation ownership: This open testing issue now owns the remaining deployed-LMI acceptance for OTel isolation #761 and X-Ray carrier #762. Their runtime implementations landed in #782; #765 added documentation clarifications. The LMI harness from #728 and task registry from #731 are also incorporated into #782. Those merges and closed defect issues do not establish fresh LMI acceptance. Validate compatible Java 3.x core/plugin/provider/layer artifacts together with the intended lifecycle/executor changes in #733/#735. Coordinate with JS #1017 under umbrella #798.

At the time of the original report, the existing E2E workflow deploys examples and runs CloudBasedIntegrationTest with a Java 17/21/25 matrix and JUnit parallelism. It does not configure an LMI-specific deployment or prove that concurrent invocations share one Java execution environment. Driver-side parallelism alone cannot exercise that requirement.

LMI-specific behavior to cover:

• Multiple OS runtime threads share the handler object and process-wide resources.
• Execution environments remain active between invocations.
• An invocation timeout does not forcibly terminate the function's code, so task cancellation, cleanup, and worker-slot recovery require explicit validation.

References: Java runtime for LMI and LMI execution environment lifecycle.

Implementation PR for the added observability scope: Java #823.

Possible Implementation

1. Deploy a dedicated LMI test stack.

Add repeatable infrastructure and small purpose-built handlers, following the existing example/cloud-test structure where practical:

• A test capacity provider and required networking/roles, or explicit parameters for an existing dedicated test capacity provider.
• Durable-enabled function versions/aliases configured for LMI, with the deployed version, runtime, capacity-provider association, and PerExecutionEnvironmentMaxConcurrency recorded in test artifacts.
• Separate fixtures/configurations for concurrency 1, 2, and a modest higher value such as 8. Treat those as LMI invocation concurrency values, independently of SDK map/parallel branch concurrency.
• Short, configurable invocation timeouts for deadline tests, distinct from the overall durable execution timeout.
• Explicit test-run identifiers and isolated resources for coordination, side-effect observation, and logs.

Verify LMI + Durable Execution availability and supported Java runtime/architecture combinations in the chosen test region before running the matrix. Use only supported combinations; the current ordinary-Lambda Java matrix should not be copied blindly. An unavailable deployment combination must be reported explicitly, without falling back to ordinary Lambda and claiming LMI coverage.

Keep the test SDK artifact tied to the commit under test.

2. Build a driver that proves concurrent execution in the same environment.

Reuse CloudDurableTestRunner and its history parsing where appropriate. Create one runner per execution because it stores mutable lastResult; share the thread-safe AWS Lambda client if useful.

Add test-only diagnostics that record:

• A JVM/execution-environment identifier generated once during initialization.
• Durable execution ARN, invocation request ID, and test-run ID.
• Runtime-handler, root-handler, and step/child task entry/exit markers.
• Executor active/queued task counts and invocation-owned task counts where available.
• Root cleanup entry/exit and the point where the outer wrapper returns the SDK response.
• Same-JVM sequence numbers or monotonic timestamps for ordering assertions.

Keep diagnostic identifiers out of durable operation names, control flow, and checkpoint identity.

Use controlled, bounded barriers to establish overlapping invocations, then require evidence of distinct invocation IDs overlapping within the same environment ID. Setting concurrency to 2, sending two requests, or seeing overlapping CloudWatch timestamps from different environments is insufficient.

Placement is service-controlled. Use a bounded warm-up/admission procedure with a clear placement deadline; if same-environment overlap cannot be established, fail/report an infrastructure-precondition failure rather than passing the concurrency case. Do not assume a small capacity provider guarantees one JVM.

For worker-recovery assertions, keep the other established slots occupied with bounded healthy work, then require a follow-up probe to enter the affected environment. Report environment replacement separately: a probe succeeding in a new environment does not prove that the old invocation released its worker. For higher-concurrency configurations, verify restored concurrent admission in the original environment rather than only one successful follow-up request.

Collect ordering at the actual execution boundaries. In particular, the current onInvocationEnd hook is not proof that the root handler has exited or that the runtime response has been returned. A test wrapper around the SDK entry point plus root/task finally markers can establish that order.

3. Cover the following scenarios against the real service.

Use real Durable Execution checkpoint/history APIs and real waiting/resumption. Local fake backends and time-skipping runners remain useful unit/integration coverage, but do not satisfy these cloud cases.

Distinguish invocation-level timeout/failure from the overall durable execution outcome. A durable execution may be retried/resumed by the service; final success alone must not hide a timed-out invocation whose old tasks are still running. Distinguish client HTTP timeouts from server-reported invocation timeouts as well.

For replay assertions, use durable history and a test-only attempt ledger with idempotent business effects. Require an already-checkpointed successful/failed step body to be skipped on replay. For work interrupted before its checkpoint, allow the retries permitted by the configured step semantics; do not assert exactly-once execution where the SDK guarantees at-least-once behavior.

4. Integrate with CI and preserve useful failure evidence.

Add a separate LMI workflow/job, with explicit opt-in for local cloud tests and PR runs. Run the bounded suite through manual dispatch and automatically on every push to main, including every merged change, using the existing test-account/OIDC model. Do not filter main-branch runs by changed paths.

• Serialize updates to shared stacks/capacity providers, or use distinct resources per run.
• Give provisioning, waiting for prior executions, placement, test execution, and history/log collection separate finite time budgets.
• Bound request counts, capacity, retries, and total test duration; ensure blocking fixtures have a test-only escape path.
• Always publish JUnit reports, deployed configuration/commit, environment/request/execution correlation, durable histories, thread/task snapshots, and relevant CloudWatch logs.
• Use event ordering and bounded polling to handle eventual log/history delivery; avoid tests that pass or fail solely on a fixed sleep.
• Report deployment/placement failures separately from assertion failures. Once a scenario has been established, a violated lifecycle assertion must not be retried until it turns green.
• Reuse one persistent test stack and its LMI functions across runs. Keep the stack and functions after test failures or timeouts; do not add automatic resource teardown or a janitor. Document ownership of the persistent stack, bucket, and capacity provider. Isolate and expire per-run control data, bound log retention, and serialize code updates with tests so unfinished executions are not resumed on a different SDK build.

5. Validate OpenTelemetry isolation and X-Ray propagation on deployed LMI.

The observability scenarios are additional acceptance work. Exercise the real public SDK wrapper, plugin factory registration, both Invocation and Execution OTel views, and a real provider/exporter. Retain the plugin-free lifecycle controls so failures can be attributed to the SDK lifecycle, plugin, runtime carrier, or export/collection path. Passing the core lifecycle suite, merging an implementation PR, or passing ordinary on-demand OTel conformance does not establish this LMI acceptance.

• Invocation isolation under proven overlap. Use the same durable operation names/sequences in independent executions, with distinct execution ARNs and upstream trace identities. Establish overlap with bounded barriers and correlate runtime request IDs, execution IDs, environment IDs, and the runtime sharing boundary described below. Assert that operation/span attribution, parent relationships, invocation duration/status and per-invocation buffers belong to the correct execution, without unexpected duplicate or cross-attributed Invocation/Workflow spans.
• Lifecycle, replay and late activity. Exercise success, handler failure, real suspension/resume, retry and actual invocation timeout while a peer remains active. Hold a completion across another invocation's End/cleanup boundary through the public execution path. Ending or resetting invocation A must not clear, end, relabel or otherwise corrupt B's spans/context. Retain checkpointed successes/failures and expected side-effect counts on replay.
• Shared provider/exporter ownership. Reuse the environment-owned provider/exporter across invocation-local plugin instances. Check that one invocation cannot shut down the shared resources or block a healthy peer indefinitely. Include bounded exporter failure/stall and flush-budget cases, observing the documented SDK contract. Report expected timeout/export losses explicitly; do not introduce an unconditional lossless-delivery or export-acknowledgment guarantee.
• Invocation-local X-Ray propagation. On deployed LMI, record the genuine runtime-provided header at invocation entry and verify the documented trace ID (Root), parent (Parent) and explicit sampling (Sampled=0/1) behavior in both views, including overlapping executions and real resume/retry. Distinct headers must not bleed between invocations or inherit stale process-wide carriers. Check propagation at concurrency 1 as well as under proven overlap. Keep sampling expectations consistent with the selected view and current invocation's runtime header.
• Carrier and compatibility matrix. Retain controlled public-wrapper/installed-artifact coverage for unavailable, absent, empty, malformed and unreadable carriers, ordinary-Lambda fallback, supported custom extractors and older runtime/context interfaces. An available invalid/empty/unreadable carrier must not borrow a stale global header. If a carrier condition cannot be produced by the real LMI runtime, label that case as a controlled compatibility fixture; it is not evidence of a native runtime case. Required genuine-runtime cases remain separate acceptance gates.
• Evidence and regression sensitivity. Record exact core/plugin/provider/layer versions and checksums, tested commits, function/version/configuration readbacks, runtime and environment identities, correlated request/execution/trace/span IDs, exported spans or trace-query results, lifecycle ledgers, JUnit and durable histories. Use bounded collection and distinguish placement/setup, SDK/plugin assertions, sampling, exporter failure and missing evidence. Retain comparable negative controls against a pinned affected implementation or deterministic defect fixture; a missing span or failed placement cannot count as an isolation pass.

Reuse the existing persistent LMI fixtures and evidence collection where practical, adding dedicated observability handlers/assertions as needed. Preserve the existing replay, checkpoint, cleanup and concurrency assertions. Keep this testing issue open until both its original lifecycle requirements and the added observability acceptance below are satisfied; required unverified cases remain unchecked.

Java sharing, ownership and carrier boundary: Prove simultaneous distinct invocations in the same JVM with runtime/request-thread identities and overlapping intervals. Capture Context.getXrayTraceId() on the runtime caller before SDK worker dispatch, then verify the immutable invocation snapshot reaches both OTel views. Check handler-owner Start/body/scope cleanup/drain/End ordering and same-thread MDC/context restoration, including reused worker threads and late continuations. Header propagation must not mutate shared system properties or environment variables. Preserve the compatibility distinction between an inherited default getter returning null (carrier unavailable) and a real runtime override returning null (authoritative absence); nonfatal getter failures and JVM-fatal identity controls use clearly labeled fixtures where genuine runtime generation is unavailable.

Acceptance Criteria

• Infrastructure can reproducibly deploy a real LMI durable function using the SDK commit under test.
• The recorded matrix includes concurrency 1 and at least one value greater than 1, using supported runtime/region combinations.
• Every same-environment concurrency case includes positive evidence of overlapping invocation IDs in one JVM.
• Cloud regressions cover all three core lifecycle/executor problems in [Bug]: LMI invocation cleanup can hang or outlive the response; shared fixed pools can deadlock #726.
• Tests assert the desired fixed behavior: they expose the affected implementation and pass after the relevant fixes, rather than counting reproduced defects as successful regression coverage.
• Real invocation timeout, task exit, cleanup budget, healthy-invocation isolation, and worker-capacity recovery are measured separately.
• At least one real suspension/resume path verifies deterministic replay, stored successes/failures, and appropriate side-effect counts.
• Default executor behavior is covered; custom fixed/virtual-thread variants are included only with explicit runtime support and executor contracts.
• Repeated warm-environment runs detect residual invocation tasks/context and resource growth with bounded tolerances.
• CI exposes deployment/precondition failures, assertion failures, skipped/unsupported combinations, and evidence-collection failures distinctly.
• Every push to main triggers the LMI cloud test suite, regardless of which paths changed.
• Failure artifacts are available for manual and main-branch runs, and the same persistent stack/functions are reused without automatic resource deletion or a janitor.
• Documentation explains prerequisites, deployment/update, local driver commands, budgets, interpreting results, and ownership/retirement of persistent resources.
• Both OTel views pass deployed-LMI attribution/isolation assertions with proven same-JVM overlap and reused operation names/IDs.
• Success, failure, suspension/resume, retry, actual timeout and late continuation/hook paths preserve peer spans/context, owner End ordering, MDC restoration and checkpoint/replay semantics.
• Environment-owned providers/exporters remain usable by peers; bounded flush/failure scenarios validate the supported lifecycle contract without claiming forced termination or lossless export after timeout.
• Genuine runtime-caller X-Ray headers survive worker dispatch and preserve required Root/Parent/Sampled behavior at concurrency 1 and under overlap, including sampled/unsampled and resume/retry cases.
• The carrier/compatibility matrix covers authoritative absence, fallback and supported legacy/custom interfaces, labels native versus controlled cases, and records compatible installed 3.x core/plugin/provider/layer artifacts.
• Passing observability run URLs and retained correlated span/trace evidence are linked here with exact deployed artifact/configuration identities and regression-sensitivity evidence; harness-only/core-only results do not satisfy these LMI gates.

Is this a breaking change?

No. This adds test infrastructure and regression coverage. Any SDK behavior or executor contract changes remain tracked in #726.

Does this require an RFC?

No public API RFC is required for the initial test suite. Document a short test design covering same-environment placement, lifecycle observation, timeout semantics, and infrastructure ownership before implementation.

Additional Context

Core lifecycle tracking: #726. The original OTel defects are #761 and #762; their remaining deployed-LMI validation is now tracked here. Cross-SDK coordination: JS #1017 and umbrella #798.

The existing local investigation established the three defects on v2.2.0. No LMI cloud validation has been performed as part of that report, so this issue defines the work needed to close that gap.

This suite belongs in the Java SDK repository as runtime-specific cloud integration coverage. If implementation exposes new language-neutral requirements, propose those separately in the shared conformance repository.