[zhongkechen]

Adds the OpenTelemetry/X-Ray LMI acceptance scope from #742 to the existing lifecycle suite. The two persistent Python 3.14 functions now collect 76 cloud cases: the original 28 lifecycle cases plus 48 observability cases across both OTel views and native concurrency 1 and 2.

This PR addresses #742; keep that issue open until the required deployed acceptance evidence is recorded. Runtime deadline and non-cooperative cleanup defects remain tracked in #741 and the separate #783 implementation.

Changes

• Exercise the real public handler and current per-invocation plugin factories with process-owned providers. Record actual exported span identities, attributes, status and timing in the existing private evidence ledger.
• Add runtime X-Ray sampled/unsampled checks, warm-process success/failure/replay, concurrent-worker attribution, exporter failure/recovery and held exports. Repeat the original deadline, retry, PENDING, late-operation and in-flight-return contracts with each OTel view enabled.
• Send the test caller's trace header through the real Invoke HTTP request, then validate the actual runtime environment carrier. Missing/invalid-header compatibility cases remain clearly labeled local fixtures.
• Package both checkout-built core 3.0.0 and OTel 2.0.0 wheels for Linux arm64, retaining wheel hashes and dependency identities. Increase the bounded cloud assertion/job budgets to 60/120 minutes for the expanded matrix.
• Synchronize with current main's plugin factory API. Preserve the existing lifecycle assertions, shared deployment, run ownership, settlement and private-storage controls.

Validation

• Full LMI harness: 134 tests passed, including public-wrapper sampling, real local suspension/replay, provider reuse, export failure, transport-carrier isolation and negative evidence controls.
• Ruff lint/format checks and git diff whitespace checks passed.
• All 76 cloud cases collect successfully.
• Core/OTel wheels and the Linux arm64 function bundle build successfully.

SDK return timing and resource snapshots are captured before post-call evidence I/O. Workflow credentials are refreshed for the longer assertion and cleanup phases.

No fresh AWS deployment or LMI acceptance run is claimed. Cloud execution remains explicit workflow dispatch while #741's required behavior is unresolved. The new observability cases are ordinary assertions; local success does not establish worker reclamation or lossless export after timeout.

Related test updates: JS #928 and Java #823.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[zhongkechen]

Per-test analysis of LMI run 35918433206

Scope: workflow run, attempt 1, PR head f385a1dfb24cdd91a5cbf3e3c00926b7919ce175. Both artifact manifests record the checked-out PR merge commit as ea88d8ee2f99562dd4f6f6dab4075d08270cf7c4. This analysis describes that completed run; it does not establish the results of the subsequent shared-function/persistent-deployment revisions.

Evidence reviewed: job/step results, the two lmi-3.14-c1-35918433206-1 / lmi-3.14-c2-35918433206-1 artifacts (cloud.xml, manifests/configuration, events.json, invocation records, service histories and cleanup reports), the harness JUnit report, and the test/assertion source at the tested revision.

Overall: 28 cloud cases — 16 PASS, 12 assertion FAILURES, 0 ERRORS, 0 SKIPS. All 72 harness tests passed. Both cloud jobs successfully built, deployed/read back their functions, collected final evidence, deleted their temporary resources, and published artifacts. Only the cloud assertion steps failed. The ledgers contain no CONTROL_ERROR or emergency ESCAPE events.

Configuration	Cloud result	Job
Python 3.14, native environment concurrency 1	8 PASS / 6 FAIL / 0 ERROR / 0 SKIP	c1
Python 3.14, native environment concurrency 2	8 PASS / 6 FAIL / 0 ERROR / 0 SKIP	c2

Configuration and interpretation

• c1 / c2 are the maximum concurrent Lambda invocations per execution environment, not SDK branch-thread counts. The concurrency-2 oracle requires live overlap in different Python processes in the same environment, with distinct request IDs and execution ARNs.
• This revision deployed separate normal and deadline functions for each configuration, with invocation timeouts of 60 s / 10 s, a separate 240 s durable execution timeout, and exactly one execution environment per function. Every invocation used $LATEST.PUBLISHED.
• The proposed cleanup/recovery acceptance grace is 5 s. Fault I/O had a 75 s emergency escape in this revision. No escape fired. Later driver releases are cleanup actions and cannot make a failed timing assertion pass.
• The six failing cases in each configuration map to the three problems now recorded in #741: (1) missing invocation-deadline handling, (2) unbounded joining of already-running losing branches, and (3) newly registered operations bypassing orphan rejection after ancestor completion. Problem 3 was newly discovered by this PR's tests and has already been added to [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741.

Passing cases

Each row passed on both c1 and c2 (eight cases per configuration).

Test	Demonstrated behavior and scope
test_checkpoint_replay_callback_and_retry	Real wait/resume and callback completion; completed successful/failed step bodies execute once, stored failure meaning and operation identity remain stable, and the configured retry records attempts 1 and 2.
test_actual_environment_process_concurrency	c1 establishes the single-worker baseline. c2 establishes overlapping live requests in distinct processes within one environment, rather than merely submitting concurrent requests.
test_warm_success_failure_and_suspension_cleanup	Repeated success/failure/replay cycles reuse observed warm processes. The tested wrapper boundaries have no residual invocation-owned SDK threads; thread/FD/RSS observations remain within the fixture's bounded allowances. This is not a claim of zero memory growth or indefinite leak freedom.
test_real_synchronous_checkpoint_settles_before_branch_join	A real service-acknowledged checkpoint response is held before delivery to the SDK waiter. The wrapper does not exit prematurely; releasing the response permits settlement and cleanup. This verifies ordering, not deadline-bounded behavior if the response never settles.
test_pending_waits_for_root_finally_and_then_replays	Root finally exits before normal PENDING; real replay completes without repeating the completed body. c2 also observes healthy-peer progress while cleanup is held. The early-PENDING defect demonstrated by the Java fixture was not reproduced here.
test_nested_single_lane_pools_progress_for_all_runtime_workers	Nested child/map/parallel work progresses under the original single-worker branch-pool settings, with correct per-invocation results and within the 15-second progress budget. Python has no public Java-style shared user-executor injection; this validates the Python counterpart, not an identical shared-fixed-executor topology.
test_root_return_or_failure_settles_inflight_work[return-inflight-SUCCEEDED]	With a losing branch deliberately held, the wrapper waits; after explicit release it settles the work and returns the expected success.
test_root_return_or_failure_settles_inflight_work[failure-inflight-FAILED]	The same settlement ordering holds on failure, retaining the original expected ValueError and message.

The last two passes do not contradict the bounded-cleanup failures below: they verify that work settles after an explicit release, while the failing early-completion tests require bounded cleanup without that release.

Failing cases

Early completion — parallel, map, and nested (three failures per configuration)

Scenario / expected: Select a winner with first_successful / min_successful through parallel, map, and nested branch pools while the loser is already performing controlled blocking I/O. The wrapper should reclaim invocation-owned work within the five-second acceptance bound, without relying on the driver to release the loser.

Actual: Every case failed at #741: result computed but wrapper pinned by losing branch. The held-loser handshake was established before winner readiness. Final ledgers also contain effects after winner +5 s. Wrappers returned only after the post-assertion release:

Scenario	c1: wrapper return after winner	c2: wrapper return after winner	Effects recorded after winner +5 s (c1 / c2)
parallel	7.108 s	7.709 s	7 / 8
map	7.209 s	7.509 s	7 / 8
nested	6.908 s	7.011 s	6 / 6

These measured delays are terminated by test cleanup; they are not measurements of an infinite production hang. The source-level unbounded wait is the shutdown(wait=True) cleanup path described in #741 problem 2. Canceling queued futures does not stop an already-running branch blocked in user I/O. In affected workloads this can retain runtime capacity and allow unwanted downstream work after the winning result is selected.

test_invocation_deadline_isolation_and_capacity_recovery (one failure per configuration)

Scenario / expected: Hold a step across the actual 10-second Lambda invocation timeout, submit recovery demand around the deadline, and recover all configured worker slots in the original environment within deadline +5 s. c2 first establishes a healthy companion in that environment.

Actual: Both cases reached and failed #741: full original-environment worker capacity did not recover. Service history independently records InvocationCompleted with Sandbox.Timedout / Task timed out after 10.00 seconds for each original request. The external ledger still records the original request's effects well beyond that timeout:

Configuration	Fault marker	Last observed effect after invocation deadline	Effects after deadline +5 s
c1	deadline-baa254d57873	+16.043 s	39
c2	deadline-76b8fd56a328	+16.233 s	40

The original wrappers eventually raised after driver cleanup. Later execution completion cannot satisfy the earlier recovery bound. This directly supports #741 problem 1: a service timeout alone does not stop the LMI Python work or restore full worker capacity.

Scope limit: The c2 companion was released and completed before the deadline; this run establishes that early healthy execution, not continuous healthy-peer progress throughout timeout recovery. The full-capacity assertion still requires both slots, so a probe using only the already-released healthy slot cannot make recovery pass. The test aborts at the recovery assertion; subsequent in-test timing assertions were not reached. The service timeout and continued effects above were checked independently in the retained evidence.

test_service_timeout_retry_does_not_repeat_completed_step (one failure per configuration)

Scenario / expected: After a real invocation timeout, stale-attempt effects must cease within the cleanup bound, and a service retry of the same durable execution must reuse the completed step rather than repeat its body.

Actual: Both fail first at #741: work continued after invocation deadline + cleanup grace. Original-attempt effects continue until +7.549 s (c1) and +8.065 s (c2) after the deadline. In c2 (deadline-158ca58c11ed), a retry of the same execution enters another process in the same environment at deadline +1.078 s, while the original process continues producing effects. This supplies direct evidence of stale-attempt/retry overlap, within #741 problem 1.

Scope limit: The formal retry-wait and completed-step assertions occur after the failing stale-effect assertion and were not executed. Final evidence nevertheless records one success body and one service StepSucceeded for that completed step in each configuration, together with the later retry. That supplementary observation does not turn these cases into passes or establish exactly-once execution for interrupted at-least-once work. No completed-step duplication was observed in these records.

test_abandoned_child_rejects_late_durable_operation (one failure per configuration)

Scenario / expected: After race-parallel has completed, release its losing branch and attempt a new child.step(name="late-work") from that branch's finally. The new operation should raise OrphanedChildException before executing its body or creating a service operation.

Actual: Both fail at #741: abandoned child executed a late user side effect. The ledger records LATE_ATTEMPT, a late-work body, and LATE_ACCEPTED; service history independently confirms admission after ancestor completion:

Configuration / marker	race-parallel ContextSucceeded	late-work StepStarted and StepSucceeded
c1 — late-operation-10904bc0b80a	21:02:31.849 UTC	21:02:33.321 UTC
c2 — late-operation-c04640fcc324	21:13:27.221 UTC	21:13:28.819 UTC

All times are on 2026-09-23. This reproduces #741 problem 3 on both configurations, extending the earlier concurrency-2 evidence already recorded in that issue. The orphan guard checks the new operation's own ID, but that ID does not inherit the already-completed ancestor's state. This creates a risk of admitting new downstream actions after the enclosing operation has completed, distinct from merely allowing an existing I/O call to continue.

The demonstrated boundary is after ancestor completion. These late bodies execute before their outer wrapper's final return; the records do not establish new work beginning after wrapper return.

What this run establishes, and what remains pending

• All 12 cloud assertion failures in this run correspond to the three defects now tracked in [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741. No additional untracked SDK defect is established by this run. They remain ordinary failures, not xfail or expected-success results. Passing cases and supplementary observations do not satisfy the portions of failed cases that were never reached.
• The earlier run 35904960112 had a c1 capacity-provider provisioning failure; its c1 tests did not execute. That was an infrastructure failure, not an expected SDK regression. In the run analyzed here, both deployments and both cleanups succeeded.
• The tested revision used temporary normal / deadline deployments with different timeouts. Current head 7bd33b18c5320effbacb86206c8ab12ed9c8ddc3 instead retains two persistent shared functions, runs all 28 cases in one job, uses a common 60-second invocation timeout, isolates data per run, and settles case work before reuse. Those deployment/lifecycle changes are not cloud-validated by the older run above.
• At the time of posting, the current persistent-deployment run is queued. Current-head local validation has 93 passing harness tests, successful packaging/Ruff, and all 28 cloud cases collected. Cloud validation must still establish first creation, subsequent updates of the same functions, and case/run isolation with the new layout.

The SDK fixes remain in #741; this PR supplies the regressions and stays Draft.

[github-actions]

Codex AI review · Finding arf_v1_gz6jzhkkxu53lgixrg4alhvbvt

[P2] Track ambiguous invocations before allowing function reuse. If Lambda accepts Invoke but its response is lost, this raises before registering the attempt in self.invocations. Per-case teardown cannot discover or stop that execution and skips settlement polling when the list is empty. Final run collection happens after subsequent cases have reused the potentially occupied function. Register the marker before dispatch, recover its execution ARN during teardown, and block reuse until settlement is established. Add a lost-response teardown test.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_sajbrsyx7vbqbtm6awgqxcckd5

[P2] Accept independently verified worker retirement as settlement. A worker retired without Python cleanup cannot emit either wrapper-exit event. Even when the deadline test establishes full capacity recovery in the original environment, this predicate remains false. Teardown then fails and configuration['pending'] blocks subsequent cases on that function. Carry request/process-correlated retirement evidence into settlement alongside wrapper exits, while continuing to reject terminal service status alone. Add coverage for retirement without a final wrapper diagnostic.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_c3i7pzkbikvxwal6boqi6pu4lw

[P2] Finish the resumed request before requiring completed telemetry. The timeout-retry lifecycle case returns while the second request still owns the unreleased fault gate. Once its OTEL_ENTER is visible, this assertion requires an OTEL_EXIT that cannot arrive until that request finishes; the gate is released only during subsequent teardown. A correctly running retry therefore produces CollectionError. Preserve the lifecycle observations, then release and finish the resumed request before validating completed telemetry. Add a regression covering an active retry.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_jmxq2wca7rviv63o7gphmdyk5g

[P2] Classify transport failures as fixture errors. Botocore transport exceptions such as ReadTimeoutError bypass this handler. If one terminates the losing branch after WINNER_READY, the gate records IO_INTERRUPTED and IO_EXIT without CONTROL_ERROR. The SDK can then finish joining that branch within the grace period, and early_completion() accepts the infrastructure failure as successful reclamation. Catch botocore transport exceptions here and add a negative control injecting one after winner readiness.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_e5ywr3cj6ap55eketmslbmdrga

[P2] Observe retry progress before checking for duplicate bodies. WRAPPER_ENTER is emitted before any steps execute. This predicate can succeed before the second invocation reaches success, allowing the subsequent body count to remain one even if that retry later repeats the step. Teardown can also stop it before duplication occurs. Wait for the second request's BLOCKED or EFFECT event from controlled-io, which follows the completed step, before checking the count. Add a negative control delaying retry progress after wrapper entry.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_qccfsm2kuodjvzkpbn7upriark

[P2] Keep a healthy peer active across the fault deadline. The healthy invocation completes before the driver waits for the fault's deadline, so this establishes ordinary overlap but cannot detect deadline cleanup interrupting unrelated active work. Recovery probes are not required to start before that boundary either. Start the healthy peer sufficiently late to retain its own timeout budget, hold it across the fault deadline, and require continued progress from the same request/process before releasing it. Add a negative control where deadline handling interrupts that peer.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_rwsnft7pyh3sm5e2maaawo4wpp

[P2] Reuse the decorated handler across warm invocations. Rebuilding durable_execution() for every runtime request prevents the warm success/failure/replay cases from exercising decorator-scoped SDK state. A regression that caches the first plugin on the decorated wrapper would still produce a fresh plugin per request here and pass the isolation assertions. Keep one decorated handler per view for the process lifetime, supply the diagnostic trace through invocation-scoped context, and verify repeated requests through that same wrapper.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_uvvhjqmaxt7odyoiwlyyol5sfc

[P2] Check context restoration in the thread that runs plugin hooks. Both this comparison and the before snapshot execute on the caller thread, while the base SDK runs plugin Start/End inside contextvars.copy_context().run on its invocation worker. Leaked worker bindings cannot change the caller's context, so this reports successful restoration even if the plugin omits its detach calls. Capture context immediately before Start and after End in the owning worker context, and add a negative control that suppresses actual detachment instead of editing the recorded boolean.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_kxttvbw642uvjas3y7xxuivbcl

[P2] Require the expected operation and attempt spans. A trace containing only Invocation and Workflow spans satisfies this validator and terminal(), including replay cases. Duplicating an operation export also passes. Consequently, the acceptance suite can report success when durable operation telemetry disappears or is duplicated. Match operation/attempt spans against the fixture or service history, enforce the expected parents and multiplicities for each view, and add negative controls that remove or duplicate non-root spans.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review · Finding arf_v1_s7pgupquuosr454q7gobb225r7

[P2] Establish reuse of the provider that experienced export failure. With concurrency two, the healthy invocation can run on the other worker; environment replacement can also provide a fresh provider. This test then passes even if export failure leaves the original provider unusable. Retain both calls' evidence and require matching environment, process, PID, and provider identities before accepting recovery. Use bounded placement attempts if needed, and add a negative control showing that recovery on a different worker is rejected.

To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.

[github-actions]

Codex AI review

Found 10 actionable issues in the LMI harness:

• Settlement: lmi_tests/cloud.py:133, lmi_tests/cloud.py:453.
• Fixture validity: lmi_tests/fixture.py:125, lmi_tests/fixture.py:459.
• Deadline/retry evidence: lmi_tests/tests/e2e/test_lifecycle.py:114, lmi_tests/tests/e2e/test_lifecycle.py:158.
• Telemetry validation: lmi_tests/telemetry.py:170, lmi_tests/telemetry_evidence.py:73, lmi_tests/tests/e2e/test_observability.py:92, lmi_tests/tests/e2e/test_observability.py:140.

Static review only; tests and AWS scenarios were not executed.

Reviewed commit 3aeb6a55a0306c505aa8669bf735c34055801a0b. Workflow run