Skip to content

fix: export visible replay completions - #869

Draft
zhongkechen wants to merge 1 commit into
mainfrom
codex/fix-854-otel-replay-completions
Draft

zhongkechen wants to merge 1 commit into
mainfrom
codex/fix-854-otel-replay-completions

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Issue

Fixes #854.

Description of changes

A callback can complete while an execution is suspended, then be skipped when replay returns its parent's cached result. Both OTel views now retain visible terminal updates from invocation input and checkpoint changes, deliver them through the existing terminal span path, and export unvisited completions before Invocation closes.

Parent registration drains deferred children under the actual replay segment. Completed parent contexts remain available for later updates, and finalization materializes pending ancestors before descendants. Successful observations are deduplicated within an invocation; each newly opened Invocation-view ReplayChildren or virtual-context segment still receives its own natural End.

Design and simplification

  • A private CompletionState owns pending updates, the parent index, in-flight claims, successful-delivery bookkeeping, and cleanup for both views. Completion payloads have one authoritative dictionary; the parent index stores only IDs and preserves observation order. Completed input parents are retained as an ID set.
  • Each plugin keeps its own parent selection, span rendering, and replay-segment policy. State access and parent-registration dequeue use the existing span-registry lock. Tracer and processor callbacks run outside it, and delivery is recorded only after span completion succeeds.
  • Finalization uses standard-library topological ordering. Completed parent references preserve identity but do not count as exported terminal spans. Child processor failures during parent registration are logged without interrupting attachment of the parent's context; undelivered metadata remains pending.
  • Simpler alternatives considered: an invocation-end-only scan cannot attach children while their actual replay parent is active; treating the full snapshot as new work repeats ordinary replay exports; synthesizing core End hooks changes the contract for every plugin. Sharing the delivery state removes duplicated transitions without combining the two views' different span lifecycles.

The root cause is that natural End hooks cannot observe children skipped by replay, active-span registries discard completed parent identities, and operation-level deduplication alone cannot distinguish reopened context segments. This is an internal bug fix using existing snapshots and hooks; it adds no public API or configuration. Core scheduling, status mapping, timestamps, provider flushing, testing-library behavior, and conformance handlers are unchanged. Export recovery and broader finalization failure isolation remain #855.

Validation

Python 3.13.17, Hatch 1.16.5:

  • Re-ran the public-handler missing-completion regression against the base OTel source at e42947e8: all 12 cases fail because the callback span is absent.
  • hatch run dev-otel:test -q --tb=short: 901 passed.
  • hatch run dev-otel:typecheck: 52 source files clean.
  • hatch fmt --check in the OTel package and git diff --check: passed.
  • Built core and OTel distributions with hatch build.
  • hatch run test-wheel-otel:test -q --tb=short: 901 OTel tests and 12 lifecycle cases passed, with installed wheel hashes and source bytes verified.

Coverage includes input/checkpoint/natural deduplication, terminal statuses, concurrent End delivery, unavailable ancestors, actual replay-parent preference, late updates, context-segment re-entry, cached-parent body skipping, unknown updated IDs, warm reuse, overlapping invocations, and child-processor isolation. Added a two-view regression that completing one child preserves its pending sibling, and strengthened the processor-failure test to assert eventual delivery exactly once within the tested invocation.

Existing behavioral expectations are preserved. Internal-state assertions in test_completion_snapshots.py, test_context_segment_reentry.py, test_terminal_input_parent_refs.py, and e2e/test_completion_snapshot_int.py now follow CompletionState; the input-parent helper test expects IDs in a set instead of complete records in a list. The PR is intentionally squashed into one commit; the pre-fix failing run above records the regression evidence.

Assumptions and limitations

  • Operation parents form the durable service's tree; cyclic input is unsupported.
  • Only visible operation metadata can be exported. An updated ID without its operation cannot reconstruct a completion. Service-shaped public-handler fixtures are covered; deployed-service qualification remains unverified.
  • Unknown parents fall back to Invocation in Invocation view or a derived logical context in Execution view. Available completed-input IDs can supply logical references after actual replay parents have had a chance to register. A reference cannot recreate a missing span.
  • Deduplication is per invocation and does not guarantee exactly-once backend delivery across failed invocations or processor errors. The installed-wheel lane retains the existing testing-dependency selection; that boundary remains [CI]: Validate the built testing-library wheel in OTel compatibility lanes #863.

Implementation decisions

  • Use one shared, invocation-owned state class and an ordered ID index, with no additional lock or retry scheduler.
  • Retain non-recording contexts after parent completion.
  • Preserve parent-first finalization and contain ordinary child-export failures during registration.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Retain terminal input and checkpoint updates until their parent registers
or invocation finalization exports them. Preserve actual replay parent
contexts and close each reopened invocation segment independently.

Share completion claims, pending state, and successful-delivery tracking
between both OTel views. Store payloads once, index children by ID, and
retain completed input parents as an ID set.

Validation: 901 OTel tests, type checking, formatting, and installed-wheel
OTel/lifecycle validation. Twelve public-handler regressions fail against
the base implementation because the callback completion is missing.

Fixes #854

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OTel]: Export unvisited replay completions with the correct parent context

1 participant