Repository navigation
fix: correct terminal telemetry status and timing - #865
Open
zhongkechen wants to merge 7 commits into
Open
zhongkechen wants to merge 7 commits into
zhongkechen wants to merge 7 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This was referenced Oct 28, 2026
Normalize terminal timestamps in lifecycle handlers so exception events and span ends use the same value. Require an explicit exception timestamp and remove error plumbing and the duplicate equal-time adjustment. Keep ordered fallback timestamps for invocation cleanup. Cover cleanup when an incomplete attempt starts ahead of the local clock, and verify that the new regression rejects removal of the fallback.
This comment has been minimized.
This comment has been minimized.
Check exact normalized durable ends for operations and attempts, including equal, reversed, and missing start times. Bound fallback ends by the callback clock only when the durable end is absent. Keep exception/end agreement assertions and verify that forcing wall-clock time or removing the clock fallback is rejected in each handler. Addresses review comment r4417357787.
Contributor
Codex AI reviewNo actionable findings after reviewing all seven changed files against rules 1–9. The strengthened timestamp assertions address the prior finding. Tests were inspected but not executed; deployed Lambda/backend behavior remains unverified. Reviewed commit |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available
Fixes #853.
Description of changes
Operations ending FAILED, TIMED_OUT, or STOPPED without diagnostic details now
produce ERROR spans in both OTel views. Supplied operation and attempt exceptions
are recorded at the same normalized timestamp used to end their span, so delayed
completion delivery cannot put the event after the span. A failed attempt without
diagnostics does not fabricate an exception event.
CANCELLED without an error remains UNSET, RETRY remains RETRYING/UNSET, and a caught
operation failure does not make a successful Invocation or Workflow span fail.
Existing exception type/message/stack preservation is retained. The README
documents the refined status mapping and timing behavior.
Design
are unchanged. The issue specifies the refined telemetry outcome contract.
once and pass the same integer to exception recording and span completion.
Invocation view's
_end_spanonly unregisters/ends the span and chooses anordered time when cleanup callers omit it. This removes error plumbing, the
repeated failed-outcome check, and the duplicate equal-time adjustment.
The exception recorder requires
timestamp: int; each view retains itsexisting normalization policy. Changing status alone would leave exception
events outside delayed spans; a shared finalizer would mix the two timing
policies.
recording independently chose the current clock while span completion used
durable or invocation-normalized time. The exception helper also synthesized
diagnostics for errorless failed attempts.
members or configuration options.
timing. Keep the implementation and assert explicit expected durable ends
for operation and attempt spans. Only absent ends use callback wall-clock
bounds. No production-code change is needed for this review finding.
Validation
120 passed in the focused status/timing and public-handler suite. Failures
demonstrate missing ERROR status, fabricated events for errorless attempts,
and exception events differing from or falling after their span end.
and their expectations are retained.
hatch run dev-otel:test -qq --tb=short: 874 passed.hatch run dev-otel:typecheck: no issues in 45 source files.hatch fmt --check: passed.hatch build: passed.hatch run test-wheel-otel:test -qq --tb=short: 874 OTel tests and12 lifecycle tests passed.
test_invocation_cleanup_keeps_future_attempt_bounded: invocationcleanup must end an incomplete attempt after its start even when the start
is ahead of the local clock. It passes with the fix and fails when the
missing-timestamp normalization branch is removed.
installed source bytes and artifact hashes.
test_terminal_exception_uses_normalized_span_endandtest_failed_attempt_exception_uses_normalized_span_endwith exact expectedExecution-view ends, including equal/reversed times and checkpointless
contexts. Missing-end cases assert callback-before <= end <= callback-after;
attempt timing is checked with and without diagnostics. Existing event/end
equality and bounds assertions remain.
forced to use the clock. The new 32 cases pass unchanged production code.
Independently forcing the clock now fails four operation cases and six
attempt cases. Removing the clock fallback fails the missing-end cases in
each handler. All four mutations were rejected and then restored.
verified against the unchanged production sources before rerunning the
complete wheel suite.
The public-handler tests cover caught checkpointed child, virtual child, and
chained-invoke failures, with and without a durable wait/replay. They assert
successful Invocation/Workflow status, bounded events, saved step execution
exactly once, and reuse of stored nonvirtual failure outcomes.
Existing expectation change:
test_operation_status_and_exceptionnow expectsERROR, rather than UNSET, for FAILED/TIMED_OUT/STOPPED without an error. It still
requires no exception event and preserves the original durable status attribute.
The matrix is expanded to include STARTED/PENDING/READY controls. Other added
tests do not replace existing behavioral expectations.
Prior CI results
All four ADOT/X-Ray and Community/S3 jobs
each reported 19 passing cases, case 11 failing, and seven uncovered requirements.
The exact same case-11 history mismatch and counts occur on
main at d7ae955.
The handler/coverage repair remains #864/#796; this PR does not claim case-25
cloud qualification.
The cloud-example workflow
failed in Python 3.11/3.13 because
GetDurableExecutionHistorywas throttled afterfour retries. The affected handlers have no OTel plugin and their polling code is
unchanged. Python 3.12/3.14 passed. No #853-caused regression was found in these
failures.
Prior combined local validation of all six scoped fixes passed 5,058
core/OTel/testing tests. Merge coordination for the shared terminal renderers is recorded in #869.
Assumptions
ERROR while the caught workflow succeeds; see the
shared execution requirement.
SimpleSpanProcessor, and InMemorySpanExporter in both views.
remains in test(otel): cover replay and chained execution #796. No conformance requirement or handler is changed.
Decisions not covered by the issue or the design note
_end_spanaccepts normalized nanoseconds. Suppliedtimestamps pass through unchanged; cleanup without a timestamp still uses
_next_ordered_timestamp()to preserve invocation ordering.only absence of the error object suppresses the event. Empty callback error
normalization remains in [Testing]: Preserve empty callback error history without changing runtime errors #862.
Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.