Skip to content

fix: correct terminal telemetry status and timing - #865

Open
zhongkechen wants to merge 7 commits into
mainfrom
codex/fix-853-otel-terminal-status
Open

zhongkechen wants to merge 7 commits into
mainfrom
codex/fix-853-otel-terminal-status

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available

Fixes #853.

Description of changes

Operations ending FAILED, TIMED_OUT, or STOPPED without diagnostic details now
produce ERROR spans in both OTel views. Supplied operation and attempt exceptions
are recorded at the same normalized timestamp used to end their span, so delayed
completion delivery cannot put the event after the span. A failed attempt without
diagnostics does not fabricate an exception event.

CANCELLED without an error remains UNSET, RETRY remains RETRYING/UNSET, and a caught
operation failure does not make a successful Invocation or Workflow span fail.
Existing exception type/message/stack preservation is retained. The README
documents the refined status mapping and timing behavior.

Design

  • Design note (rule 3): not needed; the existing lifecycle hooks and factory API
    are unchanged. The issue specifies the refined telemetry outcome contract.
  • Simpler approach (rule 2): terminal lifecycle handlers normalize the end time
    once and pass the same integer to exception recording and span completion.
    Invocation view's _end_span only unregisters/ends the span and chooses an
    ordered time when cleanup callers omit it. This removes error plumbing, the
    repeated failed-outcome check, and the duplicate equal-time adjustment.
    The exception recorder requires timestamp: int; each view retains its
    existing normalization policy. Changing status alone would leave exception
    events outside delayed spans; a shared finalizer would mix the two timing
    policies.
  • Root cause (rule 5): status was inferred from diagnostic presence; exception
    recording independently chose the current clock while span completion used
    durable or invocation-normalized time. The exception helper also synthesized
    diagnostics for errorless failed attempts.
  • Net change: +18 source lines across three existing modules; no new public API
    members or configuration options.
  • Validation design: event/span equality alone cannot detect loss of durable
    timing. Keep the implementation and assert explicit expected durable ends
    for operation and attempt spans. Only absent ends use callback wall-clock
    bounds. No production-code change is needed for this review finding.

Validation

  • Original regression-first validation, against unmodified main: 48 failed,
    120 passed
    in the focused status/timing and public-handler suite. Failures
    demonstrate missing ERROR status, fabricated events for errorless attempts,
    and exception events differing from or falling after their span end.
  • After the original fix, the same focused suite 168 passed; those tests
    and their expectations are retained.
  • hatch run dev-otel:test -qq --tb=short: 874 passed.
  • hatch run dev-otel:typecheck: no issues in 45 source files.
  • Package hatch fmt --check: passed.
  • Core and OTel hatch build: passed.
  • hatch run test-wheel-otel:test -qq --tb=short: 874 OTel tests and
    12 lifecycle tests passed
    .
  • Current local validation: Python 3.13.18 and Hatch 1.18.2. CI pins Hatch 1.16.5.
  • Added test_invocation_cleanup_keeps_future_attempt_bounded: invocation
    cleanup must end an incomplete attempt after its start even when the start
    is ahead of the local clock. It passes with the fix and fails when the
    missing-timestamp normalization branch is removed.
  • Built core/OTel source bytes match the checkout; the wheel lane also verifies
    installed source bytes and artifact hashes.
  • Strengthened test_terminal_exception_uses_normalized_span_end and
    test_failed_attempt_exception_uses_normalized_span_end with exact expected
    Execution-view ends, including equal/reversed times and checkpointless
    contexts. Missing-end cases assert callback-before <= end <= callback-after;
    attempt timing is checked with and without diagnostics. Existing event/end
    equality and bounds assertions remain.
  • Reproduced the review gap: the old 24 timing cases passed with both handlers
    forced to use the clock. The new 32 cases pass unchanged production code.
    Independently forcing the clock now fails four operation cases and six
    attempt cases. Removing the clock fallback fails the missing-end cases in
    each handler. All four mutations were rejected and then restored.
  • This revision changes tests only. The previously built core/OTel wheels were
    verified against the unchanged production sources before rerunning the
    complete wheel suite.

The public-handler tests cover caught checkpointed child, virtual child, and
chained-invoke failures, with and without a durable wait/replay. They assert
successful Invocation/Workflow status, bounded events, saved step execution
exactly once, and reuse of stored nonvirtual failure outcomes.

Existing expectation change: test_operation_status_and_exception now expects
ERROR, rather than UNSET, for FAILED/TIMED_OUT/STOPPED without an error. It still
requires no exception event and preserves the original durable status attribute.
The matrix is expanded to include STARTED/PENDING/READY controls. Other added
tests do not replace existing behavioral expectations.

Prior CI results

All four ADOT/X-Ray and Community/S3 jobs
each reported 19 passing cases, case 11 failing, and seven uncovered requirements.
The exact same case-11 history mismatch and counts occur on
main at d7ae955.
The handler/coverage repair remains #864/#796; this PR does not claim case-25
cloud qualification.

The cloud-example workflow
failed in Python 3.11/3.13 because GetDurableExecutionHistory was throttled after
four retries. The affected handlers have no OTel plugin and their polling code is
unchanged. Python 3.12/3.14 passed. No #853-caused regression was found in these
failures.

Prior combined local validation of all six scoped fixes passed 5,058
core/OTel/testing tests. Merge coordination for the shared terminal renderers is recorded in #869.

Assumptions

  • The shared case-25 requirement requires an errorless failed callback to be
    ERROR while the caught workflow succeeds; see the
    shared execution requirement.
  • Explicit OTel exception timestamps are verified against a real TracerProvider,
    SimpleSpanProcessor, and InMemorySpanExporter in both views.
  • Deployed Lambda/backend conformance qualification is unverified here and
    remains in test(otel): cover replay and chained execution #796. No conformance requirement or handler is changed.

Decisions not covered by the issue or the design note

  • Invocation view's private _end_span accepts normalized nanoseconds. Supplied
    timestamps pass through unchanged; cleanup without a timestamp still uses
    _next_ordered_timestamp() to preserve invocation ordering.
  • Retain the existing fallback diagnostic behavior for a present ErrorObject;
    only absence of the error object suppresses the event. Empty callback error
    normalization remains in [Testing]: Preserve empty callback error history without changing runtime errors #862.

Checklist

  • Every new branch has a test that fails without it (rule 4).
  • The first commit adds failing regressions (rule 5).
  • Every edited existing test expectation is listed above.

By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.

@zhongkechen
zhongkechen marked this pull request as ready for review October 27, 2026 23:18
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 27, 2026 23:19 — with GitHub Actions Active
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 27, 2026 23:19 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 27, 2026 23:41 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 28, 2026 21:09 — with GitHub Actions Active
@github-actions

This comment has been minimized.

Normalize terminal timestamps in lifecycle handlers so exception events
and span ends use the same value. Require an explicit exception timestamp
and remove error plumbing and the duplicate equal-time adjustment.

Keep ordered fallback timestamps for invocation cleanup. Cover cleanup
when an incomplete attempt starts ahead of the local clock, and verify
that the new regression rejects removal of the fallback.
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 29, 2026 20:05 — with GitHub Actions Active
@github-actions

This comment has been minimized.

Check exact normalized durable ends for operations and attempts, including
equal, reversed, and missing start times. Bound fallback ends by the
callback clock only when the durable end is absent.

Keep exception/end agreement assertions and verify that forcing wall-clock
time or removing the clock fallback is rejected in each handler.

Addresses review comment r4417357787.
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 29, 2026 20:17 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings after reviewing all seven changed files against rules 1–9. The strengthened timestamp assertions address the prior finding. Tests were inspected but not executed; deployed Lambda/backend behavior remains unverified.

Reviewed commit ef406b5783180645974afbd41cf3e1bf40ffd6b6. Workflow run

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — ef406b57 Deployed Oct 29, 2026 by zhongkechen via ai-pr-review / Codex review / Generate Codex review #2663
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OTel]: Report terminal failures and keep exception events within span bounds

1 participant