Skip to content

ci: run telemetry conformance via shared workflow - #825

Merged
ParidelPooya merged 1 commit into
feat/conformance-tests-telemetryfrom
ci/telemetry-conformance-workflow
Oct 24, 2026
Merged

ParidelPooya merged 1 commit into
feat/conformance-tests-telemetryfrom
ci/telemetry-conformance-workflow

Conversation

@hln33

@hln33 hln33 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Stack

Stacked on #822. Base branch: feat/conformance-tests-telemetry. Retarget to main and drop the branch filter after the parent merges.

Changes

  • Adds .github/workflows/telemetry-conformance-tests.yml. It calls the shared telemetry suite from the conformance repository.
  • The workflow runs on pull requests and pushes to main that touch sdk/**, telemetry-exporter/**, conformance-tests-telemetry/**, pom.xml, or the workflow file. It also supports manual dispatch.
  • prepare_command installs the sdk and telemetry-exporter modules, then compiles the handlers as a check before deployment.
  • prepare_command writes JAVA_SDK_VERSION to GITHUB_ENV. The handler pom reads it, and the runner's sam build runs in a later step.
  • setup_command is copied from otel-conformance-tests.yml. It selects Java 21.
  • Replaces the "follow-on" line in conformance-tests-telemetry/README.md with a ## CI section.

Shared workflow pin

3e4fdaf0, the head of aws/aws-durable-execution-conformance-tests#150. It includes the exact-record golden check from #160. Re-pin to the merge commit after #150 lands.

Verification

  • Workflow YAML parses with PyYAML.
  • setup_command and prepare_command pass bash -n.
  • shellcheck reports one SC2155 warning in the setup_command block copied unchanged from otel-conformance-tests.yml.
  • mvn help:evaluate -Dexpression=project.version returns 3.0.0.
  • Cloud run on this PR (run 40053690493, head 0feb2439): every workflow step through deploy and invocation passed. Prepare SDK handlers selected Java 21, installed sdk and telemetry-exporter, and compiled the handler module. sam build and Deploy succeeded. The runner invoked Telemetry1Success, saved the execution history, and fetched the telemetry record. So the CI wiring works end to end.
  • The run ends red on telemetry-1 record-shape assertions. These are exporter deviations from the JS reference format, out of scope for this PR: operationsByName.increment key order differs. The exporter fix is a separate change; this job turns green once it lands.

@hln33
hln33 added this pull request to stack #826 October 24, 2026 02:31
The workflow calls the shared telemetry suite in the conformance
repository at a pinned SHA. It runs on pull requests and pushes to main
that touch the SDK, the exporter, or the handler module. It also runs on
manual dispatch.

The prepare step installs the sdk and telemetry-exporter modules into the
local Maven repository. The handler pom depends on both at the
durable.sdk.version property. So the step must install them before it
compiles the handlers.

The handler pom reads that property from the JAVA_SDK_VERSION environment
variable. The runner's sam build runs in a later step. Only GITHUB_ENV
carries values to later steps. So the prepare step writes
JAVA_SDK_VERSION to GITHUB_ENV.
@hln33
hln33 force-pushed the ci/telemetry-conformance-workflow branch from 0feb243 to a7f0dde Compare October 24, 2026 02:31
@hln33
hln33 marked this pull request as ready for review October 24, 2026 03:01
@hln33
hln33 requested a review from a team October 24, 2026 03:01
@hln33
hln33 deployed to ai-pr-review-runtime October 24, 2026 03:01 — with GitHub Actions Active
@hln33
hln33 deployed to ai-pr-review-runtime October 24, 2026 03:02 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings in the PR changes. Residual test risk: CI/cloud tests were not run, and the pinned external reusable workflow was unavailable for local inspection.

Reviewed commit a7f0ddee60243bc3b3a9e29a027158236d8f425b. Workflow run

@ParidelPooya

Copy link
Copy Markdown
Contributor

The red telemetry-1 job is caused by the suite, not by this PR or the Java exporter.

The suite fails the record because the keys of operationsByName.increment appear in a different order from JS. The keys and values match. A JSON object is unordered, and no telemetry destination reads a field by its position. The JS order is also not stable: when an operation name repeats, the position of subType or maxAttempt depends on which occurrence set it first. So key order should not be part of the cross-SDK check.

aws/aws-durable-execution-conformance-tests#166 removes the key-order assertion. The check still requires exactly the expected keys and compares every value, and it still compares list items in order.

After #166 merges, this job should pass with no Java change. The pin 3e4fdaf0 predates #166, so it needs a new pin, or a conformance_test_ref that includes #166, until aws/aws-durable-execution-conformance-tests#150 lands on top of it.

Two other notes on the workflow:

  • 3e4fdaf0 is no longer the head of Change operation IDs to hashed values #150. The head is c3d9c992. The later commits make the report upload fail when no report exists, which keeps a run that validated nothing from going green.
  • The README link to telemetry-conformance-tests.yml#L1-L118 goes stale after the next edit. A link to the file is enough.

@ParidelPooya
ParidelPooya merged commit c516fc9 into main Oct 24, 2026
28 of 29 checks passed
@ParidelPooya
ParidelPooya deleted the ci/telemetry-conformance-workflow branch October 24, 2026 19:55

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — a7f0ddee Deployed Oct 24, 2026 by hln33 via ai-pr-review / Codex review / Generate Codex review #2686
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants