feat: add Workflow Insight instrumentation plugin - #632
Draft
wangyb-A wants to merge 1 commit into
Draft
Conversation
Port of the JS SDK's workflowInsight() plugin as a new package, aws-durable-execution-sdk-python-insight: listens to the SDK's instrumentation hooks and emits one curated WorkflowInsight record (schemaVersion 1.0, JS-identical camelCase wire format) per execution through configurable exporters (LambdaLogExporter default with the operationsByName summary; S3Exporter with the per-occurrence operations array). Mirrors the JS emit model: on-complete/on-failure/on-change scheduling with coalescing, ARN-hash sampling, content configuration (input/output omission and transforms, include_errors, per-operation result opt-in), two-phase truncation, top-level vs full-tree operation detail, and unnamed-operation dropping. Uses the invocation-hook execution_input/execution_result fields introduced in #616.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Workflow Insight instrumentation plugin as a new package,
packages/aws-durable-execution-sdk-python-insight/— a port of the JS SDK'sworkflowInsight()plugin (aws-durable-execution-sdk-js-insight), treated as thereference implementation throughout. Experimental, matching the JS plugin's status.
It listens to the SDK's instrumentation hooks and emits one curated
WorkflowInsightrecord (
schemaVersion: "1.0") per execution. The wire record keeps the JS camelCasefield names so records read identically across SDKs and land in the same stores/queries.
Behavior (mirrors the JS plugin)
LambdaLogExporterdefault (one JSON line to the function's log group,carrying the name-keyed
operationsByNamesummary) andS3Exporter(the losslessper-occurrence
operationsarray; upsert-by-execution-name;none/date/function-namepartitioning).boto3is an extra ([s3]) since Lambda provides it.on-complete/on-failure/on-changewith export coalescing —a newer record supersedes a pending one; exports never propagate errors into the
execution.
include_errorsgating operation-level error detail only, per-operation result opt-in with optional
transform.
operations oldest-first, input/output last; per-exporter
max_record_size_bytesmeasured against the exact shape each exporter emits.
top-level(default; children withparentIdsuppressed) vsfull-tree; unnamed operations are dropped (JS parity).Depends on #616 (merged)
The plugin reads
InvocationInfo.execution_input/InvocationEndInfo.execution_resultintroduced by #616 — the dependency floor is set to
>=1.8.0accordingly (first releasethat will carry those hooks). Capability note kept in the module docstring: the operations
map is reconstructed by accumulating per-operation hooks into per-execution state (keyed
by execution ARN to isolate warm-container reuse), since Python hooks carry no
end-of-invocation operations snapshot.
Conformance validation (live, us-west-2)
Validated against the cross-SDK
insightconformance suite(aws/aws-durable-execution-conformance-tests#73, 18 requirements): 18/18 on the s3
sink and 18/18 on the cloudwatch sink. Two known cross-SDK divergences are documented
in that suite rather than patched over here: operation ids pass through the SDK's native
blake2b[:64]format (JS usesMD5[:16]; the suite asserts ids as opaque), and theper-operation
error.namesurfaces the customer error class while the record-level errorcarries the SDK wrapper name (the suite asserts non-empty).
The suite's Python example handlers land in the conformance repo as a follow-up to #73
once this package is available.
Testing
hatch run test:all packages/aws-durable-execution-sdk-python-insight/tests/)covering record shaping, operations indexing, truncation phases, sampling, emit modes,
and exporter rendering
hatch fmtclean; package registered in the rootknown-first-party