feat(plugin): operation maps on invocation hooks - #623
Conversation
| self._operations_provider = operations_provider | ||
| operations = self._snapshot_operation_infos(operations_provider) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| operations_provider=lambda: execution_state.operations, | ||
| updated_operation_ids=invocation_input.updated_operation_ids, | ||
| execution_input=input_event, |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
| The same value the durable handler receives as its ``event`` argument, | ||
| parsed from the execution's stored input payload. ``None`` when the | ||
| execution has no input payload, or when the info was built without one. |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
9cdcfe9 to
3210772
Compare
| """ | ||
| aws_request_id = lambda_context.aws_request_id if lambda_context else None | ||
| self._operations_provider = operations_provider | ||
| operations = self._snapshot_operation_infos(operations_provider) |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| A point-in-time view of the execution's operation map: as observed at the | ||
| start of the invocation on ``on_invocation_start``, and as observed at the | ||
| end of the invocation on ``on_invocation_end``. Empty on the very first | ||
| invocation-start, before any operation has been checkpointed. |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment has been minimized.
This comment has been minimized.
3210772 to
fb8cb0f
Compare
|
|
||
| def __init__(self, provider: Callable[[], dict[str, OperationInfo]] | None) -> None: | ||
| self._provider = provider | ||
| self._lock = Lock() |
There was a problem hiding this comment.
Codex AI review
[P2] Keep invocation info deepcopy-safe. dataclasses.asdict() deep-copies non-dict fields, but this embedded lock is not deepcopyable. Plugins that serialize or copy invocation info will raise, and dispatch will silently drop the hook. Implement __deepcopy__ to materialize a lock-free mapping or expose a plain dictionary, with a regression test using asdict().
| self._operations_provider = operations_provider | ||
| operations = self._snapshot_operation_infos(operations_provider) |
There was a problem hiding this comment.
Codex AI review
[P2] Skip snapshots when no plugins are active. durable_execution always supplies the provider, so default executions still snapshot here and again at invocation end despite having no plugin executor. Since ExecutionState.operations already returns a copy and _snapshot_operation_infos copies it again, this adds four O(n) dictionary copies per invocation. Short-circuit invocation hook setup when self._plugins is empty and test that the provider is not called.
| def test_first_invocation_has_empty_maps(self): | ||
| with self.executor.run(): | ||
| self._start(operations={}, updated_operation_ids=[]) |
There was a problem hiding this comment.
Codex AI review
[P2] Exercise the real first-invocation path. _start hard-codes is_first_invocation=False, and production supplies ExecutionState.operations after pagination, where a normal first invocation already contains the root EXECUTION operation. This synthetic empty map neither tests its name nor validates the new wiring. Add the required end-to-end suspend/replay coverage through durable_execution, asserting the root map, UpdatedOperationIds subset, and refreshed end map; also remove the public empty-map guarantee unless execution entries are intentionally filtered.
Codex AI reviewFound three issues affecting serialization, no-plugin performance, and production-path coverage. Repository code was not executed per review constraints. Reviewed commit |
| end of the invocation on ``on_invocation_end``. Empty on the very first | ||
| invocation-start, before any operation has been checkpointed. |
There was a problem hiding this comment.
Claude AI review
The map is not empty on the first invocation-start. on_invocation_start snapshots execution_state.operations after fetch_paginated_operations has loaded the initial state, and a fresh execution's initial state contains the EXECUTION operation in STARTED status (see execution_test.py:2089-2093). So on the first invocation this map typically holds one entry — the execution operation itself — which is also consistent with how the existing on_operation_change hook builds its operations map from self.operations.
Impact: a plugin author who trusts this docstring and, e.g., counts or iterates operations on the first invocation expecting emptiness will be off by one. Fix by correcting the wording to state that the first invocation-start typically contains just the execution operation (it is empty only in the rare case where the initial state page carried no operations).
| end of the invocation on ``on_invocation_end``. Empty on the very first | |
| invocation-start, before any operation has been checkpointed. | |
| end of the invocation on ``on_invocation_end``. On the first | |
| invocation-start it typically contains just the execution operation | |
| (mirroring ``on_operation_change``); it is empty only when the initial | |
| state page carried no operations. |
Claude AI reviewSolid, well-tested change. The eager-snapshot / lazy-convert design is thread-safe (the One low-severity finding: the Residual test risk: the unit tests exercise Reviewed commit |
Invocation-level plugin infos carried no view of the execution's operation state, so Python plugins could not see what the JS SDK exposes on every invocation hook. Add to the invocation infos: - InvocationInfo.operations, the checkpointed operation map converted to OperationInfo, on both invocation-start and invocation-end - InvocationStartInfo.updated_operations, the subset named by the invocation input's UpdatedOperationIds, i.e. operations completed externally while the execution was suspended Both are kw-only with empty-map defaults, so existing constructor calls and plugins are unaffected. The end hook re-reads the map so it reports end-of-invocation state rather than the start snapshot. The map is snapshotted when the hook fires but converted to OperationInfo only on first access, so an operation-heavy execution does not pay per invocation for a view no plugin reads, while a plugin that stashes the info still sees the state as of its hook. Refs #617
fb8cb0f to
bd64ec2
Compare
Addresses the Codex review comment on #623. The thread lock in the lazy operation map made the enclosing frozen info uncopyable: dataclasses.asdict() deep-copies any field that is not a dict, list, tuple or dataclass, and a lock cannot be copied. Both asdict() and deepcopy() raised TypeError, and since plugin exceptions are swallowed a plugin that serialized an info would have its hook silently dropped. Drop the lock: building the map is pure and idempotent, so a race can only duplicate work, never corrupt the result, and the assignment is atomic. Add __deepcopy__ so the view materializes into an ordinary dict when copied, which also keeps the provider closure out of the copy. Adds regression tests using asdict() and deepcopy() over both hook infos and both maps, including the not-yet-read case. Refs #617
Closes #617.
Part of a three-PR stack — merge in this order:
execution_input/execution_result(basemain)feat/plugin-invocation-input-result)plugin-parity-invocation-info)SDK-only. Touches
plugin.py,execution.py,plugin_test.pyand nothing else.What
The invocation-level plugin info objects carried no view of the execution's
operation state, so Python plugins could not see what the JS SDK exposes on
every invocation hook.
InvocationInfo.operations— the checkpointed operation map converted toOperationInfo, on invocation-start and invocation-end. Mirrors JS,where it sits on
InvocationBaseInfo.InvocationStartInfo.updated_operations— the subset named by the durableinvocation input's
UpdatedOperationIds, i.e. operations completedexternally while the execution was suspended. Start hook only, matching JS's
InvocationInfo.updatedOperations.Both are kw-only with empty-map defaults, so existing plugins, hook
constructors, and positional callers are unaffected.
Notable decisions
start-of-invocation snapshot. This matches JS, which overrides
operations: toOperationInfoMap(...)at everyonInvocationEndsite, and itis strictly more informative — the suspending invocation reports 1 operation
at start and 2 at end.
operation-heavy executions. Fully lazy was wrong: a plugin that stashes the
info and reads it later would observe a later state than its own hook. So
the raw map is snapshotted when the hook fires (a cheap dict copy) and only
the per-operation
OperationInfoconversion is deferred and cached. A unittest pins this.
is_replayedis leftFalseon map entries, matching JS'stoOperationInfoMap(isReplay: false) — these describe stored state, notreplay events.
plugins=parameter, which emits aFutureWarning.Testing
hatch fmt --checkclean.applied: 23/23, exit 0 — 10-19 flips PASSED with no change to the
requirement, and 10-1..10-18 / 10-20..10-23 are unaffected.
Known pre-existing flake (not from this change)
One earlier suite run showed 10-22 (
PluginOperationChangeShape) red; animmediate re-run on byte-identical code was 23/23 clean. The only edit on that
path here is a pure refactor (an inline dict comprehension replaced by
_to_operation_info_map, the same conversion), and 60/60 local in-processrepeats of the 10-22 handler satisfy its primary matcher. JS's handler for that
requirement is structurally identical, so it is likely cross-SDK. Tracked
separately.