Expand cross-SDK scenario and RPC E2E coverage - #2724
Open
stephentoub wants to merge 22 commits into
Open
stephentoub wants to merge 22 commits into
stephentoub wants to merge 22 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add deterministic E2E wire and result-projection coverage for every previously unreferenced public C# RPC method. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add deterministic scenario parity coverage and audit every generated caller-facing RPC method through a local fake runtime. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add deterministic offline Go coverage for missing scenario workflows and every previously unreferenced generated RPC method. Fix client metadata union decoding and fail pending TCP requests promptly on transport loss. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add deterministic fake-runtime coverage for the complete generated RPC contract and representative public SDK workflows from the C# scenario baseline. Restore the missing mode-handler replay fixtures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The Rust atomic-replacement scenario is not atomic, and one .NET ordering test relies on a fixed timing delay.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Expands deterministic cross-SDK scenario and generated RPC coverage while fixing transport, serialization, and extension-context defects.
Changes:
- Adds cross-language workflow scenarios and replay snapshots.
- Audits generated RPC surfaces across all SDKs.
- Fixes Python/Go/.NET transport and decoding behavior plus Rust attachment support.
| File | Description |
|---|---|
test/snapshots/scenario_testing_utility/should_send_wait_observe_idle_events_and_delete_suggestion_session.yaml |
Adds utility workflow replay. |
test/snapshots/scenario_testing_tools/should_preserve_scenario_tool_invocation_identity_arguments_and_text.yaml |
Adds tool identity replay. |
test/snapshots/scenario_testing_tools/should_isolate_scenario_tool_handler_error.yaml |
Adds tool-error replay. |
test/snapshots/scenario_testing_tools/should_deliver_expanded_scenario_tool_result_to_the_model.yaml |
Adds expanded-result replay. |
test/snapshots/scenario_testing_tools/should_cancel_scenario_tool_handler_when_session_disposes.yaml |
Adds cancellation replay. |
test/snapshots/scenario_testing_tools/should_advertise_scenario_tool_schema_override_and_availability.yaml |
Adds tool-schema replay. |
test/snapshots/scenario_testing_skills_and_agents/should_reload_atomically_replaced_skill_and_replay_it_on_resume.yaml |
Adds skill-reload fixture. |
test/snapshots/scenario_testing_skills_and_agents/should_classify_agent_method_not_found_as_remote_protocol_error.yaml |
Adds protocol-error fixture. |
test/snapshots/scenario_testing_sends/should_order_idle_queued_and_immediate_scenario_delivery.yaml |
Adds delivery-order replay. |
test/snapshots/scenario_testing_runtime/should_ping_then_reuse_client_across_two_sessions.yaml |
Adds runtime-reuse replay. |
test/snapshots/scenario_testing_providers/should_apply_reasoning_context_and_auto_atomically_without_implicit_reset.yaml |
Adds provider-state fixture. |
test/snapshots/scenario_testing_persistence/should_truncate_history_and_resend_from_boundary.yaml |
Adds history-truncation replay. |
test/snapshots/scenario_testing_persistence/should_retry_from_existing_history_with_empty_sendmessages.yaml |
Adds empty-batch replay. |
test/snapshots/scenario_testing_persistence/should_page_persisted_events_backward_without_resuming.yaml |
Adds event-pagination replay. |
test/snapshots/scenario_testing_permissions/should_forward_exact_scenario_permission_callback_payload.yaml |
Adds permission replay. |
test/snapshots/scenario_testing_mcp/should_preserve_disabled_scenario_mcp_servers_across_reload_and_resume.yaml |
Adds MCP-state replay. |
test/snapshots/scenario_testing_lifecycle_recovery/should_suspend_disconnect_and_resume_scenario_state_without_delete.yaml |
Adds lifecycle-resume replay. |
test/snapshots/scenario_testing_lifecycle_recovery/should_abort_active_scenario_turn_and_remain_usable.yaml |
Updates abort-recovery replay. |
test/snapshots/scenario_testing_js_extension_bridge/should_surface_structured_canvaserror_from_js_extension.yaml |
Adds canvas-error fixture. |
test/snapshots/scenario_testing_js_extension_bridge/should_bridge_js_extension_canvas_context_log_and_session_continuation.yaml |
Adds extension-bridge replay. |
test/snapshots/scenario_testing_event_subscriptions/should_stop_closed_and_replaced_scenario_event_sources.yaml |
Adds event-source replay. |
test/snapshots/scenario_testing_event_subscriptions/should_deliver_mixed_scenario_event_stream_in_order_after_handler_lag.yaml |
Adds ordered-event replay. |
test/snapshots/scenario_testing_empty_runtime/empty_mode_minimal_toolless_session_has_no_tools.yaml |
Adds empty-runtime replay. |
test/snapshots/scenario_testing_control_state/should_report_processing_while_scenario_tool_is_running.yaml |
Adds processing-state replay. |
test/snapshots/scenario_testing_composition/should_send_scenario_message_with_metadata_and_extension_context.yaml |
Adds extension-context replay. |
test/snapshots/scenario_testing_composition/should_retry_resume_on_replacement_client_after_recoverable_setup_failure.yaml |
Adds replacement-client replay. |
test/snapshots/scenario_testing_composition/should_resume_with_reattached_scenario_host_state.yaml |
Adds host-state replay. |
test/snapshots/scenario_testing_composition/should_read_persisted_scenario_events_without_resuming.yaml |
Adds persisted-event replay. |
test/snapshots/scenario_testing_composition/should_not_emit_redundant_model_change_when_resuming_same_model.yaml |
Adds model-resume replay. |
test/snapshots/scenario_testing_composition/should_classify_queued_and_immediate_scenario_messages_while_busy.yaml |
Adds busy-delivery replay. |
test/snapshots/scenario_testing_cloud/should_notify_steerability_then_send_first_message_without_remote_enable.yaml |
Adds cloud-steering replay. |
test/snapshots/scenario_testing_canvas/should_run_ordered_scenario_canvas_lifecycle_with_exact_context_and_snapshot.yaml |
Adds canvas-lifecycle fixture. |
test/snapshots/scenario_testing_canvas/should_reattach_scenario_canvas_and_route_all_callbacks_after_resume.yaml |
Adds canvas-resume replay. |
test/snapshots/scenario_testing_canvas/should_handle_structured_scenario_canvas_error.yaml |
Adds canvas-error fixture. |
test/snapshots/scenario_testing_callbacks/should_run_scenario_prompt_and_tool_hooks_with_full_context_and_suppression.yaml |
Adds hook replay. |
test/snapshots/scenario_testing_callbacks/should_cancel_scenario_host_callback_when_channel_disconnects.yaml |
Adds callback-cancellation replay. |
test/snapshots/scenario_testing_callbacks/should_auto_switch_scenario_mode_after_rate_limit.yaml |
Adds rate-limit replay. |
test/snapshots/scenario_testing_callbacks/should_approve_scenario_exit_plan_with_full_callback_and_event_state.yaml |
Adds plan-approval replay. |
test/snapshots/mode_handlers/should_invoke_exit_plan_mode_handler_when_model_uses_tool.yaml |
Updates plan-handler response. |
test/snapshots/mode_handlers/should_invoke_auto_mode_switch_handler_when_rate_limited.yaml |
Updates auto-mode response. |
test/harness/test-mcp-app-server.mjs |
Adds local MCP app server. |
test/harness/replayingCapiProxy.ts |
Adds replay-only proxy mode. |
test/harness/replayingCapiProxy.test.ts |
Tests replay-only behavior. |
scripts/codegen/python.ts |
Fixes external union loading. |
rust/tests/e2e/skills.rs |
Adds skill reload/resume coverage. |
rust/tests/e2e/event_fidelity.rs |
Adds delivery-order coverage. |
rust/tests/e2e.rs |
Registers RPC surface tests. |
rust/src/types.rs |
Adds extension-context attachments. |
python/e2e/test_scenario_session_setup_e2e.py |
Tests session setup ordering. |
python/e2e/test_scenario_sends_e2e.py |
Tests send serialization and cancellation. |
python/e2e/test_scenario_lifecycle_recovery_e2e.py |
Tests recoverable lifecycle failures. |
python/e2e/test_scenario_cloud_e2e.py |
Tests cloud workflows. |
python/e2e/test_scenario_canvas_e2e.py |
Tests canvas callback routing. |
python/copilot/session.py |
Exposes extension-context attachments. |
python/copilot/generated/rpc.py |
Uses generated union loaders. |
python/copilot/_jsonrpc.py |
Normalizes process-exit write errors. |
nodejs/test/session-event-codegen.test.ts |
Tests union-loader codegen. |
nodejs/test/e2e/scenario_testing_recovery.e2e.test.ts |
Adds lifecycle recovery scenarios. |
nodejs/test/e2e/scenario_testing_persistence.e2e.test.ts |
Adds persistence scenario. |
nodejs/test/e2e/scenario_testing_composition.e2e.test.ts |
Adds resume composition scenario. |
nodejs/test/e2e/rpc_workspace_checkpoints.e2e.test.ts |
Expands workspace diff modes. |
nodejs/test/e2e/permissions.e2e.test.ts |
Verifies full permission payload. |
nodejs/test/e2e/client_options.e2e.test.ts |
Covers resumed provider options. |
nodejs/test/e2e/canvas.e2e.test.ts |
Covers structured canvas errors. |
nodejs/test/e2e/abort.e2e.test.ts |
Expands abort-state assertions. |
java/sdk/src/test/java/com/github/copilot/ScenarioTestCli.java |
Adds in-process scenario runtime. |
java/sdk/src/test/java/com/github/copilot/ScenarioCoverageE2ETest.java |
Adds Java workflow coverage. |
java/sdk/src/test/java/com/github/copilot/RpcSurfaceTestCli.java |
Adds RPC recording runtime. |
go/rpc/sessions_client_metadata_json.go |
Decodes metadata unions. |
go/rpc/sessions_client_metadata_json_test.go |
Tests metadata decoding. |
go/internal/jsonrpc2/jsonrpc2.go |
Completes requests on disconnect. |
go/internal/jsonrpc2/jsonrpc2_test.go |
Tests disconnect completion. |
go/internal/e2e/scenario_testing_control_state_e2e_test.go |
Tests Go processing state. |
dotnet/test/Unit/JsonRpcTests.cs |
Tests disposal cleanup. |
dotnet/test/Unit/CloneTests.cs |
Covers launch-provider cloning. |
dotnet/test/Harness/ReplayProxy.cs |
Passes replay-only configuration. |
dotnet/test/Harness/E2ETestContext.cs |
Exposes replay-only setup. |
dotnet/test/Harness/E2ETestBase.cs |
Configures replay-only suites. |
dotnet/test/E2E/ScenarioTestingUtilityE2ETests.cs |
Adds utility scenario. |
dotnet/test/E2E/ScenarioTestingToolsE2ETests.cs |
Adds tool scenarios. |
dotnet/test/E2E/ScenarioTestingSkillsAndAgentsE2ETests.cs |
Adds skills and agent scenarios. |
dotnet/test/E2E/ScenarioTestingServerControlE2ETests.cs |
Adds server-control scenarios. |
dotnet/test/E2E/ScenarioTestingPersistenceE2ETests.cs |
Adds persistence scenarios. |
dotnet/test/E2E/ScenarioTestingPermissionsE2ETests.cs |
Adds permission scenarios. |
dotnet/test/E2E/ScenarioTestingLifecycleRecoveryE2ETests.cs |
Adds lifecycle recovery scenarios. |
dotnet/test/E2E/ScenarioTestingEventSubscriptionsE2ETests.cs |
Adds event subscription scenarios. |
dotnet/test/E2E/ScenarioTestingEmptyRuntimeE2ETests.cs |
Adds empty-runtime scenario. |
dotnet/test/E2E/ScenarioTestingE2ETestBase.cs |
Defines replay-only scenario base. |
dotnet/test/E2E/ScenarioTestingControlStateE2ETests.cs |
Adds control-state scenarios. |
dotnet/test/E2E/ScenarioTestingCloudE2ETests.cs |
Adds cloud scenarios. |
dotnet/test/E2E/ModeHandlersE2ETests.cs |
Removes redundant mode-handler tests. |
dotnet/test/E2E/ExternalToolCancellationE2ETests.cs |
Removes redundant cancellation test. |
dotnet/src/Types.cs |
Adds launch-provider option. |
dotnet/src/JsonRpc.cs |
Guarantees disposal cleanup. |
dotnet/src/Client.cs |
Registers providers and process-exit handling. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Handle malformed Java framing, make cancellation intent explicit, and remove timing races from the Go and Node scenario tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Generated by SDK Consistency Review Agent for #2724 · copilot · sonnet50 · 110.2 AIC · ⌖ 11.7 AIC · ⊞ 7.8K
Synchronize the lagged event test on real tool execution and atomically replace the Rust skill file without deleting the active version first. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
Generated by SDK Consistency Review Agent for #2724 · copilot · sonnet50 · 116.7 AIC · ⌖ 11.8 AIC · ⊞ 7.8K
Preserve socket-backed Python transports, wait for asynchronous Rust canvas reattachment, and make assisted permission-mode setup deterministic. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
Preserve startup stderr assertions across process exit-code readiness and snapshot live .NET events through a concurrent queue. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…wn race Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
SDK Consistency ReviewReviewed the authoritative PR delta (
Findings✅ No new SDK-only feature introduced. Each change fixes a defect specific to that language's transport/runtime characteristics (process lifecycle detection, union deserialization) rather than adding capability to only one SDK. This is consistent with the PR's stated goal. 💡 Pre-existing parity gap, not introduced by this PR, but worth a follow-up: the Python fix adds
Since this PR just closed exactly this kind of gap for Python, it may be worth filing a follow-up issue to align the Node.js and Java public attachment surfaces with the generated types (Go/.NET already do this correctly via their generated i️ Also pre-existing (not part of this PR's diff): the Summary: the changes in this PR are internally consistent and appropriately scoped as language-specific bug fixes revealed by the new test coverage. No blocking consistency issues found; two non-blocking follow-up suggestions noted above for future attachment-surface parity work. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "github.com"See Network Configuration for more information.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The SDKs had broad feature tests, but they did not systematically cover representative production workflows or guarantee that every public generated RPC method was referenced by a test. This made it easy for important composition, lifecycle, callback, persistence, and generated-contract regressions to go unnoticed.
What changed
ScenarioTestingbaseline in C# covering realistic startup, session, send, event, recovery, permission, callback, provider, MCP, canvas, persistence, queue, factory, and remote-control workflows.The expanded coverage found and fixes several SDK defects, including C# extension launch-provider registration and transport-failure completion, Python generated union dispatch and transport error normalization, Go metadata-union decoding and pending TCP request completion, and Rust extension-context optionality.
Validation
net8.0andnet472; RPC surface tests passed on both frameworks; final integrated regression passed 98/98 onnet8.0.gofmt,go vet, build, unit tests, focused E2Es, and repeated reliability runs passed. A broad run encountered existing replay misses and timeout behavior outside the changed coverage.mvn verifypassed; native Node validation passed 93/93.Review notes
This is intentionally a large test-focused change because the scenario catalog and generated RPC audits are cross-language consistency guarantees. Most tests share the existing replay infrastructure; fake-runtime tests are reserved for host-owned or contract-only surfaces that cannot be exercised deterministically through the replayed runtime.
Generated by Copilot