feat: add semantic UI click and type actions - #49
Merged
Merged
Conversation
- Host: click/type/setValue use UIA control patterns and keyboard input only, with single-target proof, disabled/read-only refusal and a focus re-check before any text is sent; inspectionVersion 3 stops an older helper from reporting an action as a successful inspection. - Gateway: wincode_ui_click and wincode_ui_type (19 published tools), contract validation before any helper launch, audit op records the action name. - Skill and docs: describe the actions, prefer mode="setValue" for background writes, and require user approval before any foreground focus. - Tests: real WPF fixture end-to-end side effects plus boundary and version gates.
The gateway acceptance script asserted 17 published tools and the architecture note carried the same stale count; the UI action tools make 19.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two semantic UI action tools on top of the existing read-only UIA evidence path.
wincode_ui_clickandwincode_ui_type(mode:"type"ormode:"setValue") act on exactly one control that a single bounded search proves unique; zero or multiple matches act on nothing.clickandsetValueneed no foreground;mode:"type"requests keyboard focus on the control and re-reads it, refusing withFOCUS_FAILED(sending no text) when that cannot be confirmed.inspectionVersion3 stops an older helper from reporting a requested action as a successful inspection: the Gateway rejects actions from a v2 helper instead of trustingsuccess:true.actionMethod,actionTargetandinputLength; input text is never echoed back. Auditoprecordsclick/type/setValue.mode:"setValue"for background writes and require user approval before any foreground focus, so focus is never taken from a game, chat or meeting the user is actively using.ACTION_FAILEDis documented as possibly partially completed rather than "nothing happened".Test plan
npm run check— 18 stages including delivery verification: passednpm run test:ui— 39 tests including the new real-WPF end-to-end case: passedtests/ui-action-mcp.test.tscovers boundary rejection before any helper launch, the v2-helper version gate, and real fixture side effects (click, toggle, keyboard typing with clear,setValueincluding clearing via an empty string), with ambiguous, disabled, missing and read-only targets verified to leave application state unchanged.