From 65233040a8de76f8b925dd606dfd4537d56d14d6 Mon Sep 17 00:00:00 2001 From: linnnn89 <216342082+linnnn89@users.noreply.github.com> Date: Sat, 26 Sep 2026 23:39:14 +0800 Subject: [PATCH 1/2] feat: add semantic UI click and type actions - Host: click/type/setValue use UIA control patterns and keyboard input only, with single-target proof, disabled/read-only refusal and a focus re-check before any text is sent; inspectionVersion 3 stops an older helper from reporting an action as a successful inspection. - Gateway: wincode_ui_click and wincode_ui_type (19 published tools), contract validation before any helper launch, audit op records the action name. - Skill and docs: describe the actions, prefer mode="setValue" for background writes, and require user approval before any foreground focus. - Tests: real WPF fixture end-to-end side effects plus boundary and version gates. --- CHANGELOG.md | 5 + README.md | 18 +- SECURITY.md | 4 +- ...15\347\275\256\346\214\207\345\215\227.md" | 2 +- ...43\350\256\241\345\210\222\344\271\246.md" | 2 +- ...56\346\265\201\350\257\264\346\230\216.md" | 2 +- package-lock.json | 4 +- package.json | 4 +- skills/wincode/SKILL.md | 8 +- skills/wincode/references/diagnostics.md | 6 +- skills/wincode/references/ui.md | 52 +++- src/Adapters/FlaUiAdapter.ts | 35 ++- src/Core/Config.ts | 2 +- src/Core/ToolRouter.ts | 5 + src/Core/UiContracts.ts | 64 ++++- src/Gateway/UiTools.ts | 88 +++++- .../fixtures/wpf-ui-review/MainWindow.xaml.cs | 63 ++++ tests/request-admission.test.ts | 4 +- tests/tool-contracts.test.ts | 9 +- tests/ui-action-mcp.test.ts | 199 +++++++++++++ .../WinCode.Code.Host.csproj | 2 +- tools/WinCode.Tray/WinCode.Tray.csproj | 2 +- tools/WinCode.UIA.Host/Program.cs | 20 ++ tools/WinCode.UIA.Host/README.md | 27 +- tools/WinCode.UIA.Host/UiActionExecutor.cs | 268 ++++++++++++++++++ tools/WinCode.UIA.Host/UiAudit.cs | 4 +- tools/WinCode.UIA.Host/UiHostContracts.cs | 30 +- .../WinCode.UIA.Host/WinCode.UIA.Host.csproj | 2 +- 28 files changed, 889 insertions(+), 42 deletions(-) create mode 100644 tests/ui-action-mcp.test.ts create mode 100644 tools/WinCode.UIA.Host/UiActionExecutor.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e3c719..7a006b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## 0.16.0 (unreleased) + +- Add semantic Windows UI actions as `wincode_ui_click` and `wincode_ui_type` (keyboard `type` or `setValue`). A control is acted on only when one bounded search proves the exact selector unique inside the target window; disabled controls, unconfirmed keyboard focus, read-only values, unsupported patterns and ambiguous or incomplete searches are refused with explicit codes. Only UI Automation patterns and keyboard input are used — no coordinate mouse simulation, no activation, restore or z-order change of the target window. The helper inspection structure is now version 3, so an older helper can no longer report a requested action as a successful inspection. Results report the pattern used, the target identity and the accepted input length without echoing the input text; actions are audited as `click`/`type`/`setValue` rather than `inspect`. +- Keep UI action requests on the existing admission, mutex, timeout, cancellation and helper-reaping path, and preserve completed action outcomes when the request deadline expires during finalization instead of reporting an action that already happened as unexecuted. + ## 0.15.0 (unreleased) - Report Roslyn snapshot diagnostic totals before sample limits, bounded error-code/project counts and explicit sample omissions; preserve unknown totals for legacy Hosts and conservative semantic coverage. Update Skill guidance to distinguish dependency loading from excluded generator coverage. diff --git a/README.md b/README.md index 2f09f5e..5f37a3e 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ WinCode is a local server implementing the Model Context Protocol (MCP) for AI c - **UI inspection with source navigation:** Inspect a control, find candidate XAML declarations and related C# code, then read the relevant source lines. File paths, line numbers and content hashes make the findings traceable. - **Faster, more accurate inspection in everyday use:** In our day-to-day Windows/.NET development, WinCode makes UI inspection and source navigation faster and more accurate than screenshot-based Computer Use workflows. Direct access to structured control properties and source locations reduces reliance on image interpretation and repeated interaction. Targeted queries and compact responses also reduce the amount of data the model needs to process. - **Project and code analysis:** Explore declared solution and project references, search text and symbols, read selected code, and assess change impact. Built-in text analysis works by default; optional Roslyn integration provides compiler-backed C# symbol and reference analysis. +- **Semantic UI actions:** `wincode_ui_click` and `wincode_ui_type` act on exactly one control that a single bounded search proves unique. They use UI Automation patterns and keyboard input only—never coordinate mouse simulation—and never activate, restore or raise the target window. Disabled controls, ambiguous or incomplete matches, read-only values and unconfirmed keyboard focus are refused rather than guessed. A recorded test with a 222-node window reduced response text from approximately **62 KB to 1.6 KB** by querying a specific control instead of returning the full tree. See the [test record](docs/codex_worklog.md). @@ -140,6 +141,8 @@ Use actual paths and line numbers from the search result. `lineRanges` selects k | `wincode_safe_move_to_trash` | Move validated workspace files to `trash/` and record the actual completed or partial outcome. | | `wincode_ui_list_windows` | List visible top-level windows with process and title filters. | | `wincode_ui_inspect` | Read controls and optional states or screenshots. | +| `wincode_ui_click` | Click one uniquely matched control through UI Automation patterns. | +| `wincode_ui_type` | Type text into, or set the value of, one uniquely matched control. | | `wincode_ui_review` | Inspect UI and return candidate XAML/C# source locations. | | `wincode_hello_world` | Read instance identity, workspace binding, capabilities and known status. | | `wincode_diagnose_project` | Actively check SDKs, Git and the local environment. | @@ -155,16 +158,16 @@ Use actual paths and line numbers from the search result. `lineRanges` selects k ### Scope and limitations -- **Desktop access:** UI inspection is read-only. It does not click controls, type text or read input-field values. Background mode requires both PID and HWND, supports non-minimized windows, and does not activate or restore the target. UI inspection requires an interactive Windows desktop session and does not support headless operation. +- **Desktop access:** UI inspection is read-only: it does not click controls, type text or read input-field values. The separate `wincode_ui_click` and `wincode_ui_type` tools are destructive: they act on one uniquely matched control through UI Automation patterns or keyboard input, never mouse simulation, and never activate or restore the target window. They refuse disabled controls, ambiguous or incomplete matches, read-only values and unconfirmed keyboard focus: `type` requests keyboard focus on the control (which may bring its window to the front), so use it only when the user has approved foreground interaction, and prefer `mode:"setValue"` for background work. Background mode requires both PID and HWND, supports non-minimized windows, and does not activate or restore the target. UI inspection requires an interactive Windows desktop session and does not support headless operation. - **UIA and visual rendering:** Available properties depend on the application's UIA provider. WPF is covered by the project's desktop tests; other frameworks and custom-rendered controls may expose less information. Colors, icons and rendering quality require visual review. Background screenshots use `PrintWindow` without screen-capture fallback; check the returned capture-quality indicators. - **Source mapping:** XAML and C# matches identify candidate source locations, with `runtimeSourceVerified: false`. The tool does not verify that the running build matches the source, resolve dynamic bindings or runtime templates, or determine the active `DataContext`. - **Analysis coverage:** Project structure analysis reads `.sln` and `.csproj` declarations without MSBuild evaluation. Text-based references are heuristic. Review completeness, omissions and diagnostics before drawing conclusions; no matches in a limited scan do not establish absence across the project. - **Resource limits:** Requests, traversal and response size have explicit limits. Overload returns `SERVER_BUSY`; queue time counts toward the timeout. Output limits do not represent process memory limits. Full concurrency, cache and process-lifecycle details are in the [architecture guide](WinCode-架构与数据流说明.md). -- **Inspection notice:** During inspection, a semi-transparent `REC / WinCoding` status overlay is displayed without taking focus, and minimal local audit metadata is recorded under `%LOCALAPPDATA%/WinCode/logs/ui-audit`. See the [diagnostics guide](skills/wincode/references/diagnostics.md) for audit-log maintenance. +- **Inspection notice:** During UI inspection and UI actions, a semi-transparent `REC / WinCoding` status overlay is displayed without taking focus, and minimal local audit metadata is recorded under `%LOCALAPPDATA%/WinCode/logs/ui-audit`, including the action name for actions. See the [diagnostics guide](skills/wincode/references/diagnostics.md) for audit-log maintenance. ### Development and documentation -Current source version: **0.15.0**. See [CHANGELOG](CHANGELOG.md) for version history and migration notes. Windows 11 x64 is the reference platform; ports to other operating systems require adaptation and separate validation. +Current source version: **0.16.0**. See [CHANGELOG](CHANGELOG.md) for version history and migration notes. Windows 11 x64 is the reference platform; ports to other operating systems require adaptation and separate validation. ```powershell npm run check # Builds, core regression, stdio integration and delivery verification @@ -192,6 +195,7 @@ WinCode 是面向 AI 编程智能体的本地模型上下文协议（Model Conte - **后台 UI 检查：**无需激活目标窗口或切换键盘焦点，即可读取运行中应用的控件信息。智能体检查目标窗口时，用户可以继续使用其他应用。 - **支持纯文本大语言模型：**以结构化 JSON 返回控件名称、层级、属性和状态。通过支持 MCP 的智能体客户端，DeepSeek 等以纯文本方式使用的模型也能检查桌面界面，无需输入图像；截图为可选功能。 - **结合源码分析 UI：**检查运行时控件，查找可能对应的 XAML 声明和相关 C# 代码，再读取具体源码。结果包含文件路径、行号和内容哈希，便于核查。 +- **语义化 UI 操作：**`wincode_ui_click` 与 `wincode_ui_type` 只操作同一次有界搜索证明唯一的控件；仅使用 UI Automation 模式与键盘输入，不做坐标鼠标模拟，也不激活、还原或置顶目标窗口。禁用控件、歧义或不完整的匹配、只读值以及无法确认的键盘焦点都会被拒绝，而不是靠猜测继续。 - **实际使用中更快、更准确：**在日常 Windows/.NET 开发中，使用 WinCode 检查 UI 和定位源码，比基于截图的 Computer Use 工作流更快、更准确。通过直接获取结构化的控件属性和源码位置，可以减少对图像识别的依赖和反复交互；配合定向查询与精简响应，还能减少模型需要处理的数据量。 - **项目与代码分析：**查看解决方案和项目中声明的引用关系，搜索文本与符号，按需读取代码，并评估变更影响。默认提供内置文本分析，可选的 Roslyn 集成支持基于编译器语义的 C# 符号与引用分析。 @@ -308,6 +312,8 @@ npm run delivery:verify | `wincode_safe_move_to_trash` | 将通过路径校验的工作区文件移至 `trash/`，记录实际完成或部分完成的结果。 | | `wincode_ui_list_windows` | 列出可见顶层窗口，支持按进程和标题筛选。 | | `wincode_ui_inspect` | 读取控件信息，以及可选的状态或截图。 | +| `wincode_ui_click` | 通过 UI Automation 模式点击唯一命中的控件。 | +| `wincode_ui_type` | 向唯一命中的控件输入文本，或直接写入其值。 | | `wincode_ui_review` | 检查 UI 并返回 XAML/C# 源码候选位置。 | | `wincode_hello_world` | 读取实例身份、工作区绑定、能力及已知状态。 | | `wincode_diagnose_project` | 主动检查 SDK、Git 和本地环境。 | @@ -323,16 +329,16 @@ npm run delivery:verify ### 适用范围与限制 -- **桌面访问：**UI 检查为只读操作，不点击控件、不输入文本，也不读取输入框的值。后台模式需同时指定 PID 和 HWND，仅支持未最小化的窗口，检查过程中不激活或还原目标窗口。该功能需要交互式 Windows 桌面会话，不支持在无头环境（Headless）中运行。 +- **桌面访问：**UI 检查为只读操作，不点击控件、不输入文本，也不读取输入框的值。单独的 `wincode_ui_click` 与 `wincode_ui_type` 是破坏性操作：只通过 UI Automation 模式或键盘输入操作唯一命中的控件，不使用鼠标坐标模拟，也不激活或还原目标窗口。禁用控件、歧义或不完整的匹配、只读值以及无法确认的键盘焦点都会被拒绝；`type` 会向目标控件索取键盘焦点（可能把该窗口带到前台），因此只在用户授权前台交互时使用，后台写入优先 `mode:"setValue"`。后台模式需同时指定 PID 和 HWND，仅支持未最小化的窗口，检查过程中不激活或还原目标窗口。该功能需要交互式 Windows 桌面会话，不支持在无头环境（Headless）中运行。 - **UIA 与视觉效果：**可读取的属性取决于目标应用的 UIA 提供程序。项目的桌面测试覆盖 WPF，其他框架和自绘控件可能提供较少的信息。颜色、图标和渲染质量需要结合图像检查。后台截图使用 `PrintWindow`，不回退到屏幕截图；应检查返回的截图质量提示。 - **源码映射：**XAML 和 C# 的匹配结果提供了可能相关的源码位置，`runtimeSourceVerified` 为 `false`。工具不验证运行版本与源码是否一致，不解析动态绑定或运行时模板，也不确定当前的 `DataContext`。 - **分析范围：**项目结构分析仅静态读取 `.sln` 和 `.csproj` 中的声明，不进行 MSBuild 项目评估。文本引用搜索采用启发式方法。应结合完整性、省略项和诊断信息判断结果；在有限范围内未找到匹配，并不代表整个项目中不存在匹配内容。 - **资源限制：**请求数量、遍历范围和响应大小均有限制。超过处理容量时返回 `SERVER_BUSY`，排队时间计入超时。输出限制不等于进程内存上限。并发、缓存与进程生命周期的详细说明见 [架构文档](WinCode-架构与数据流说明.md)。 -- **检查提示：**检查期间会显示半透明的 `REC / WinCoding` 状态浮层（Overlay），不会获取键盘焦点。同时，将最小必要的审计元数据记录到本地目录 `%LOCALAPPDATA%/WinCode/logs/ui-audit`。审计日志维护方式见 [诊断手册](skills/wincode/references/diagnostics.md)。 +- **检查提示：**UI 检查与 UI 操作期间都会显示半透明的 `REC / WinCoding` 状态浮层（Overlay），不会获取键盘焦点。同时，将最小必要的审计元数据记录到本地目录 `%LOCALAPPDATA%/WinCode/logs/ui-audit`，操作类请求会记录动作名。审计日志维护方式见 [诊断手册](skills/wincode/references/diagnostics.md)。 ### 开发与文档 -当前源码版本为 **0.15.0**。版本历史和迁移说明见 [CHANGELOG](CHANGELOG.md)。项目以 Windows 11 x64 为基准平台，移植至其他操作系统需要适配并单独验证。 +当前源码版本为 **0.16.0**。版本历史和迁移说明见 [CHANGELOG](CHANGELOG.md)。项目以 Windows 11 x64 为基准平台，移植至其他操作系统需要适配并单独验证。 ```powershell npm run check # 构建、核心回归、stdio 集成和交付校验 diff --git a/SECURITY.md b/SECURITY.md index 2b084fb..0699c48 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,8 +1,8 @@ # Security policy / 安全策略 -The latest 0.13.x version and current `main` are maintained; `main` contains the unreleased 0.15.0 source with fixed workspaces. Older versions do not have a separate backport commitment. Supported runtimes are Node 24 (primary) and Node 22 (compatibility), on Windows x64; build requirements are in [CONTRIBUTING](CONTRIBUTING.md). +The latest 0.13.x version and current `main` are maintained; `main` contains the unreleased 0.16.0 source with fixed workspaces and semantic UI actions. Older versions do not have a separate backport commitment. Supported runtimes are Node 24 (primary) and Node 22 (compatibility), on Windows x64; build requirements are in [CONTRIBUTING](CONTRIBUTING.md). -目前维护最新 0.13.x 版本与 `main`；`main` 中的源码版本为 0.15.0，已包含固定工作区，尚未发布 GitHub Release。不承诺对旧版本单独回补。Windows x64 上以 Node 24 为主要环境、22 为兼容环境；构建要求见贡献指南。 +目前维护最新 0.13.x 版本与 `main`；`main` 中的源码版本为 0.16.0，已包含固定工作区与语义化 UI 操作，尚未发布 GitHub Release。不承诺对旧版本单独回补。Windows x64 上以 Node 24 为主要环境、22 为兼容环境；构建要求见贡献指南。 Report suspected vulnerabilities through [GitHub private vulnerability reporting](https://github.com/linnnn89/WinCode/security/advisories/new). Include the affected version/build identity, reproduction steps, expected and observed behavior, and a minimal sanitized example. Do not include credentials, personal databases or private source unnecessarily. Avoid publishing exploit details in a public issue before coordination with the maintainer. diff --git "a/WinCode-Skill\345\210\266\344\275\234\344\270\216MCP\351\205\215\347\275\256\346\214\207\345\215\227.md" "b/WinCode-Skill\345\210\266\344\275\234\344\270\216MCP\351\205\215\347\275\256\346\214\207\345\215\227.md" index 00d20ee..2b04da3 100644 --- "a/WinCode-Skill\345\210\266\344\275\234\344\270\216MCP\351\205\215\347\275\256\346\214\207\345\215\227.md" +++ "b/WinCode-Skill\345\210\266\344\275\234\344\270\216MCP\351\205\215\347\275\256\346\214\207\345\215\227.md" @@ -1,6 +1,6 @@ # WinCode Skill 安装、维护与 MCP 配置指南 -适用于 **0.15.0**，核对日期 2026-09-12（北京时间）。以下使用本机 `I:/WinCode` 路径举例；其他机器必须替换路径。客户端界面名称随版本变化，以实际界面为准。 +适用于 **0.16.0**，核对日期 2026-09-12（北京时间）。以下使用本机 `I:/WinCode` 路径举例；其他机器必须替换路径。客户端界面名称随版本变化，以实际界面为准。 ## 1. 三个独立对象 diff --git "a/WinCode-\344\270\213\344\270\200\350\275\256\345\267\245\347\250\213\345\214\226\350\277\255\344\273\243\350\256\241\345\210\222\344\271\246.md" "b/WinCode-\344\270\213\344\270\200\350\275\256\345\267\245\347\250\213\345\214\226\350\277\255\344\273\243\350\256\241\345\210\222\344\271\246.md" index 24794ed..faf08d8 100644 --- "a/WinCode-\344\270\213\344\270\200\350\275\256\345\267\245\347\250\213\345\214\226\350\277\255\344\273\243\350\256\241\345\210\222\344\271\246.md" +++ "b/WinCode-\344\270\213\344\270\200\350\275\256\345\267\245\347\250\213\345\214\226\350\277\255\344\273\243\350\256\241\345\210\222\344\271\246.md" @@ -1,6 +1,6 @@ # WinCode 后续测试计划 -更新：2026-09-12（北京时间）。当前版本为 **0.15.0**；按需会话改动基于 `af9c1b6`。版本和 CI 状态见 [README](README.md)，实现方式见[架构说明](WinCode-架构与数据流说明.md)，历史结果见[工作日志](docs/codex_worklog.md)。 +更新：2026-09-12（北京时间）。当前版本为 **0.16.0**；按需会话改动基于 `af9c1b6`，语义 UI 操作改动基于 `3d283ed`。版本和 CI 状态见 [README](README.md)，实现方式见[架构说明](WinCode-架构与数据流说明.md)，历史结果见[工作日志](docs/codex_worklog.md)。 固定工作区、保持已加载的 Host、请求数量限制、独立的设计时输出目录、共享缓存回归，以及 PR #40/#41 的代码导航、UI 精简输出和测试修正已合并，Node 22/24 和 CodeQL 检查通过。后续本地验收已完成 TavernDesk 的 8 个上下文场景、6 个 UI 产品任务、两个独立 Gateway 的审计争用及恢复，以及实际 Codex 连接中的单项目 Roslyn 流程。对应 PR 的交付状态见工作日志；这些测试完成项从待办移除，不扩大为所有项目或所有 UI 环境均已验证。 diff --git "a/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" "b/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" index 3d19de2..3c2e85f 100644 --- "a/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" +++ "b/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" @@ -1,6 +1,6 @@ # WinCode 架构与数据流 -**适用版本：0.15.0；更新：2026-09-11（北京时间）。已验证的 main 基线 `631f8ba` 包含 PR #40/#41，Node 22/24 和 CodeQL 检查通过，尚未发布 GitHub Release。本机已完成双连接 UI 及实际 Codex 连接中的单项目 Roslyn 验收，范围和结果见 [README](README.md)。** +**适用版本：0.16.0；更新：2026-09-11（北京时间）。已验证的 main 基线 `631f8ba` 包含 PR #40/#41，Node 22/24 和 CodeQL 检查通过，尚未发布 GitHub Release。本机已完成双连接 UI 及实际 Codex 连接中的单项目 Roslyn 验收，范围和结果见 [README](README.md)。** 本说明描述当前源码中已实现的结构。GitHub 分支保护的历史只读核查日期为 2026-09-08；本轮核对 PR 检查状态，不把它等同重新审计全部保护设置。历史实测结果见[工作记录](docs/codex_worklog.md)。源码版本、磁盘构建和客户端当前连接是三个不同对象，不能互相替代。 diff --git a/package-lock.json b/package-lock.json index 09441c2..0920279 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "wincode-mcp", - "version": "0.15.0", + "version": "0.16.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "wincode-mcp", - "version": "0.15.0", + "version": "0.16.0", "license": "MIT", "dependencies": { "@modelcontextprotocol/client": "2.0.0", diff --git a/package.json b/package.json index b261b61..7923bab 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "wincode-mcp", - "version": "0.15.0", + "version": "0.16.0", "description": "Windows-first MCP gateway: .NET project graph, evidence-bounded context, honest change-impact, long-running process hygiene", "main": "dist/index.js", "type": "module", @@ -25,7 +25,7 @@ "test:tavern-context": "tsx scripts/verify-tavern-context.ts", "test:e2e": "tsx scripts/test-mcp-client.ts", "typecheck": "tsc -p tsconfig.test.json", - "test:ui": "tsx --test --test-concurrency=1 tests/flaui-adapter.test.ts tests/ui-inspect-mcp.test.ts tests/ui-window-list.test.ts", + "test:ui": "tsx --test --test-concurrency=1 tests/flaui-adapter.test.ts tests/ui-inspect-mcp.test.ts tests/ui-window-list.test.ts tests/ui-action-mcp.test.ts", "test:all": "npm run check && npm run check:desktop", "test:ui-query": "tsx scripts/verify-ui-query.ts", "test:ui-code": "tsx --test tests/ui-code-runtime.test.ts", diff --git a/skills/wincode/SKILL.md b/skills/wincode/SKILL.md index c046b3e..a7cd80d 100644 --- a/skills/wincode/SKILL.md +++ b/skills/wincode/SKILL.md @@ -1,16 +1,16 @@ --- name: wincode -description: 使用 WinCode MCP 读取 Windows/.NET 项目源码、引用和变更影响，按需查看桌面 UI。 +description: 使用 WinCode MCP 分析 Windows/.NET 项目源码、引用与变更影响，并通过 Windows UI Automation 检查控件、执行语义点击与表单填写、验证桌面交互流程。适用于源码定位、界面问题排查及支持 UIA 的 Windows 应用测试。 --- # WinCode -适用于 WinCode 0.15.0。已有正确工作区的 MCP 连接时直接使用；采用按需模式时，首次需要 WinCode 才按[诊断手册的会话入口](references/diagnostics.md#skill-按需会话)启动。Skill 被发现或读取不需要预启动任何进程。 +适用于 WinCode 0.16.0。已有正确工作区的 MCP 连接时直接使用；采用按需模式时，首次需要 WinCode 才按[诊断手册的会话入口](references/diagnostics.md#skill-按需会话)启动。Skill 被发现或读取不需要预启动任何进程。 只读取与当前任务有关的手册： - [代码与工作区](references/code.md)：源码搜索、上下文、引用、影响分析和 Roslyn 配置。 -- [窗口与 UI](references/ui.md)：窗口选择、截图、控件读取和源码候选。 +- [窗口与 UI](references/ui.md)：窗口与控件定位、后台语义操作、前台键盘输入（需授权）、结果验证和源码候选。 - [诊断与恢复](references/diagnostics.md)：按需会话的启动、复用、结果读取和关闭，以及版本和故障恢复。按需模式先只读该手册首节。 连接固定到启动工作区；已知根一致时直接查询，不例行重复打开。`WORKSPACE_MISMATCH` 时选择目标项目的连接，可参考 `connectionGuide`；`workspace_open` 只能确认或恢复原工作区。 @@ -22,3 +22,5 @@ description: 使用 WinCode MCP 读取 Windows/.NET 项目源码、引用和变 日常导航用 `wincode_search_text` 限定目录查字面量、`wincode_file_outline` 查看行数和声明，再把返回的 `nextRequest` 交给 `wincode_prepare_context`。先看 `summary` 的范围和缺口，再核对正文与覆盖率。UI 首轮可显式用 `responseFormat:"compact"`，需要几何或更多信息时按 `expansionRequests` 展开；仅使用本连接已声明的能力。 优先按文件、符号和行范围获取小结果。参数遵循手册与实际 Schema；保留截断、降级和歧义，UI 源码候选不等于已验证的运行时映射。`SERVER_BUSY` 或超时后先按诊断手册处理，不自动重放请求或重启连接。 + +用户要求执行或测试界面流程时，可自主完成定位、点击与输入；只要求评估或查看时保持只读。后台路径（inspect、click、`mode:"setValue"`）不需要前台；`mode:"type"` 会索取键盘焦点，只在用户授权前台交互后使用，用户正在游戏、聊天或会议中时先询问并由用户自己切换窗口。细节见 [窗口与 UI](references/ui.md)。 diff --git a/skills/wincode/references/diagnostics.md b/skills/wincode/references/diagnostics.md index e40dc85..de13a86 100644 --- a/skills/wincode/references/diagnostics.md +++ b/skills/wincode/references/diagnostics.md @@ -48,9 +48,9 @@ node "/dist/Client/SkillSessionCli.js" --workspace "<目标 跨项目入口：新构建的 `WORKSPACE_MISMATCH` 响应包含 `connectionGuide`，其中 `configuration.command/args` 是独立 STDIO 连接配置，`verification` 给出连接后检查工作区的调用。也可运行 `node /dist/index.js --print-connection --workspace <目标绝对路径>` 输出同一配置；不创建缓存、不注册或重启客户端、不启动项目 Host。配置默认 local-text，不复制已有 Roslyn、开发或托盘选项；目录存在性在实际连接启动时校验。选择已有正确连接优先，建立新连接仍遵守用户授权。刷新连接后再使用新增导航工具或 UI 精简参数，磁盘重建和 Skill 同步不会热替换旧 MCP Schema。 -0.15.0 的 WORKSPACE_MISMATCH 是固定工作区拒绝：检查 activeWorkspace/requestedWorkspace，选择对应项目连接。错误发生在工作区资源变更之前，不表示旧根已切换或需要清空缓存。hello.health.workspaceBinding 给出固定根及启动来源；argument 是显式 CLI 参数，cwd 是启动目录回退，configuration 是嵌入式配置。显式 --workspace 必须有绝对目录值；已有连接不会因磁盘重建或配置保存自行更新。 +0.16.0 的 WORKSPACE_MISMATCH 是固定工作区拒绝：检查 activeWorkspace/requestedWorkspace，选择对应项目连接。错误发生在工作区资源变更之前，不表示旧根已切换或需要清空缓存。hello.health.workspaceBinding 给出固定根及启动来源；argument 是显式 CLI 参数，cwd 是启动目录回退，configuration 是嵌入式配置。显式 --workspace 必须有绝对目录值；已有连接不会因磁盘重建或配置保存自行更新。 -0.15.0 的 health.admission 返回 business/status 的 active、executing、waiting、accepted、completed、rejected、cancelled、timedOut、peakActive，以及累计 waitMs/executionMs 和 maxWaitMs。每实例最多 32 个未完成业务请求、4 个共享轻量状态请求；内层互斥保持 FIFO，运行中取消须在实际清理后归还容量。workspace_open 占用业务容量，但不计入它自己等待排空的 inFlight。状态不等待慢查询或同根恢复；tools/list 满额以协议错误 data.errorCode=SERVER_BUSY 表达。 +0.16.0 的 health.admission 返回 business/status 的 active、executing、waiting、accepted、completed、rejected、cancelled、timedOut、peakActive，以及累计 waitMs/executionMs 和 maxWaitMs。每实例最多 32 个未完成业务请求、4 个共享轻量状态请求；内层互斥保持 FIFO，运行中取消须在实际清理后归还容量。workspace_open 占用业务容量，但不计入它自己等待排空的 inFlight。状态不等待慢查询或同根恢复；tools/list 满额以协议错误 data.errorCode=SERVER_BUSY 表达。 计时口径：waitMs/maxWaitMs 按已结束请求累计其显式队列等待；executionMs 是队列以外的墙钟耗时，包含 I/O 和取消清理，不是 CPU 用时。active/executing/waiting 为当前请求数；取消/超时计数是 completed 的子集。 @@ -135,7 +135,7 @@ Gateway 通过子进程私有环境传递所属 PID；两个 .NET Host 在项目 ## 手动 Roslyn 释放与可选托盘（0.14.0） -自动释放关闭，本版不创建 idle timer。用户可按 README 手动启动独立 Tray，并给希望管理的 Gateway 启动参数添加 --tray 后刷新连接。托盘只管理已注册的实例，不扫描/终止外部客户端或目标应用；当前新增两个只读导航工具后共 17 个公开工具名称，仍没有让 Agent 自动代替用户释放的管理工具。默认不启用托盘连接、不设置自启动。 +自动释放关闭，本版不创建 idle timer。用户可按 README 手动启动独立 Tray，并给希望管理的 Gateway 启动参数添加 --tray 后刷新连接。托盘只管理已注册的实例，不扫描/终止外部客户端或目标应用；公开工具名称以当前连接实际暴露的 Schema 为准，不在文档里写死数量；没有让 Agent 自动代替用户释放的管理工具。默认不启用托盘连接、不设置自启动。 手动释放遇到业务在途、语义排队/收尾、工作区确认或恢复门时拒绝，不自动延后执行。释放完成后新请求继续；旧 symbolLocation 返回 SNAPSHOT_STALE，显式重新搜索取得当前定位。保留 Gateway、watcher、缓存与最后诊断。清理失败进入 restart_gateway 恢复门，不能靠反复点击清除错误。local-text 没有可释放的 Roslyn。 diff --git a/skills/wincode/references/ui.md b/skills/wincode/references/ui.md index 7f627db..b86c4d4 100644 --- a/skills/wincode/references/ui.md +++ b/skills/wincode/references/ui.md @@ -2,7 +2,7 @@ UI 工具同样占用每实例 32 个业务受理槽；既有 UI/健康探测互斥保留，排队消耗请求预算。SERVER_BUSY 不表示已启动 Helper，不自动重试。hwnd 最长 32 字符；原始参数合计受 64 KiB UTF-8 JSON 预算限制。 -0.15.0 中，每个 Gateway 的源码范围固定于启动根；换项目选择对应连接。目标 PID/HWND 不是工作区身份，UI 源码候选仍按所选连接解释。可选托盘与 UIA 取证 Host 独立，退出托盘不终止 MCP 或卸载正在使用的 Roslyn；手动释放仅影响选定实例的 Code Host。源码缓存修复不证明 UI 候选对应同一运行时状态，多实例窗口隔离仍需单独验收。 +0.16.0 中，每个 Gateway 的源码范围固定于启动根；换项目选择对应连接。目标 PID/HWND 不是工作区身份，UI 源码候选仍按所选连接解释。可选托盘与 UIA 取证 Host 独立，退出托盘不终止 MCP 或卸载正在使用的 Roslyn；手动释放仅影响选定实例的 Code Host。源码缓存修复不证明 UI 候选对应同一运行时状态，多实例窗口隔离仍需单独验收。 ## 规范字段 @@ -19,6 +19,8 @@ UI 工具同样占用每实例 32 个业务受理槽；既有 UI/健康探测互 | `backgroundOnly` | 可选布尔值，默认 `false`；为 `true` 时必须同时提供 `pid` 和 `hwnd` | | `readStates` | 可选布尔值，默认 `false`；只读状态，不执行动作或读取输入值 | | `query` | 可选对象；至少有一个规范定位字段 `automationId/name/controlType`，每个为非空白字符串、最长 256；可选 `maxSearchNodes` 整数 1–5000、默认 1000，`maxMatches` 整数 1–20、默认 10。仅有未知字段不构成有效查询 | +| `wincode_ui_click` | `pid`/`hwnd` 至少一个；`targetAutomationId`/`targetName`/`targetControlType` 至少一个，每个为非空白字符串、最长 256 | +| `wincode_ui_type` | 具备上述 click 的全部字段，另必填 `inputText`（最长 4096；mode=type 必须非空，mode=setValue 允许空字符串以清空值）；可选 `clearBefore`（布尔，默认 false，仅 mode=type 有效）、`mode`（`type`/`setValue`，默认 `type`） | | `wincode_ui_review` | 接受上述 inspect 的全部规范字段，另必填 `candidateFiles`：1–16 个相对 `.xaml` 路径、每项最长 512；可选 `candidateCodeFiles`：1–8 个相对 `.cs` 路径、每项最长 512；可选 `textQueries`：最多 5 个非空字面字符串、每项最长 80 | 候选路径必须在工作区内，不得包含通配符或父目录逃逸；参数合法不保证文件存在或运行窗口与源码对应，仍检查结果中的缺口。`query:{automationId:"SaveButton",maxSearchNodes:1000}` 是规范示例；`query:{automationID:"SaveButton"}` 缺少规范定位条件，仍会报错。没有未列出的 UI 工具别名。 @@ -58,4 +60,50 @@ captureQuality 在标注前检查原始像素，最多采样 1024 点；suspect- 多个控件若指向同一文件、相邻赋值，可在确认文件未变化后复用当前会话已展示的精确行；有缺口时合并为一次有界 lineRanges 请求。不要因为每个候选都带 nextRequest 就机械重复读取。复用仅限已经核对的正文，不代表这些运行时 UI 证据获得了跨调用有效期保证，也不扩大运行时绑定结论。 -内置 Host 强制显示半透明 REC/WinCoding 标志并记录极简审计，不绕过。审计提示按诊断手册处理。取证不授权点击、输入或读取其他窗口。启动测试应用时使用专用隔离数据模式；已有明确的固定测试 profile 时复用它，避免每次重复初始化，禁止使用个人数据库。 +## 操作方式与授权 + +先按任务选定模式，再调用。 + +| 任务 | 用法 | +| --- | --- | +| 查看界面、定位控件 | `wincode_ui_inspect` | +| 后台点击、勾选、选择条目 | `wincode_ui_click`：Invoke/Toggle/SelectionItem，不需要前台 | +| 后台写入或清空输入框 | `wincode_ui_type` 且**显式**传 `mode:"setValue"`：ValuePattern，不需要前台 | +| 验证逐键输入、快捷键、IME 行为 | `wincode_ui_type` 的 `mode:"type"`：需要前台授权 | +| 控件不支持 ValuePattern 的后台写入 | 报告该步骤无法后台完成，不自动改走键盘输入 | + +用户要求执行或测试明确的界面流程时，自主完成定位、点击、输入与结果检查，不为每个常规步骤重复确认；用户只要求评估、查看或审查时保持只读。动作会改变目标应用状态，按影响判断是否需要额外确认：超出任务范围的发布、发送、删除或真实业务提交，先说明再执行。 + +`mode:"type"` 会向目标控件索取键盘焦点（UIA SetFocus），可能把该窗口带到前台并中断用户当前输入，因此只在用户授权前台交互时使用： + +- 用户只授权后台测试时固定用 `setValue`，不因为 `type` 更接近真实输入而擅自切换。 +- 授权前确认用户当前没有正在使用该应用。用户正在游戏、聊天（QQ/微信）、会议或演示中时，不索取焦点，也不建议用户为此切走；说明需要前台键盘输入并询问何时方便，由用户决定。 +- 授权后由用户把目标窗口切到前台，工具只做一次焦点确认。`FOCUS_FAILED` 表示当时没有确认到焦点：报告它并等用户处理，不重复重试，也不用脚本、快捷键模拟或窗口置顶代替用户切换。 +- 前台路径只为验证键盘相关行为；业务结果仍用后台 `inspect` 复核。 + +## 语义操作 + +`wincode_ui_click` 与 `wincode_ui_type` 只操作同一次有界搜索证明唯一的控件，命中 0 个或多个时不做任何操作。选择器沿用取证时的 `targetAutomationId`/`targetName`/`targetControlType`（区分大小写的精确 AND 条件）与同一个 PID/HWND。 + +执行一个动作： + +1. 用当前 UI 证据确定目标与预期结果；已有可靠 PID/HWND 就直接用，窗口关闭或句柄失效才重新列窗。 +2. 动作后检查预先确定的结果：控件状态、页面变化、新窗口或提示文本。需要等待时做有界只读检查，不重发原动作。 +3. 结果符合预期才继续；验收条件满足后停止。观察不到就报告未验证，不把 `success:true` 当成测试通过。 + +只用 UIA 控件模式与键盘输入：不做坐标鼠标模拟，不以 Shell 方式激活、还原或置顶窗口。结果回报 `actionMethod`（`InvokePattern`/`TogglePattern`/`SelectionItemPattern`/`ValuePattern`/`keyboard:type`/`keyboard:clear+type`）、`actionTarget` 与 `inputLength`；**不回显输入文本**。`clearBefore` 仅 `type` 有效，`setValue` 传空字符串即可清空值。 + +| 情况 | 处理 | +| --- | --- | +| `TARGET_NOT_FOUND` / `TARGET_AMBIGUOUS` | 重新观察并收紧选择器，有新证据后再试；不逐个试点 | +| `TARGET_SEARCH_INCOMPLETE` | 扩大预算或收窄范围，不把已有的单个匹配当成唯一 | +| `TARGET_DISABLED` / `NO_CLICK_PATTERN` / `NO_VALUE_PATTERN` / `VALUE_READONLY` | 检查流程前提，或改用受支持的路径 | +| `FOCUS_FAILED` | 保持后台约束；只有已授权前台交互时才考虑前台路径 | +| `ACTION_FAILED`、执行中超时或取消 | **可能已经部分完成**：`type` 的 `clearBefore` 可能已清空旧内容，`click` 的调用方也可能已执行一部分；先只读观察实际状态，再决定继续、修正或报告 | +| `SERVER_BUSY` 等繁忙拒绝 | `workStarted:false`，未启动 Helper；按诊断手册处理，不自动重放 | + +只有 `TARGET_*`、`TARGET_DISABLED`、`NO_*_PATTERN`、`VALUE_READONLY` 与 `FOCUS_FAILED` 发生在实际调用之前；`ACTION_FAILED` 不等于未执行。常见失败不必都交还用户：勾选失败后先读勾选状态，已达到目标就不再 toggle。 + +操作与只读取证共用受理槽、互斥、超时与 Helper 回收；请求在收尾阶段过期时仍返回已完成动作的真实结果。审计 `op` 记为 `click`/`type`/`setValue`。旧 Helper 不具备该能力时按 `VERSION_MISMATCH` 拒绝，不把参数改成普通 inspect 冒充成功。 + +内置 Host 强制显示半透明 REC/WinCoding 标志并记录极简审计，不绕过。审计提示按诊断手册处理。启动测试应用时使用专用隔离数据模式；已有固定测试 profile 时复用它，禁止使用个人数据库。 diff --git a/src/Adapters/FlaUiAdapter.ts b/src/Adapters/FlaUiAdapter.ts index 7e6c632..89a6d64 100644 --- a/src/Adapters/FlaUiAdapter.ts +++ b/src/Adapters/FlaUiAdapter.ts @@ -17,6 +17,9 @@ import { import { UiInspectRequest, validateUiQuery, + validateUiAction, + isUiAction, + UI_INSPECTION_VERSIONS, UiInspectResult, UiErrorCodes, UI_INSPECT_DEFAULTS, @@ -252,6 +255,17 @@ export class FlaUiAdapter implements IAdapter { return this.inspect({ ...request, action: 'listWindows', timeoutMs: 3000 }, signal, operation); } + /** + * 语义操作与只读取证共享同一互斥、超时、取消与进程回收路径； + * 差异只在请求校验与结果形状，避免出现第二套 Helper 生命周期。 + */ + async performUiAction( + request: UiInspectRequest, signal?: AbortSignal, operation?: OperationContext + ): Promise { + const result = await this.inspect(request, signal, operation); + return result; + } + async inspect( request: UiInspectRequest, signal?: AbortSignal, operation?: OperationContext ): Promise { @@ -277,6 +291,11 @@ export class FlaUiAdapter implements IAdapter { const requestId = request.requestId || randomUUID(); try { validateUiQuery(request.query, request.readStates); } catch (error) { return {schemaVersion: "1.0", protocolVersion: "1.0", requestId, success: false, errorCode: UiErrorCodes.INVALID_ARGUMENT, errorMessage: (error as Error).message}; } + if (isUiAction(request.action)) { + // 拒绝必须发生在启动 Helper 之前：破坏性动作没有"参数不对但仍先执行"的余地。 + try { validateUiAction(request); } + catch (error) { return {schemaVersion: "1.0", protocolVersion: "1.0", requestId, success: false, errorCode: UiErrorCodes.INVALID_ARGUMENT, errorMessage: (error as Error).message}; } + } const normRequest: UiInspectRequest & { requestId: string } = { ...request, requestId, @@ -410,11 +429,19 @@ export class FlaUiAdapter implements IAdapter { }; } // Old/custom helpers must not silently ignore a scoped query and return a whole window. - if ((request.query || request.readStates) && parsed.success && parsed.inspectionVersion !== 2) { + if ((request.query || request.readStates) && parsed.success && (parsed.inspectionVersion ?? 0) < UI_INSPECTION_VERSIONS.QUERY_AND_STATES) { return { schemaVersion: '1.0', protocolVersion: '1.0', requestId: request.requestId, success: false, errorCode: UiErrorCodes.VERSION_MISMATCH, errorMessage: 'Query/state inspection requires a v0.9 helper (inspectionVersion 2).', auditNotice: parsed.auditNotice }; } + // 旧 Helper 不认得 action 字段，会退化成一次只读取证并返回 success=true。 + // 那种结果不能当作操作已执行，必须在协议层按版本拒绝。 + if (isUiAction(request.action) && parsed.success && (parsed.inspectionVersion ?? 0) < UI_INSPECTION_VERSIONS.ACTIONS) { + return { schemaVersion: '1.0', protocolVersion: '1.0', requestId: request.requestId, + success: false, errorCode: UiErrorCodes.VERSION_MISMATCH, + errorMessage: `The ${request.action} action requires an inspectionVersion ${UI_INSPECTION_VERSIONS.ACTIONS} helper; this helper ignored the requested action.`, + auditNotice: parsed.auditNotice }; + } return parsed; } catch (jsonErr) { return { @@ -481,6 +508,12 @@ export class FlaUiAdapter implements IAdapter { maxDepth: request.maxDepth ?? this.config.adapters.flaui?.maxDepth ?? UI_INSPECT_DEFAULTS.MAX_DEPTH, maxNodes: request.maxNodes ?? this.config.adapters.flaui?.maxNodes ?? UI_INSPECT_DEFAULTS.MAX_NODES, timeoutMs: timeoutMs, + // 语义操作字段按 action 透传；未提供的字段不写入 payload，避免被下游当成显式条件。 + targetAutomationId: request.targetAutomationId, + targetName: request.targetName, + targetControlType: request.targetControlType, + inputText: request.inputText, + clearBefore: request.clearBefore, }); let stdoutData = ''; diff --git a/src/Core/Config.ts b/src/Core/Config.ts index f3d4c97..1690c34 100644 --- a/src/Core/Config.ts +++ b/src/Core/Config.ts @@ -1,6 +1,6 @@ import path from 'node:path'; -export const WINCODE_VERSION = '0.15.0'; +export const WINCODE_VERSION = '0.16.0'; /** * Bounded waits for every external process/RPC. None of these may be Infinity. diff --git a/src/Core/ToolRouter.ts b/src/Core/ToolRouter.ts index 4e451e6..5521a67 100644 --- a/src/Core/ToolRouter.ts +++ b/src/Core/ToolRouter.ts @@ -609,6 +609,11 @@ export class ToolRouter { return this.flaui.inspect(request, signal, this.admission.operation(signal)); } + /** 破坏性动作共用 inspect 的受理、互斥与超时；动作语义由 request.action 决定。 */ + async performUiAction(request: UiInspectRequest, signal?: AbortSignal): Promise { + return this.flaui.performUiAction(request, signal, this.admission.operation(signal)); + } + async listUiWindows(request: import('./UiContracts.js').UiListWindowsRequest, signal?: AbortSignal): Promise { return this.flaui.listWindows(request, signal, this.admission.operation(signal)); } diff --git a/src/Core/UiContracts.ts b/src/Core/UiContracts.ts index 08c27a6..0e0fac1 100644 --- a/src/Core/UiContracts.ts +++ b/src/Core/UiContracts.ts @@ -1,5 +1,6 @@ /** - * Bounded Windows UI inspection contracts. Optional query/state fields require inspectionVersion 2. + * Bounded Windows UI inspection contracts. Optional query/state fields require inspectionVersion 2; + * semantic actions (click/type/setValue) require inspectionVersion 3. */ export interface UiRect { @@ -41,6 +42,40 @@ export interface UiQuery { maxSearchNodes?: number; maxMatches?: number; } +export type UiAction = 'click' | 'type' | 'setValue'; + +/** 取证结构版本：2 增加 query/readStates，3 增加语义操作。旧 Helper 不得被当作新能力。 */ +export const UI_INSPECTION_VERSIONS = { QUERY_AND_STATES: 2, ACTIONS: 3 } as const; + +export function isUiAction(action: unknown): action is UiAction { + return action === 'click' || action === 'type' || action === 'setValue'; +} + +/** + * 语义操作的共享边界校验：在启动原生 Helper 之前拒绝缺少唯一目标或输入不合规的请求。 + * 只有同一次有界搜索证明唯一的控件才允许被操作，因此至少需要一个精确定位条件。 + */ +export function validateUiAction(request: UiInspectRequest): void { + if (!isUiAction(request.action)) throw new Error('validateUiAction requires a ui action.'); + // 查询与状态读取属于取证范围；与破坏性动作混用会让调用方误以为动作被限定在同一范围内。 + if (request.query !== undefined || request.readStates === true) + throw new Error('query/readStates describe inspection and are not accepted for an action.'); + const selectors = [request.targetAutomationId, request.targetName, request.targetControlType]; + if (!selectors.some(value => value !== undefined)) throw new Error('A target selector (targetAutomationId, targetName or targetControlType) is required.'); + for (const value of selectors) + if (value !== undefined && (typeof value !== 'string' || !value.trim() || value.length > 256 || /[\x00-\x1f]/.test(value))) + throw new Error('Invalid target selector.'); + if (request.clearBefore !== undefined && typeof request.clearBefore !== 'boolean') throw new Error('clearBefore must be boolean.'); + if (request.clearBefore === true && request.action !== 'type') throw new Error('clearBefore is only supported for the type action.'); + if (request.action === 'click') { + if (request.inputText !== undefined) throw new Error('inputText is not accepted for the click action.'); + return; + } + if (typeof request.inputText !== 'string' || request.inputText.length > 4096) throw new Error('inputText is required and must be at most 4096 characters.'); + // type 需要真实按键输入，空文本无意义；setValue 允许用空字符串清空值。 + if (request.action === 'type' && request.inputText.length === 0) throw new Error('inputText must not be empty for the type action.'); +} + /** Shared MCP/adapter boundary; rejected scopes must never launch the native helper. */ export function validateUiQuery(query: unknown, readStates: unknown): void { if (readStates !== undefined && typeof readStates !== "boolean") throw new Error("readStates must be boolean."); @@ -64,7 +99,7 @@ export interface UiInspectRequest { readStates?: boolean; schemaVersion?: string; requestId?: string; - action?: 'inspect' | 'health' | 'ping' | 'listWindows'; + action?: 'inspect' | 'health' | 'ping' | 'listWindows' | UiAction; processName?: string; titleContains?: string; maxWindows?: number; @@ -75,6 +110,14 @@ export interface UiInspectRequest { maxDepth?: number; maxNodes?: number; timeoutMs?: number; + /** 语义操作的目标定位条件；click/type/setValue 至少需要一个。 */ + targetAutomationId?: string; + targetName?: string; + targetControlType?: string; + /** type 的按键文本或 setValue 的写入值；结果只回报长度，不回显内容。 */ + inputText?: string; + /** 仅 type 有效：先清空目标控件的既有内容。 */ + clearBefore?: boolean; } export type UiTruncateReason = 'maxDepth' | 'maxNodes' | 'timeout' | 'budgetLimit' | 'maxWindows' | 'enumerationFailed'; @@ -98,6 +141,12 @@ export interface UiInspectResult { success: boolean; action?: string; status?: string; + /** 实际使用的 UIA 模式或输入方式；只描述执行方式，不声明应用已产生预期副作用。 */ + actionMethod?: string; + actionTarget?: { propertyIssues?: string[]; automationId?: string; name?: string; controlType?: string; + className?: string; bounds?: UiRect; isEnabled?: boolean; isOffscreen?: boolean }; + /** 已接受的输入字符数；输入内容本身不回显。 */ + inputLength?: number; pid?: number; hwnd?: string; captureOrigin?: UiRect; @@ -153,6 +202,17 @@ export const UiErrorCodes = { PLATFORM_NOT_SUPPORTED: 'PLATFORM_NOT_SUPPORTED', BUSY: 'BUSY', SHUTDOWN: 'SHUTDOWN', + // 语义操作（click/type/setValue）结果码。只有唯一命中的目标才允许被操作。 + TARGET_NOT_FOUND: 'TARGET_NOT_FOUND', + TARGET_AMBIGUOUS: 'TARGET_AMBIGUOUS', + TARGET_SEARCH_INCOMPLETE: 'TARGET_SEARCH_INCOMPLETE', + TARGET_DISABLED: 'TARGET_DISABLED', + NO_CLICK_PATTERN: 'NO_CLICK_PATTERN', + NO_VALUE_PATTERN: 'NO_VALUE_PATTERN', + VALUE_READONLY: 'VALUE_READONLY', + FOCUS_FAILED: 'FOCUS_FAILED', + ACTION_FAILED: 'ACTION_FAILED', + UNKNOWN_ACTION: 'UNKNOWN_ACTION', } as const; export type UiErrorCode = (typeof UiErrorCodes)[keyof typeof UiErrorCodes]; diff --git a/src/Gateway/UiTools.ts b/src/Gateway/UiTools.ts index 5d8e074..dfecfad 100644 --- a/src/Gateway/UiTools.ts +++ b/src/Gateway/UiTools.ts @@ -1,6 +1,7 @@ import { defineTool, jsonResult } from './ToolDefinition.js'; import { uiResponse } from './UiResponse.js'; -import { UI_INSPECT_DEFAULTS, validateUiQuery, validateWindowQuery, type UiInspectRequest, type UiListWindowsRequest } from '../Core/UiContracts.js'; +import { UI_INSPECT_DEFAULTS, validateUiQuery, validateWindowQuery, validateUiAction, + type UiAction, type UiInspectRequest, type UiListWindowsRequest } from '../Core/UiContracts.js'; import { validateCandidateFiles } from '../Core/UiSourceMapper.js'; import { validateCandidateCodeFiles } from '../Core/UiCodeMapper.js'; import { validateTextQueries } from '../Core/UiTextSearch.js'; @@ -12,9 +13,40 @@ function validateInspect(args: UiInspectRequest): void { if (args.hwnd !== undefined && !args.hwnd.trim()) throw new Error('hwnd must be non-empty.'); } +/** 动作工具在任何受理与进程启动之前先做完整契约校验。 */ +function validateActionRequest(args: UiActionArgs, action: UiAction): void { + if (args.hwnd !== undefined && (typeof args.hwnd !== 'string' || !args.hwnd.trim())) throw new Error('hwnd must be non-empty.'); + validateUiAction({ + action, pid: args.pid, hwnd: args.hwnd, + targetAutomationId: args.targetAutomationId, targetName: args.targetName, + targetControlType: args.targetControlType, inputText: args.inputText, clearBefore: args.clearBefore, + }); +} + type UiInspectArgs = UiInspectRequest & { responseFormat?: 'full' | 'compact' }; type UiReviewArgs = UiInspectArgs & { candidateFiles: string[]; candidateCodeFiles?: string[]; textQueries?: string[] }; +/** 动作参数：定位与输入字段与取证参数分开声明，避免把 query/readStates 误当成动作范围。 */ +type UiActionArgs = { + pid?: number; hwnd?: string; + targetAutomationId?: string; targetName?: string; targetControlType?: string; + inputText?: string; clearBefore?: boolean; +}; +type UiTypeArgs = UiActionArgs & { inputText: string; mode?: 'type' | 'setValue' }; + +const actionTargetProperties = { + pid: { type: 'integer', minimum: 1, + description: 'Process ID of the target Windows desktop application.' }, + hwnd: { type: 'string', maxLength: 32, + description: 'Window handle of the target window (hex e.g. "0x00120ABC" or decimal string).' }, + targetAutomationId: { type: 'string', minLength: 1, maxLength: 256, + description: 'Exact case-sensitive AutomationId of the single control to act on.' }, + targetName: { type: 'string', minLength: 1, maxLength: 256, + description: 'Exact case-sensitive Name of the single control to act on.' }, + targetControlType: { type: 'string', minLength: 1, maxLength: 256, + description: 'Exact case-sensitive control type (for example "Button"); combine with other fields to stay unique.' }, +} as const; + const invalidArguments = (errorMessage: string) => jsonResult({ schemaVersion: '1.0', protocolVersion: '1.0', success: false, errorCode: 'INVALID_ARGUMENT', errorMessage }, false, true); @@ -88,6 +120,58 @@ const inspectDefinition = defineTool({ }); const uiInspectTool = inspectDefinition.tool; +const clickDefinition = defineTool({ + name: 'wincode_ui_click', + description: 'Clicks exactly one Windows UI Automation control identified by an exact selector. Only UI Automation Invoke/Toggle/SelectionItem patterns are used: no mouse simulation, no window activation. The selector must match one control inside the target window, otherwise nothing is clicked. Requires either pid or hwnd.', + annotations: { readOnlyHint: false, destructiveHint: true }, + inputSchema: { + type: 'object', additionalProperties: true, + properties: actionTargetProperties, + anyOf: [{ required: ['pid'] }, { required: ['hwnd'] }], + }, +}, { + invalidArguments, + validate: args => validateActionRequest(args, 'click'), + requestBudget: 'ui', + // 点击可能已经发生：请求在收尾阶段过期也不能把副作用报告成未执行。 + preserveOutcomeOnInterruption: true, + execute: async (args, { router, signal }) => { + const result = await router.performUiAction({ ...args, hwnd: args.hwnd?.trim(), action: 'click' }, signal); + return jsonResult(result, false, !result.success); + }, +}); + +const typeDefinition = defineTool({ + name: 'wincode_ui_type', + description: 'Writes text into exactly one Windows UI Automation control identified by an exact selector. Prefer mode="setValue" for background work: it writes through ValuePattern, needs no keyboard focus, and accepts an empty string to clear the value. mode="type" requests keyboard focus on the control, which may bring its window to the front: use it only when the user approved foreground interaction. Text is never echoed back. Requires either pid or hwnd.', + annotations: { readOnlyHint: false, destructiveHint: true }, + inputSchema: { + type: 'object', additionalProperties: true, + properties: { + ...actionTargetProperties, + inputText: { type: 'string', maxLength: 4096, + description: 'Text to write. mode="type" requires at least one character; mode="setValue" also accepts an empty string, which clears the value. It is never returned in the result.' }, + clearBefore: { type: 'boolean', default: false, + description: 'mode="type" only: select and delete the existing content before typing.' }, + mode: { type: 'string', enum: ['type', 'setValue'], default: 'type', + description: 'type (default) sends keyboard input and needs confirmed focus; setValue writes the value through ValuePattern without keyboard focus.' }, + }, + required: ['inputText'], + anyOf: [{ required: ['pid'] }, { required: ['hwnd'] }], + }, +}, { + invalidArguments, + validate: args => validateActionRequest(args, args.mode === 'setValue' ? 'setValue' : 'type'), + requestBudget: 'ui', + preserveOutcomeOnInterruption: true, + execute: async (args, { router, signal }) => { + const { mode, ...rest } = args; + const action = mode === 'setValue' ? 'setValue' : 'type'; + const result = await router.performUiAction({ ...rest, hwnd: rest.hwnd?.trim(), action }, signal); + return jsonResult(result, false, !result.success); + }, +}); + export const UI_TOOLS = [ defineTool({ name: 'wincode_ui_list_windows', @@ -113,6 +197,8 @@ export const UI_TOOLS = [ }, }), inspectDefinition, + clickDefinition, + typeDefinition, defineTool({ name: 'wincode_ui_review', description: 'Collects one UI snapshot and literal AutomationId candidates in supplied WPF XAML files. Optional C# files provide Click/simple Binding candidates and scoped next requests. Reports ambiguity; runtime/source identity and binding causality remain unverified.', diff --git a/tests/fixtures/wpf-ui-review/MainWindow.xaml.cs b/tests/fixtures/wpf-ui-review/MainWindow.xaml.cs index 9d4def5..d5e3c98 100644 --- a/tests/fixtures/wpf-ui-review/MainWindow.xaml.cs +++ b/tests/fixtures/wpf-ui-review/MainWindow.xaml.cs @@ -91,6 +91,69 @@ public MainWindow() } Content = panel; } + if (Environment.GetCommandLineArgs().Contains("--action-fixture")) + { + // 语义操作夹具：在 code-behind 中替换内容，不改 XAML，避免移动源码审查断言的 XAML 行号。 + // actionEcho 是 TextBlock，其 UIA Name 就是文本，因此"点击/输入是否真的到达应用" + // 可以由独立的只读取证观察到，而不是只看操作工具自己报告的 success。 + var panel = new StackPanel { Margin = new Thickness(16) }; + void Add(FrameworkElement control, string id) + { + System.Windows.Automation.AutomationProperties.SetAutomationId(control, id); + panel.Children.Add(control); + } + var echo = new TextBlock { Text = "idle" }; + Add(echo, "actionEcho"); + + // 键盘输入要求目标控件持有键盘焦点。夹具在自己的测试模式下提供一次显式获取焦点的机会， + // 让"目标应用本来就持有键盘焦点"这一前提在测试中可复现；产品 Host 绝不激活目标窗口， + // 无法确认焦点时直接拒绝输入。 + var focusGate = new System.Windows.Threading.DispatcherTimer { Interval = TimeSpan.FromMilliseconds(200) }; + var focusAttempts = 0; + focusGate.Tick += (_, _) => + { + if (IsActive || ++focusAttempts > 25) { focusGate.Stop(); return; } + Activate(); + }; + var focusButton = new Button { Content = "Focus Window", Height = 30 }; + focusButton.Click += (_, _) => { Activate(); focusGate.Start(); }; + Add(focusButton, "actionFocus"); + + var clicks = 0; + var increment = new Button { Content = "Increment", Height = 30 }; + increment.Click += (_, _) => echo.Text = $"clicked:{++clicks}"; + Add(increment, "actionIncrement"); + + // 两个控件共用同一 AutomationId：语义操作必须拒绝歧义，而不是挑第一个。 + var duplicateFirst = new Button { Content = "Duplicate", Height = 30 }; + duplicateFirst.Click += (_, _) => echo.Text = "ambiguous-click"; + Add(duplicateFirst, "actionDuplicate"); + var duplicateSecond = new Button { Content = "Duplicate", Height = 30 }; + duplicateSecond.Click += (_, _) => echo.Text = "ambiguous-click"; + Add(duplicateSecond, "actionDuplicate"); + + var toggle = new CheckBox { Content = "Toggle" }; + toggle.Checked += (_, _) => echo.Text = "toggled:true"; + toggle.Unchecked += (_, _) => echo.Text = "toggled:false"; + Add(toggle, "actionToggle"); + + var disabled = new Button { Content = "Disabled", IsEnabled = false, Height = 30 }; + disabled.Click += (_, _) => echo.Text = "disabled-click"; + Add(disabled, "actionDisabled"); + + var readOnly = new TextBox { Text = "read-only", IsReadOnly = true, Height = 28 }; + Add(readOnly, "actionReadOnly"); + + var input = new TextBox { Text = "initial", Height = 28 }; + input.TextChanged += (_, _) => echo.Text = "text:" + input.Text; + Add(input, "actionInput"); + + var valueTarget = new TextBox { Text = "initial", Height = 28 }; + valueTarget.TextChanged += (_, _) => echo.Text = "value:" + valueTarget.Text; + Add(valueTarget, "actionValue"); + + Content = panel; + } if (Environment.GetCommandLineArgs().Contains("--query-fixture")) { var panel = new StackPanel(); void Add(FrameworkElement control, string id) { diff --git a/tests/request-admission.test.ts b/tests/request-admission.test.ts index 49dd4d8..e877414 100644 --- a/tests/request-admission.test.ts +++ b/tests/request-admission.test.ts @@ -271,7 +271,7 @@ it('cancelled startup waiters are removed and passive requests stay available du const controllers = Array.from({ length: 8 }, () => new AbortController()); const pending = controllers.map(c => call('wincode_find_code_symbol', { query: 'Api' }, c.signal).catch(e => e)); await until(() => router.admission.snapshot().sharedWaiters === 8, 'startup waiters must be observable'); - assert.notEqual((await call('wincode_hello_world')).isError, true); assert.equal((await client.listTools()).tools.length, 17); + assert.notEqual((await call('wincode_hello_world')).isError, true); assert.equal((await client.listTools()).tools.length, 19); controllers.forEach(c => c.abort()); await Promise.all(pending); await until(() => router.admission.pendingCount === 0, 'cancelled startup calls must release capacity'); assert.equal(router.admission.snapshot().sharedWaiters, 0); @@ -372,7 +372,7 @@ it('128-call burst admits 32, rejects overflow before execution, and preserves F const health = body(await call('wincode_hello_world')).health; assert.equal(health.admission.business.active, 32); assert.equal(health.admission.business.waiting, 31); assert.equal(health.admission.business.executing, 1); - assert.equal((await client.listTools()).tools.length, 17); + assert.equal((await client.listTools()).tools.length, 19); assert.equal(body(await call('workspace_open', { path: path.join(root, 'other') })).errorCode, 'WORKSPACE_MISMATCH'); assert.equal(body(await call('workspace_open', { path: root })).errorCode, 'SERVER_BUSY'); assert.equal((await router.releaseRoslynMemory()).status, 'busy'); diff --git a/tests/tool-contracts.test.ts b/tests/tool-contracts.test.ts index 652b352..54482e7 100644 --- a/tests/tool-contracts.test.ts +++ b/tests/tool-contracts.test.ts @@ -130,6 +130,8 @@ const examples: Record> = { wincode_diagnose_project: {}, wincode_plan_refactoring: { target: 'Target', goal: 'Improve reliability' }, wincode_safe_move_to_trash: { filePath: 'Target.ts', reason: 'fixture' }, wincode_ui_list_windows: { pid: 5 }, wincode_ui_inspect: { pid: 5, query: { name: 'Save' } }, + wincode_ui_click: { pid: 5, targetAutomationId: 'btnSave' }, + wincode_ui_type: { pid: 5, targetAutomationId: 'txtUser', inputText: 'fixture text', mode: 'setValue' }, wincode_ui_review: { pid: 5, candidateFiles: ['View.xaml'], candidateCodeFiles: ['View.cs'], textQueries: ['Save'] }, }; @@ -152,6 +154,8 @@ const expectedCalls: Record = { wincode_safe_move_to_trash: { method: 'moveToTrash', args: ['Target.ts', 'fixture', ''] }, wincode_ui_list_windows: { method: 'listUiWindows', args: [{ pid: 5 }, ''] }, wincode_ui_inspect: { method: 'inspectUi', args: [{ pid: 5, query: { name: 'Save' }, hwnd: undefined }, ''] }, + wincode_ui_click: { method: 'performUiAction', args: [{ pid: 5, targetAutomationId: 'btnSave', hwnd: undefined, action: 'click' }, ''] }, + wincode_ui_type: { method: 'performUiAction', args: [{ pid: 5, targetAutomationId: 'txtUser', inputText: 'fixture text', hwnd: undefined, action: 'setValue' }, ''] }, wincode_ui_review: { method: 'reviewUi', args: [{ pid: 5, hwnd: undefined }, ['View.xaml'], '', ['Save'], ['View.cs']] }, }; @@ -173,8 +177,9 @@ it('calls all published tools and the hidden alias; unknown fields do not reach stub('listUiWindows', { success: true, windows: [] }); stub('inspectUi', { success: true }); stub('reviewUi', { success: true }); + stub('performUiAction', { success: true }); const published = (await client.listTools()).tools; - assert.equal(published.length, 17); + assert.equal(published.length, 19); assert.ok(!published.some(tool => tool.name === 'wincode_workspace_open')); assert.deepEqual(new Set([...published.map(tool => tool.name), 'wincode_workspace_open']), new Set(Object.keys(examples))); const responses = new Map(); @@ -204,7 +209,7 @@ it('calls all published tools and the hidden alias; unknown fields do not reach it('validates every declared top-level argument without coercion before admission', async () => fixture(async (client, router, admissions) => { let calls = 0; for (const method of ['openWorkspace', 'listDirectory', 'analyzeWorkspace', 'findCodeSymbols', 'findCodeReferences', 'diagnoseProject', - 'planRefactoring', 'prepareContext', 'moveToTrash', 'analyzeChangeImpact', 'getRuntimeHealth', 'listUiWindows', 'inspectUi', 'reviewUi']) + 'planRefactoring', 'prepareContext', 'moveToTrash', 'analyzeChangeImpact', 'getRuntimeHealth', 'listUiWindows', 'inspectUi', 'performUiAction', 'reviewUi']) (router as any)[method] = async () => { calls++; return {}; }; const tools = (await client.listTools()).tools; const workspace = tools.find(tool => tool.name === 'workspace_open')!; diff --git a/tests/ui-action-mcp.test.ts b/tests/ui-action-mcp.test.ts new file mode 100644 index 0000000..28c0fc3 --- /dev/null +++ b/tests/ui-action-mcp.test.ts @@ -0,0 +1,199 @@ +import { describe, it, before, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawn, ChildProcess } from 'node:child_process'; +import fs from 'node:fs'; +import fsp from 'node:fs/promises'; +import path from 'node:path'; +import { Client, InMemoryTransport } from '@modelcontextprotocol/client'; +import { WinCodeMcpServer } from '../src/Gateway/McpServer.js'; +import { ToolRouter } from '../src/Core/ToolRouter.js'; +import { getDefaultConfig } from '../src/Core/Config.js'; +import { FlaUiAdapter } from '../src/Adapters/FlaUiAdapter.js'; +import { killProcessTree } from '../src/Core/ResourceManager.js'; + +const root = process.cwd(); + +/** 只允许 ASCII 键盘输入：FlaUI 的 Keyboard.Type(string) 无法为非 ASCII 字符构造按键事件。 */ +const TYPED_TEXT = 'typed-text'; + +it('ui actions are rejected at the boundary before any helper is launched', async () => { + const adapter = new FlaUiAdapter(getDefaultConfig(root)); + let launched = 0; + (adapter as any).executeHost = async () => { launched++; throw new Error('the helper must not be launched'); }; + const rejected = async (request: Record, expectedMessage: RegExp) => { + const result = await adapter.performUiAction(request as any); + assert.equal(result.success, false, JSON.stringify(request)); + assert.equal(result.errorCode, 'INVALID_ARGUMENT', JSON.stringify(result)); + assert.match(result.errorMessage ?? '', expectedMessage); + }; + try { + // 没有唯一目标就不允许操作；这也是"必须先用 inspect 确认目标"的可执行版本。 + await rejected({ action: 'click', pid: 1 }, /selector/); + await rejected({ action: 'click', pid: 1, targetAutomationId: ' ' }, /selector/); + await rejected({ action: 'click', pid: 1, targetAutomationId: 42 }, /selector/); + await rejected({ action: 'click', pid: 1, targetName: 'x', targetControlType: 'y', query: { name: 'Save' } }, /inspection/); + await rejected({ action: 'click', pid: 1, targetName: 'Save', readStates: true }, /inspection/); + // click 不接受输入文本；type 必须带非空输入；setValue 允许空串清空。 + await rejected({ action: 'click', pid: 1, targetAutomationId: 'a', inputText: 'x' }, /inputText/); + await rejected({ action: 'type', pid: 1, targetAutomationId: 'a', inputText: '' }, /empty/); + await rejected({ action: 'type', pid: 1, targetAutomationId: 'a', inputText: 'x'.repeat(4097) }, /4096/); + await rejected({ action: 'type', pid: 1, targetAutomationId: 'a' }, /inputText/); + await rejected({ action: 'click', pid: 1, targetAutomationId: 'a', clearBefore: true }, /clearBefore/); + await rejected({ action: 'click', pid: 1, targetAutomationId: 'a', clearBefore: 'yes' }, /clearBefore/); + assert.equal(launched, 0, 'rejected action requests must not start the native helper'); + // setValue 的空串用于清空值：它必须通过边界校验并真的走到 Helper，而不是被提前拒绝。 + const cleared = await adapter.performUiAction({ action: 'setValue', pid: 1, targetAutomationId: 'actionValue', inputText: '' } as any); + assert.notEqual(cleared.errorCode, 'INVALID_ARGUMENT', JSON.stringify(cleared)); + assert.equal(launched, 1, 'an empty setValue is a valid request and must reach the native helper'); + } finally { await adapter.dispose(); } +}); + +it('an old helper cannot report a requested action as a successful inspection', async () => { + const adapter = new FlaUiAdapter(getDefaultConfig(root)); + const parse = (hostResponse: unknown, request: Record) => + (adapter as any).parseHostResponse(JSON.stringify(hostResponse), + { requestId: 'fixture', pid: 1, targetAutomationId: 'btnSave', ...request }); + try { + const older = { schemaVersion: '1.0', protocolVersion: '1.0', requestId: 'fixture', + success: true, inspectionVersion: 2, action: 'click', actionMethod: 'InvokePattern' }; + // 旧 Helper 忽略 action 字段后返回的是一次成功的只读取证，绝不能被当作动作已执行。 + const refused = parse(older, { action: 'click' }); + assert.equal(refused.success, false); + assert.equal(refused.errorCode, 'VERSION_MISMATCH'); + // 同一版本下，显式失败仍按 Helper 自己的原因上报，不伪装成版本问题。 + const failed = parse({ ...older, success: false, errorCode: 'WINDOW_NOT_FOUND' }, { action: 'click' }); + assert.equal(failed.errorCode, 'WINDOW_NOT_FOUND'); + // 升级后的 Helper 必须被接受，且新增的 query/state 门不会把 v3 误判为不兼容。 + const current = parse({ ...older, inspectionVersion: 3 }, { action: 'click' }); + assert.equal(current.success, true); + assert.equal(current.actionMethod, 'InvokePattern'); + assert.equal(parse({ ...older, inspectionVersion: 3, queryResult: { status: 'unique', searchComplete: true, visitedNodes: 1, matches: [] } }, + { query: { name: 'Save' } }).success, true); + // 新增的动作门不得放宽既有的 query/状态门：真正更旧的 Helper 仍必须被拒绝。 + assert.equal(parse(older, { query: { name: 'Save' } }).success, true); + assert.equal(parse({ ...older, inspectionVersion: 1 }, { query: { name: 'Save' } }).errorCode, 'VERSION_MISMATCH'); + assert.equal(parse({ ...older, inspectionVersion: undefined }, { readStates: true }).errorCode, 'VERSION_MISMATCH'); + } finally { await adapter.dispose(); } +}); + +describe('semantic UI actions against the real WPF fixture', () => { + const cacheDir = path.resolve(root, 'test-tmp/mcp_ui_action_test'); + let router: ToolRouter; + let server: WinCodeMcpServer; + let client: Client; + let fixture: ChildProcess | null = null; + let pid = 0; + let hwnd = ''; + + before(async () => { + await fsp.mkdir(cacheDir, { recursive: true }); + const config = getDefaultConfig(root); + config.cacheDir = path.join(cacheDir, 'cache'); + router = new ToolRouter(config); + server = new WinCodeMcpServer(router); + await router.initialize(); + const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); + await (server as any).server.connect(serverTransport); + client = new Client({ name: 'ui-action-suite', version: '1.0.0' }, { capabilities: {} }); + await client.connect(clientTransport); + + // 夹具是独立的真实 WPF 应用：验证的是应用自身的副作用，而不是 Host 自报的 success。 + const executable = path.resolve(root, 'tests/fixtures/wpf-ui-review/bin/Release/net10.0-windows/win-x64/publish/wpf-ui-review.exe'); + const built = fs.existsSync(executable); + fixture = spawn(built ? executable : 'dotnet', + built ? ['--action-fixture', '--auto-close=120000'] + : ['run', '--project', 'tests/fixtures/wpf-ui-review/wpf-ui-review.csproj', '--no-build', '--', '--action-fixture', '--auto-close=120000'], + { cwd: root, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: false }); + await new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error('fixture launch timed out waiting for READY')), 20000); + let buffer = ''; + fixture?.stdout?.on('data', (chunk: Buffer) => { + buffer += chunk.toString('utf8'); + const match = buffer.match(/READY\s+(\d+)\s+(0x[0-9a-fA-F]+)/); + if (match) { clearTimeout(timer); pid = parseInt(match[1], 10); hwnd = match[2]; resolve(); } + }); + fixture?.on('error', error => { clearTimeout(timer); reject(error); }); + fixture?.on('close', code => { clearTimeout(timer); reject(new Error(`fixture exited prematurely with code ${code}`)); }); + }); + await new Promise(resolve => setTimeout(resolve, 400)); + }); + + after(async () => { + if (fixture) { await killProcessTree(fixture).catch(() => {}); fixture = null; } + try { await client?.close(); } catch {} + try { await server?.stop(); } catch {} + await fsp.rm(cacheDir, { recursive: true, force: true }).catch(() => {}); + }); + + const call = async (name: string, args: Record) => { + const response = await client.callTool({ name, arguments: args }); + return { isError: response.isError === true, body: JSON.parse((response.content as any)[0].text) }; + }; + /** 独立只读取证：TextBlock 的 UIA Name 就是它的文本，因此能看到动作的真实后果。 */ + const echo = async () => { + const { body } = await call('wincode_ui_inspect', { pid, hwnd, query: { automationId: 'actionEcho' } }); + assert.equal(body.success, true, JSON.stringify(body)); + assert.equal(body.queryResult?.status, 'unique', JSON.stringify(body.queryResult)); + return body.queryResult.matches[0].name as string; + }; + + it('clicks, refuses ambiguous/disabled targets and writes text without echoing it', { timeout: 180000 }, async () => { + assert.equal(await echo(), 'idle'); + + const clicked = await call('wincode_ui_click', { pid, hwnd, targetAutomationId: 'actionIncrement' }); + assert.equal(clicked.isError, false, JSON.stringify(clicked.body)); + assert.equal(clicked.body.success, true); + assert.equal(clicked.body.actionMethod, 'InvokePattern'); + assert.equal(clicked.body.actionTarget.automationId, 'actionIncrement'); + assert.equal(clicked.body.actionTarget.isEnabled, true); + // 副作用必须由应用自身产生，而不是工具自报。 + assert.equal(await echo(), 'clicked:1'); + + const ambiguous = await call('wincode_ui_click', { pid, hwnd, targetAutomationId: 'actionDuplicate' }); + assert.equal(ambiguous.isError, true); + assert.equal(ambiguous.body.errorCode, 'TARGET_AMBIGUOUS'); + assert.equal(await echo(), 'clicked:1', 'an ambiguous selector must not click anything'); + + const disabled = await call('wincode_ui_click', { pid, hwnd, targetAutomationId: 'actionDisabled' }); + assert.equal(disabled.body.errorCode, 'TARGET_DISABLED'); + assert.equal(await echo(), 'clicked:1'); + + const missing = await call('wincode_ui_click', { pid, hwnd, targetAutomationId: 'actionDoesNotExist' }); + assert.equal(missing.body.errorCode, 'TARGET_NOT_FOUND'); + assert.equal(await echo(), 'clicked:1'); + + const toggled = await call('wincode_ui_click', { pid, hwnd, targetAutomationId: 'actionToggle' }); + assert.equal(toggled.body.success, true, JSON.stringify(toggled.body)); + assert.equal(await echo(), 'toggled:true'); + + // 键盘输入需要一个在前台的窗口：显式让夹具取得焦点，而不是让 Host 去激活它。 + const focus = await call('wincode_ui_click', { pid, hwnd, targetAutomationId: 'actionFocus' }); + assert.equal(focus.body.success, true, JSON.stringify(focus.body)); + await new Promise(resolve => setTimeout(resolve, 800)); + + const typed = await call('wincode_ui_type', { pid, hwnd, targetAutomationId: 'actionInput', inputText: TYPED_TEXT, clearBefore: true }); + assert.equal(typed.isError, false, JSON.stringify(typed.body)); + assert.equal(typed.body.actionMethod, 'keyboard:clear+type'); + assert.equal(typed.body.inputLength, TYPED_TEXT.length); + assert.equal(JSON.stringify(typed.body).includes(TYPED_TEXT), false, 'the sent text must not be echoed back'); + assert.equal(await echo(), 'text:' + TYPED_TEXT); + + const written = await call('wincode_ui_type', { pid, hwnd, targetAutomationId: 'actionValue', inputText: 'via-value', mode: 'setValue' }); + assert.equal(written.body.success, true, JSON.stringify(written.body)); + assert.equal(written.body.actionMethod, 'ValuePattern'); + assert.equal(await echo(), 'value:via-value'); + + const readOnly = await call('wincode_ui_type', { pid, hwnd, targetAutomationId: 'actionReadOnly', inputText: 'nope', mode: 'setValue' }); + assert.equal(readOnly.body.success, false); + assert.equal(readOnly.body.errorCode, 'VALUE_READONLY'); + assert.equal(await echo(), 'value:via-value'); + + // 空串是合法的 setValue：schema 不得用 minLength 提前拒绝，且控件值必须真的被清空。 + const typeSchema = (await client.listTools()).tools.find(tool => tool.name === 'wincode_ui_type')!.inputSchema as any; + assert.equal(typeSchema.properties.inputText.minLength, undefined, 'an empty inputText must stay schema-valid for mode=setValue'); + const clearedValue = await call('wincode_ui_type', { pid, hwnd, targetAutomationId: 'actionValue', inputText: '', mode: 'setValue' }); + assert.equal(clearedValue.body.success, true, JSON.stringify(clearedValue.body)); + assert.equal(clearedValue.body.inputLength, 0); + assert.equal(await echo(), 'value:'); + }); +}); diff --git a/tools/WinCode.Code.Host/WinCode.Code.Host.csproj b/tools/WinCode.Code.Host/WinCode.Code.Host.csproj index 39c0cb9..2537c4e 100644 --- a/tools/WinCode.Code.Host/WinCode.Code.Host.csproj +++ b/tools/WinCode.Code.Host/WinCode.Code.Host.csproj @@ -1,6 +1,6 @@ - 0.15.0 + 0.16.0 Exe net10.0 enable diff --git a/tools/WinCode.Tray/WinCode.Tray.csproj b/tools/WinCode.Tray/WinCode.Tray.csproj index 7adf57a..1627f83 100644 --- a/tools/WinCode.Tray/WinCode.Tray.csproj +++ b/tools/WinCode.Tray/WinCode.Tray.csproj @@ -1,6 +1,6 @@ - 0.15.0 + 0.16.0 WinExe net10.0-windows win-x64 diff --git a/tools/WinCode.UIA.Host/Program.cs b/tools/WinCode.UIA.Host/Program.cs index a9a8065..6cbc964 100644 --- a/tools/WinCode.UIA.Host/Program.cs +++ b/tools/WinCode.UIA.Host/Program.cs @@ -91,6 +91,26 @@ public static void Main(string[] args) return; } + if (UiActionExecutor.IsAction(request.Action)) + { + // 破坏性操作：定位条件必须先成立，且审计记录在任何目标读写之前落盘。 + if (!UiActionExecutor.ValidActionRequest(request)) + { + WriteErrorResponse(request.RequestId, "INVALID_ARGUMENT", + "Action requires at least one target selector (targetAutomationId/targetName/targetControlType), " + + "a resolvable pid or hwnd, and an inputText of at most 4096 characters."); + return; + } + var actionTimeoutMs = request.TimeoutMs is > 0 ? request.TimeoutMs.Value : 10000; + currentAudit = UiAudit.Start(request.Pid, request.Hwnd, "none", request.Action!); + using var actionCts = CancellationTokenSource.CreateLinkedTokenSource(owner?.Token ?? CancellationToken.None); + actionCts.CancelAfter(actionTimeoutMs); + using var actionNotice = RecordingIndicator.Show(); + indicatorDisplayed = true; + WriteSuccessResponse(request.RequestId, UiActionExecutor.Execute(request, actionCts.Token)); + return; + } + if (request.Pid <= 0 && string.IsNullOrWhiteSpace(request.Hwnd)) { WriteErrorResponse(request.RequestId, "INVALID_ARGUMENT", "Either 'pid' or 'hwnd' must be provided."); diff --git a/tools/WinCode.UIA.Host/README.md b/tools/WinCode.UIA.Host/README.md index fc9cf0a..40223c1 100644 --- a/tools/WinCode.UIA.Host/README.md +++ b/tools/WinCode.UIA.Host/README.md @@ -1,12 +1,12 @@ # WinCode.UIA.Host -0.15.0 的 Windows UI Automation（FlaUI.UIA3）一次性取证进程。实现入口为 [Program.cs](Program.cs)，面向 Agent 的规范参数见 [UI 手册](../../skills/wincode/references/ui.md)，整体数据流见 [架构说明](../../WinCode-架构与数据流说明.md)。 +0.16.0 的 Windows UI Automation（FlaUI.UIA3）一次性进程：执行有界取证，并在调用方显式请求时执行一个语义 UI 操作。实现入口为 [Program.cs](Program.cs)，面向 Agent 的规范参数见 [UI 手册](../../skills/wincode/references/ui.md)，整体数据流见 [架构说明](../../WinCode-架构与数据流说明.md)。 ## 职责和边界 -接收 stdin JSON，执行有界窗口发现或 UIA 取证，输出 stdout JSON 后退出。Gateway 的 FlaUIAdapter 管理自有 Host 的超时、取消和进程回收；被检查的应用不属于其进程所有权。原生 Helper 在开始工作前验证所属 Gateway 的进程身份；所属进程退出时取消并按既有宽限清理自身。Gateway 启动保留配置/交付检查，实际 UIA 健康探测延后到显式诊断或首次操作。 +接收 stdin JSON，执行有界窗口发现、UIA 取证或一个语义 UI 操作，输出 stdout JSON 后退出。Gateway 的 FlaUIAdapter 管理自有 Host 的超时、取消和进程回收；被检查的应用不属于其进程所有权。原生 Helper 在开始工作前验证所属 Gateway 的进程身份；所属进程退出时取消并按既有宽限清理自身。Gateway 启动保留配置/交付检查，实际 UIA 健康探测延后到显式诊断或首次操作。 -Host 不点击、不输入、不写目标控件属性，不主动启动或终止目标应用。它会产生自身进程、审计日志及按策略显示的取证提示，因此不应描述为“零副作用”。审计和内容哈希是诊断证据，不是防篡改或来源签名。 +Host 默认只读：不点击、不输入、不写目标控件属性，也不主动启动或终止目标应用。仅当请求显式指定 `click`/`type`/`setValue` 且定位唯一时才操作控件；此时只用 UIA 控件模式（Invoke/Toggle/SelectionItem/Value）与键盘输入，不做坐标鼠标模拟，不以 Shell 方式激活、还原或置顶目标窗口；`type` 会向目标控件索取键盘焦点（可能把该窗口带到前台），无法确认时拒绝输入（`FOCUS_FAILED`）。它会产生自身进程、审计日志及按策略显示的取证提示，因此不应描述为“零副作用”。 ## 请求与结果 @@ -29,7 +29,26 @@ Host 不点击、不输入、不写目标控件属性，不主动启动或终止 } ``` -协议版本 `1.0`、取证结构 `inspectionVersion: 2` 和程序集产品版本是不同概念。实际 Host 的 `hostIdentity` 用于核对版本、构建配置和框架，不能从请求或磁盘文件名推断响应身份。 +语义操作使用同一协议，用 `action` 选择动作，并以 `targetAutomationId`/`targetName`/`targetControlType` 唯一定位控件： + +```json +{ + "schemaVersion": "1.0", + "requestId": "example-click", + "action": "click", + "pid": 12345, + "hwnd": "0x123ABC", + "targetAutomationId": "btnSave", + "timeoutMs": 10000 +} +``` + +`type` 另有必填 `inputText` 与可选 `clearBefore`（仅 `type` 有效）；`setValue` 通过 ValuePattern 写入，不依赖键盘焦点，并允许空字符串用于清空值（`mode:"type"` 则要求非空）。 +这三个取值下 `query`/`readStates`/`capture` 都不参与。结果中的 `actionMethod` 是实际使用的模式，`actionTarget` 是目标控件身份证据， +`inputLength` 是接受的字符数（输入文本不回显）；`success:true` 只代表调用被接受，不代表应用已产生预期副作用，需要证据时另发一次 `inspect`。 +目标缺失、歧义、搜索不完整、控件禁用、无可用模式、只读值或焦点未确认都会以明确错误码拒绝，并发生在实际调用之前。`ACTION_FAILED` 不属于这一类：`type` 的 `clearBefore` 可能已清空旧内容，操作也可能由提供程序部分完成，因此它不是“未执行”，需要先只读观察实际状态。 + +协议版本 `1.0`、取证结构版本（当前为 `inspectionVersion: 3`：2 增加 query/readStates，3 增加语义操作）和程序集产品版本是不同概念。实际 Host 的 `hostIdentity` 用于核对版本、构建配置和框架，不能从请求或磁盘文件名推断响应身份。 结果应结合 `success/errorCode`、目标窗口、搜索完整性与匹配数量、截断原因、状态证据、截图信息和审计状态解释。查询字段按大小写精确 AND 匹配；只有遍历完整且唯一才展开命中子树。截断后的单个候选不能认定唯一，未知状态不等于 false，多窗口歧义不能自动挑第一个。 diff --git a/tools/WinCode.UIA.Host/UiActionExecutor.cs b/tools/WinCode.UIA.Host/UiActionExecutor.cs new file mode 100644 index 0000000..5bd319d --- /dev/null +++ b/tools/WinCode.UIA.Host/UiActionExecutor.cs @@ -0,0 +1,268 @@ +using System.Diagnostics; +using FlaUI.Core; +using FlaUI.Core.AutomationElements; +using FlaUI.Core.Input; +using FlaUI.Core.WindowsAPI; +using FlaUI.UIA3; +using static WinCode.UIA.Host.WindowResolver; + +namespace WinCode.UIA.Host; + +/// +/// 语义化 UI 操作：先用有界搜索取得唯一目标控件，再只用 UIA 控件模式或键盘输入操作它。 +/// 不做坐标鼠标模拟，不激活或还原目标窗口；无法确认键盘焦点落在目标控件时拒绝输入， +/// 避免把调用方文本送进未知窗口。成功只代表调用被接受，不代表应用已产生预期副作用。 +/// +internal static class UiActionExecutor +{ + private const int MaxSearchNodes = 2000; + /// 只需要区分 0/1/多个；命中多个即视为歧义，不继续扩大搜索。 + private const int MaxMatches = 2; + private const int FocusWaitMs = 500; + private const int FocusPollMs = 20; + private const int MaxInputLength = 4096; + private const int MaxMessageLength = 200; + + internal static bool IsAction(string? action) => action is "click" or "type" or "setValue"; + + /// 协议层前置校验：定位条件组合、字段形状与输入文本长度必须先成立。 + internal static bool ValidActionRequest(InspectRequest request) + { + if (request.Pid <= 0 && string.IsNullOrWhiteSpace(request.Hwnd)) return false; + var selectors = new[] { request.TargetAutomationId, request.TargetName, request.TargetControlType }; + if (!selectors.Any(value => value != null)) return false; + if (selectors.Any(value => value != null && !ValidSelector(value!))) return false; + return request.Action switch + { + "click" => request.InputText == null, + "type" => request.InputText is { Length: > 0 and <= MaxInputLength }, + "setValue" => request.InputText is { Length: <= MaxInputLength }, + _ => false, + }; + } + + private static bool ValidSelector(string value) => + !string.IsNullOrWhiteSpace(value) && value.Length <= 256 && !value.Any(character => character < 32); + + internal static InspectResponse Execute(InspectRequest request, CancellationToken cancellationToken) + { + var targetHwnd = ResolveTargetWindow(request, out var resolvedPid, out var candidateWindows, out var resolveError); + if (targetHwnd == IntPtr.Zero) + return Failed(request, resolveError ?? "WINDOW_NOT_FOUND", + $"Could not resolve target window for PID {request.Pid} / HWND {request.Hwnd}.", candidateWindows); + + using var automation = new UIA3Automation(); + var root = automation.FromHandle(targetHwnd); + if (root == null) + return Failed(request, "UIA_ELEMENT_NOT_AVAILABLE", + "Unable to create UIA AutomationElement from target window handle."); + + var identity = new ResolvedWindow(resolvedPid > 0 ? resolvedPid : request.Pid, $"0x{targetHwnd.ToInt64():X}"); + var query = new UiQueryDto + { + AutomationId = request.TargetAutomationId, + Name = request.TargetName, + ControlType = request.TargetControlType, + }; + var walker = automation.TreeWalkerFactory.GetControlViewWalker(); + var search = new BoundedUiSearch(); + search.Run(root, walker.GetFirstChild, walker.GetNextSibling, + element => UiTreeReader.MatchesQuery(element, query), MaxSearchNodes, MaxMatches, cancellationToken); + + if (search.Matches.Count > 1) + return Failed(request, "TARGET_AMBIGUOUS", + "Selector matched multiple controls; only a unique target may be acted on.", identity); + if (!search.Complete) + return Failed(request, "TARGET_SEARCH_INCOMPLETE", + $"Target search stopped early ({search.Reason ?? "unknown"}); uniqueness is unproven.", identity); + if (search.Matches.Count == 0) + return Failed(request, "TARGET_NOT_FOUND", "No control in the target window matched the selector.", identity); + + var target = search.Matches[0]; + var evidence = Describe(target); + cancellationToken.ThrowIfCancellationRequested(); + + return request.Action switch + { + "click" => Click(request, target, evidence, identity), + "type" => Type(request, automation, target, query, evidence, identity, cancellationToken), + "setValue" => SetValue(request, target, evidence, identity), + _ => Failed(request, "UNKNOWN_ACTION", $"Unknown action: {request.Action}", identity), + }; + } + + private static InspectResponse Click(InspectRequest request, AutomationElement target, UiTargetDto evidence, ResolvedWindow identity) + { + if (Disabled(target)) + return Failed(request, "TARGET_DISABLED", "Target control is disabled; no click pattern was invoked.", identity, evidence); + if (target.Patterns.Invoke.TryGetPattern(out var invoke)) + return RunPattern(request, evidence, identity, "InvokePattern", "clicked", () => invoke.Invoke()); + if (target.Patterns.Toggle.TryGetPattern(out var toggle)) + return RunPattern(request, evidence, identity, "TogglePattern", "toggled", () => toggle.Toggle()); + if (target.Patterns.SelectionItem.TryGetPattern(out var selection)) + return RunPattern(request, evidence, identity, "SelectionItemPattern", "selected", () => selection.Select()); + // 坐标鼠标模拟被有意排除：它会移动真实指针、可能激活目标窗口，且无法证明命中了同一个控件。 + return Failed(request, "NO_CLICK_PATTERN", + "Target control exposes none of Invoke, Toggle or SelectionItem; coordinate mouse simulation is not supported.", + identity, evidence); + } + + private static InspectResponse Type(InspectRequest request, UIA3Automation automation, AutomationElement target, + UiQueryDto query, UiTargetDto evidence, ResolvedWindow identity, CancellationToken cancellationToken) + { + if (Disabled(target)) + return Failed(request, "TARGET_DISABLED", "Target control is disabled; no text was sent.", identity, evidence); + if (!FocusConfirmed(automation, target, query, cancellationToken)) + return Failed(request, "FOCUS_FAILED", + "Keyboard focus could not be confirmed on the target control; no text was sent.", + identity, evidence); + + var text = request.InputText!; + try + { + // 焦点已确认落在目标控件，后续键盘输入才会进入该控件。 + if (request.ClearBefore) ClearFocusedInput(); + Keyboard.Type(text); + } + catch (OperationCanceledException) { throw; } + catch (Exception error) + { + return Failed(request, "ACTION_FAILED", Shorten(error.Message), identity, evidence); + } + return Succeeded(request, evidence, identity, request.ClearBefore ? "keyboard:clear+type" : "keyboard:type", "typed", text.Length); + } + + private static InspectResponse SetValue(InspectRequest request, AutomationElement target, UiTargetDto evidence, ResolvedWindow identity) + { + if (Disabled(target)) + return Failed(request, "TARGET_DISABLED", "Target control is disabled; its value was not written.", identity, evidence); + if (!target.Patterns.Value.TryGetPattern(out var value)) + return Failed(request, "NO_VALUE_PATTERN", + "Target control does not expose ValuePattern; use the type action instead.", identity, evidence); + + bool readOnly = false; + try { readOnly = value.IsReadOnly.Value; } + catch (OperationCanceledException) { throw; } + catch { /* 读取失败不构成只读证据，交由写入结果说明。 */ } + if (readOnly) + return Failed(request, "VALUE_READONLY", "Target control reports a read-only value.", identity, evidence); + + try { value.SetValue(request.InputText!); } + catch (OperationCanceledException) { throw; } + catch (Exception error) + { + return Failed(request, "ACTION_FAILED", Shorten(error.Message), identity, evidence); + } + return Succeeded(request, evidence, identity, "ValuePattern", "value-set", request.InputText!.Length); + } + + private static InspectResponse RunPattern(InspectRequest request, UiTargetDto evidence, ResolvedWindow identity, + string method, string status, Action operation) + { + try { operation(); } + catch (OperationCanceledException) { throw; } + catch (Exception error) + { + return Failed(request, "ACTION_FAILED", Shorten(error.Message), identity, evidence); + } + return Succeeded(request, evidence, identity, method, status); + } + + /// 请求焦点后复查 UIA 键盘焦点确实落在同一选择器上，才允许后续输入。 + private static bool FocusConfirmed(UIA3Automation automation, AutomationElement target, UiQueryDto query, CancellationToken cancellationToken) + { + try { target.Focus(); } + catch (OperationCanceledException) { throw; } + catch { return false; } + + var elapsed = Stopwatch.StartNew(); + while (elapsed.ElapsedMilliseconds < FocusWaitMs) + { + cancellationToken.ThrowIfCancellationRequested(); + try + { + var focused = automation.FocusedElement(); + if (focused != null && UiTreeReader.MatchesQuery(focused, query) == true) return true; + } + catch (OperationCanceledException) { throw; } + catch { /* 焦点窗口切换过程中读取失败可重试。 */ } + Thread.Sleep(FocusPollMs); + } + return false; + } + + private static void ClearFocusedInput() + { + Keyboard.TypeSimultaneously(VirtualKeyShort.CONTROL, VirtualKeyShort.KEY_A); + Keyboard.Type(VirtualKeyShort.DELETE); + Thread.Sleep(FocusPollMs); + } + + private static bool Disabled(AutomationElement element) + { + try { return element.Properties.IsEnabled.TryGetValue(out var enabled) && !enabled; } + catch (OperationCanceledException) { throw; } + catch { return false; } + } + + private static UiTargetDto Describe(AutomationElement element) + { + var issues = new List(); + var dto = new UiTargetDto(); + void Read(string name, IAutomationProperty property, Action assign) => + UiPropertyEvidence.Read(name, property, assign, issues); + static string? Clip(string? value) => value?.Length > 256 ? value[..256] : value; + Read("automationId", element.Properties.AutomationId, value => dto.AutomationId = Clip(value)); + Read("name", element.Properties.Name, value => dto.Name = Clip(value)); + Read("className", element.Properties.ClassName, value => dto.ClassName = Clip(value)); + Read("controlType", element.Properties.ControlType, value => dto.ControlType = value.ToString()); + Read("isEnabled", element.Properties.IsEnabled, value => dto.IsEnabled = value); + Read("isOffscreen", element.Properties.IsOffscreen, value => dto.IsOffscreen = value); + Read("bounds", element.Properties.BoundingRectangle, rect => dto.Bounds = new RectDto(rect.X, rect.Y, rect.Width, rect.Height)); + if (issues.Count > 0) dto.PropertyIssues = issues; + return dto; + } + + private static string Shorten(string message) => + message.Length <= MaxMessageLength ? message : message[..MaxMessageLength]; + + private static InspectResponse Succeeded(InspectRequest request, UiTargetDto target, ResolvedWindow identity, + string method, string status, int? inputLength = null) => new() + { + SchemaVersion = "1.0", + ProtocolVersion = "1.0", + RequestId = request.RequestId, + Success = true, + Action = request.Action, + Status = status, + ActionMethod = method, + ActionTarget = target, + InputLength = inputLength, + Pid = identity.Pid, + Hwnd = identity.Hwnd, + }; + + private static InspectResponse Failed(InspectRequest request, string errorCode, string errorMessage, + ResolvedWindow identity, UiTargetDto? target = null, List? candidateWindows = null) => new() + { + SchemaVersion = "1.0", + ProtocolVersion = "1.0", + RequestId = request.RequestId, + Success = false, + Action = request.Action, + ErrorCode = errorCode, + ErrorMessage = errorMessage, + ActionTarget = target, + CandidateWindows = candidateWindows, + Pid = identity.Pid, + Hwnd = identity.Hwnd, + }; + + /// 无法解析窗口时仍未确定目标，故用请求值报告边界。 + private static InspectResponse Failed(InspectRequest request, string errorCode, string errorMessage, + List? candidateWindows = null) => + Failed(request, errorCode, errorMessage, + new ResolvedWindow(request.Pid, request.Hwnd ?? string.Empty), null, candidateWindows); + + private readonly record struct ResolvedWindow(int Pid, string Hwnd); +} diff --git a/tools/WinCode.UIA.Host/UiAudit.cs b/tools/WinCode.UIA.Host/UiAudit.cs index 3ef4dde..3734577 100644 --- a/tools/WinCode.UIA.Host/UiAudit.cs +++ b/tools/WinCode.UIA.Host/UiAudit.cs @@ -48,7 +48,9 @@ public static UiAudit Start(int pid, string? hwnd, string? capture, string opera var bytes = Measure(audit.directory); var start = Encode(new { v = 1, t = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(), id = audit.id, phase = "start", helper = Environment.ProcessId, target = pid, hwnd = CanonicalHandle(hwnd), - op = operation == "listWindows" ? "windows" : "inspect", + // 操作名必须如实记录：破坏性动作绝不能被记成 "inspect"。 + op = operation switch { "listWindows" => "windows", "click" => "click", "type" => "type", + "setValue" => "setValue", _ => "inspect" }, capture = capture is "original" or "annotated" ? capture : "none" }); var statePath = Path.Combine(audit.directory, StateName); int stateReserve = File.Exists(statePath) ? 0 : 32; diff --git a/tools/WinCode.UIA.Host/UiHostContracts.cs b/tools/WinCode.UIA.Host/UiHostContracts.cs index f1ccde7..086405b 100644 --- a/tools/WinCode.UIA.Host/UiHostContracts.cs +++ b/tools/WinCode.UIA.Host/UiHostContracts.cs @@ -35,13 +35,33 @@ public class InspectRequest public int? MaxWindows { get; set; } public string? SchemaVersion { get; set; } public string? RequestId { get; set; } - public string? Action { get; set; } // "inspect" | "health" | "ping" + /// "inspect" | "health" | "ping" | "listWindows" | "click" | "type" | "setValue" + public string? Action { get; set; } public int Pid { get; set; } public string? Hwnd { get; set; } public string? Capture { get; set; } // "none" | "original" | "annotated" public int? MaxDepth { get; set; } public int? MaxNodes { get; set; } public int? TimeoutMs { get; set; } + // 语义操作（action != inspect）的定位条件与输入；至少提供一个定位字段。 + public string? TargetAutomationId { get; set; } + public string? TargetName { get; set; } + public string? TargetControlType { get; set; } + public string? InputText { get; set; } + public bool ClearBefore { get; set; } +} + +/// 被操作控件的身份证据；不包含输入文本，避免在结果中回显敏感内容。 +public class UiTargetDto +{ + public List? PropertyIssues { get; set; } + public string? AutomationId { get; set; } + public string? Name { get; set; } + public string? ControlType { get; set; } + public string? ClassName { get; set; } + public RectDto? Bounds { get; set; } + public bool? IsEnabled { get; set; } + public bool? IsOffscreen { get; set; } } public sealed record HostBuildIdentity(string Version, string? InformationalVersion, string? Configuration, string Framework) @@ -56,7 +76,8 @@ public sealed record HostBuildIdentity(string Version, string? InformationalVers public class InspectResponse { public HostBuildIdentity HostIdentity { get; } = HostBuildIdentity.Current; - public int InspectionVersion { get; set; } = 2; + /// 取证结构版本：2 增加 query/readStates，3 增加语义操作（click/type/setValue）。 + public int InspectionVersion { get; set; } = 3; public long? HelperPeakWorkingSetBytes { get; set; } public QueryResultDto? QueryResult { get; set; } public bool? TreeComplete { get; set; } @@ -73,6 +94,11 @@ public class InspectResponse public bool Success { get; set; } public string? Action { get; set; } public string? Status { get; set; } + /// 实际使用的 UIA 模式或输入方式；只描述执行方式，不声明应用已做出反应。 + public string? ActionMethod { get; set; } + public UiTargetDto? ActionTarget { get; set; } + /// 接受的输入字符数；不回显输入文本。 + public int? InputLength { get; set; } public string? ErrorCode { get; set; } public string? ErrorMessage { get; set; } public int? Pid { get; set; } diff --git a/tools/WinCode.UIA.Host/WinCode.UIA.Host.csproj b/tools/WinCode.UIA.Host/WinCode.UIA.Host.csproj index 6e84c5f..a1fe0d1 100644 --- a/tools/WinCode.UIA.Host/WinCode.UIA.Host.csproj +++ b/tools/WinCode.UIA.Host/WinCode.UIA.Host.csproj @@ -1,7 +1,7 @@ - 0.15.0 + 0.16.0 true Exe net10.0-windows From ae1d984b2312c075c3c3eb362dd16a07682825c0 Mon Sep 17 00:00:00 2001 From: linnnn89 <216342082+linnnn89@users.noreply.github.com> Date: Sat, 26 Sep 2026 23:51:55 +0800 Subject: [PATCH 2/2] fix: expect the expanded tool count in Roslyn gateway acceptance The gateway acceptance script asserted 17 published tools and the architecture note carried the same stale count; the UI action tools make 19. --- ...\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" | 2 +- scripts/verify-roslyn-gateway.mjs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git "a/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" "b/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" index 3c2e85f..a0649f7 100644 --- "a/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" +++ "b/WinCode-\346\236\266\346\236\204\344\270\216\346\225\260\346\215\256\346\265\201\350\257\264\346\230\216.md" @@ -57,7 +57,7 @@ Codex 可选择 Skill 按需入口：首次使用才创建交互执行会话，` | 原生 Host | 按 PID/HWND 取证，执行有界 UIA 搜索及截图 | [Program.cs](tools/WinCode.UIA.Host/Program.cs)、[BoundedUiSearch](tools/WinCode.UIA.Host/BoundedUiSearch.cs)、[UiAudit](tools/WinCode.UIA.Host/UiAudit.cs) | | 构建交付层 | 锁定构建、回归、stdio 验证、产物身份、Skill 一致性 | [check.mjs](scripts/check.mjs)、[delivery-manifest](scripts/delivery-manifest.mjs)、[sync-skill](scripts/sync-skill.mjs) | -`ExtensionManager` 目前保留兼容接口，没有内置注册项，不承担实际插件生态或工具发现职责。Gateway 当前列出 17 个工具名称，其中包含影响分析别名；工具名称数量不等于独立业务能力数量。 +`ExtensionManager` 目前保留兼容接口，没有内置注册项，不承担实际插件生态或工具发现职责。Gateway 当前列出 19 个工具名称，其中包含影响分析别名；工具名称数量不等于独立业务能力数量。 ## 2. 一次请求怎样通过系统 diff --git a/scripts/verify-roslyn-gateway.mjs b/scripts/verify-roslyn-gateway.mjs index 9338ffe..246d891 100644 --- a/scripts/verify-roslyn-gateway.mjs +++ b/scripts/verify-roslyn-gateway.mjs @@ -129,7 +129,7 @@ try { assert.equal(initial.health.text.semanticConfigured, false); report.scenarios.push('explicit production CLI selects Roslyn; hello does not load a project'); const listed = await client.listTools(); - assert.equal(listed.tools.length, 17); + assert.equal(listed.tools.length, 19); assert.ok(listed.tools.find(tool => tool.name === 'wincode_find_references').inputSchema.properties.symbolLocation); report.scenarios.push('existing tools expose the validated optional symbolLocation contract'); const target = await integerTarget();