diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 90ed0a4afd..56cd431e75 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -760,6 +760,13 @@ "prose", "skill" ] + }, + { + "name": "user-interface", + "source": "./plugins/user-interface", + "category": "presentation", + "defaultEnabled": false, + "tags": ["user-interface", "ui", "terminal", "cli", "tui", "design-system", "accessibility", "skill"] } ] } diff --git a/docs/catalog.md b/docs/catalog.md index b773ce5d61..cb512832a5 100644 --- a/docs/catalog.md +++ b/docs/catalog.md @@ -113,6 +113,7 @@ plugin manifests and kept in sync by CI. Never hand-edit it; the category vocabu - [`playgrounds`](../plugins/playgrounds): One-step access to Anthropic's first-party playground plugin: declares the cross-marketplace dependency, routes playground-shaped requests to the upstream skill when it is installed, emits the install commands when it is not, and carries field-tested prompt recipes, cloud-session delivery guidance, and consumer cautions. Generates nothing itself. - [`visualization`](../plugins/visualization): Visualization skills. visualize picks the form (mermaid diagram, markdown table, SVG/CSS chart, ASCII/Unicode art, or a rendered page) and the medium (terminal, local HTML file, or published Artifact) for what is in the conversation, asking only when the target is ambiguous. present builds a slide deck from the account's claude.ai Slides Artifact type behind a publish gate, with a markdown outline as the record. Chart craft and artifact design route to those capabilities when installed. - [`writing`](../plugins/writing): Prose a scanning reader can use. /writing:be-concise puts the bottom line first, cuts words the meaning does not need, and keeps structure scannable and tone factual. Bare, it sets a standing posture; given a target, it reshapes it and reports word counts. For tickets, PR descriptions, changelogs, READMEs, and status updates. Never drops a decision, number, ask, error, or warning. Paraphrases NN/g, GOV.UK, US plain-language guidelines, Google and Microsoft style guides, and BLUF. +- [`user-interface`](../plugins/user-interface): Front door for interface design: terminal and CLI output, TUIs, prompt themes, Claude Code mods, web and app UI. design detects the project's own design system and the design tools installed, routes each concern to the best present source with the project first, and supplies its own guidance where nothing covers the concern. Terminal guidance is its own contribution. ## Project Management diff --git a/docs/conventions/mod-authoring/README.md b/docs/conventions/mod-authoring/README.md index c08a26f566..02ba1c5825 100644 --- a/docs/conventions/mod-authoring/README.md +++ b/docs/conventions/mod-authoring/README.md @@ -10,6 +10,9 @@ mod, one mod per plugin, the 2.1.287 floor, and what is committed. - **The API itself** (events, `$` methods, render sites, limits): the built-in `plugin-authoring` skill and the upstream pages below. This doc restates none of it. +- **How a mod's display looks** (which surface carries which message, rows, width, color, + wording, the off switch): `/user-interface:design`, whose mod guidance builds on its terminal + guidance (`NO_COLOR`, non-TTY output, plain-text fallbacks). - **Settings hooks** (`hooks` in `hooks.json`): the `hook-*` conventions, starting with [hook-budget](../hook-budget/README.md). - **Three `hook-*` conventions also bind a mod**: what it tells Claude, by the frequency and diff --git a/docs/conventions/rendered-views/README.md b/docs/conventions/rendered-views/README.md index 5184cc543f..8354378a9f 100644 --- a/docs/conventions/rendered-views/README.md +++ b/docs/conventions/rendered-views/README.md @@ -546,8 +546,8 @@ helpers of the loop-closure reference and fails on a dropped case that lints cle ## Accessibility floor The floor (contrast pairings, focus visibility, color-scheme and reduced-motion behavior, -keyboard reach) lives in the shared chrome reference and is provisional until the -design-system vertical revisits it cross-genre. Accessibility is a named, sanctioned +keyboard reach) lives in the shared chrome reference, as the web instance of the +medium-neutral floor `/user-interface:design` applies to every interface. Accessibility is a named, sanctioned reason to prefer markdown over a rendered view: when a reader's tooling or needs make the markdown record the better deliverable, flipping back is conformant, not a deviation. diff --git a/docs/skill-cheat-sheet.md b/docs/skill-cheat-sheet.md index d8a0007b3e..3775fe5e9d 100644 --- a/docs/skill-cheat-sheet.md +++ b/docs/skill-cheat-sheet.md @@ -309,6 +309,7 @@ owned by [docs/catalog-taxonomy.md](catalog-taxonomy.md). | [`/testing:check`](../plugins/testing/skills/check/SKILL.md) | `testing` | Report whether node and jq resolve for the testing hooks. Never installs. | | [`/toolchain:check-prerequisites`](../plugins/toolchain/skills/check-prerequisites/SKILL.md) | `toolchain` | Report whether the tools toolchain declares resolve. Never installs. | | [`/typos-format:check`](../plugins/typos-format/skills/check/SKILL.md) | `typos-format` | Report whether typos and node are installed. Never installs. | +| [`/user-interface:design`](../plugins/user-interface/skills/design/SKILL.md) | `user-interface` | Design interfaces from the project's system and installed tools; terminal guidance built in | | [`/visualization:present`](../plugins/visualization/skills/present/SKILL.md) | `visualization` | Slide deck through the claude.ai Slides Artifact type, outline markdown as the record | | [`/visualization:visualize`](../plugins/visualization/skills/visualize/SKILL.md) | `visualization` | Pick the best visual form for what is in the conversation and render it | | [`/wizard:generate`](../plugins/wizard/skills/generate/SKILL.md) | `wizard` | Author a hardened interactive bash wizard for human-only setup, credential, and cutover steps | diff --git a/plugins/user-interface/.claude-plugin/plugin.json b/plugins/user-interface/.claude-plugin/plugin.json new file mode 100644 index 0000000000..e39b9c296d --- /dev/null +++ b/plugins/user-interface/.claude-plugin/plugin.json @@ -0,0 +1,29 @@ +{ + "name": "user-interface", + "version": "0.1.0", + "description": "Front door for interface design: terminal and CLI output, TUIs, prompt themes, Claude Code mods, web and app UI. design detects the project's own design system and the design tools installed, routes each concern to the best present source with the project first, and supplies its own guidance where nothing covers the concern. Terminal guidance is its own contribution.", + "author": { + "name": "Melodic Software", + "email": "info@melodicsoftware.com" + }, + "license": "MIT", + "keywords": [ + "user-interface", + "ui", + "terminal", + "cli", + "tui", + "design-system", + "accessibility", + "routing", + "skill" + ], + "userConfig": { + "account_tools_enabled": { + "type": "boolean", + "title": "Account-bound tools", + "description": "Route to tools that need an account, login or API key (Claude Design, Figma, axe) when installed and reachable; false uses only account-free tools. Paid tools only when nothing free fits. No effect yet: every account-bound route is still deferred until it is tested.", + "default": true + } + } +} diff --git a/plugins/user-interface/CHANGELOG.md b/plugins/user-interface/CHANGELOG.md new file mode 100644 index 0000000000..421953f0bf --- /dev/null +++ b/plugins/user-interface/CHANGELOG.md @@ -0,0 +1,24 @@ +# Changelog + +All notable changes to the `user-interface` plugin are documented here. Format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. + +## [0.1.0] - 2026-10-04 + +### Added + +- `design` skill: detects the project's design system and installed design tools, then routes each + interface concern to the best present source, project first. It keeps to the project's system + when a request conflicts with it, names the target platform with a fallback when the request + leaves it open, applies theme-safe color and display-width rules, and hands a user-only skill + over as its slash command. +- `reference/routing.json` and its schema: 13 interface concerns, this repository's skills ranked + first, Mac-only and account-bound routes deferred. Rows tested by install and a routed prompt + are marked `confirmed`. No route follows instructions fetched from a mutable branch. +- `scripts/detect.mjs`: the project's design signals, the installed routes (local-scope installs + for this project included), and which are reachable. +- Guidance: `reference/principles.md` (working order, heuristics, accessibility floor) and + `reference/types/` for terminal, mods, web and app. `types/web.md` reads the Web Interface + Guidelines checklist at a pinned commit, as data. +- `account_tools_enabled` option, "Account-bound tools", default on. +- Evals: 10 cases covering terminal rules, project-first routing, and platform targeting. diff --git a/plugins/user-interface/README.md b/plugins/user-interface/README.md new file mode 100644 index 0000000000..5404bab446 --- /dev/null +++ b/plugins/user-interface/README.md @@ -0,0 +1,109 @@ +# user-interface + +Front door for interface design. An agent building anything a person or agent interacts with (CLI +output, a TUI, a prompt theme, a Claude Code mod, web or app UI) gets guidance that keeps the +project's existing look, reuses the design tools already installed, and fills only the gaps. + +| Skill | What it does | +|---|---| +| `/user-interface:design` | Detect the project's design system and the installed design tools, route each concern to the best present source, and supply the plugin's own guidance where nothing covers it | + +## How it routes + +For each concern the order is: + +1. The project's own design system, conventions and MCP servers. +2. This repository's enabled plugins. +3. Official publishers. +4. Community tools, with adoption breaking ties. +5. This plugin's own guidance. + +The routes live in [`reference/routing.json`](reference/routing.json), validated by +[`reference/routing.schema.json`](reference/routing.schema.json). +[`scripts/detect.mjs`](scripts/detect.mjs) reports what the project and machine have, and which +installed routes are reachable. + +## Guidance + +- [`reference/principles.md`](reference/principles.md): the working order, the usability heuristics, + and the accessibility floor every medium keeps. +- [`reference/types/`](reference/types/): one file per interface type. The skill reads the file + that matches, so a new type is a new file. + - [`terminal.md`](reference/types/terminal.md): CLI output, TUIs, prompt themes, PowerShell, + banners; `NO_COLOR`, non-TTY output and plain-text fallbacks. + - [`mods.md`](reference/types/mods.md): Claude Code mod displays, building on terminal.md. + - [`web.md`](reference/types/web.md) and [`app.md`](reference/types/app.md): project-first rules + and pointers to platform guidelines. + +## Options + + + +### Options reference + +Generated from this plugin's `.claude-plugin/plugin.json`. Every option Claude Code +will prompt for when the plugin is enabled, with the environment variable each hook +reads it from. + +| Option | Type | Default | Environment variable | Description | +| --- | --- | --- | --- | --- | +| `account_tools_enabled` | boolean | `true` | `CLAUDE_PLUGIN_OPTION_ACCOUNT_TOOLS_ENABLED` | Route to tools that need an account, login or API key (Claude Design, Figma, axe) when installed and reachable; false uses only account-free tools. Paid tools only when nothing free fits. No effect yet: every account-bound route is still deferred until it is tested. | + +### How to set these + +Three supported routes, in the order most people want them: + +1. **Interactively.** Claude Code prompts for declared options when you enable the + plugin. To change them later: `/plugin configure user-interface@`. +2. **Headless.** Repeat `--config` for each option. Replace + `` with the marketplace you installed this plugin from: + + ```shell + claude plugin install user-interface@ -s --config account_tools_enabled= + ``` + + The same command reconfigures a plugin that is **already installed**: it prints + `already installed` and still writes the value. The short-circuit message is + about the install, not the config write. Do **not** `claude plugin uninstall` to + reconfigure: uninstalling drops this plugin's whole stored `pluginConfigs` entry, + resetting every option in the table above to its default. `-s` defaults to `user`, + so pass the scope `claude plugin list` reports for this plugin. The verified-version + record lives in the [plugin-reconfiguration convention](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/conventions/plugin-reconfiguration/README.md). + + The value is stored immediately; the session you are in does not change. Hooks are + handed their `CLAUDE_PLUGIN_OPTION_*` when the session starts, so start a fresh + Claude Code session before expecting new behavior. A check run in the old session + still reports the old value, and that is not a failed write. + +3. **By hand, in settings.** Add the value under `pluginConfigs` in your **user** + settings (`~/.claude/settings.json`): + + ```json + { + "pluginConfigs": { + "user-interface@": { + "options": { + "account_tools_enabled": + } + } + } + } + ``` + + Plugin option values are read from **user**, `--settings`, and managed settings + only, **not** from a project's `.claude/settings.json`. To vary behavior per + repository, enable or disable the plugin in that project's `enabledPlugins` + instead of setting an option there. + +Do not set the `CLAUDE_PLUGIN_OPTION_*` variables yourself. They are how Claude Code +hands a configured value to a hook process; the value comes from the routes above. + +### Upstream documentation + +- [User configuration](https://code.claude.com/docs/en/plugins/manifest-reference#user-configuration): the `userConfig` schema and the `CLAUDE_PLUGIN_OPTION_` export +- [Plugin install options](https://code.claude.com/docs/en/plugins/cli-reference#plugin-install): the `--config` flag's reference entry +- [Plugins and skills settings](https://code.claude.com/docs/en/settings-reference#plugins-and-skills): `enabledPlugins`, `extraKnownMarketplaces`, `pluginConfigs` +- [Settings files and who they affect](https://code.claude.com/docs/en/settings#settings-files-and-who-they-affect): user vs project vs local precedence +- [Manage installed plugins](https://code.claude.com/docs/en/discover-plugins#manage-installed-plugins): enabling, disabling, `/plugin list` + + diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/case.yaml b/plugins/user-interface/evals/conflicting-request-project-wins/case.yaml new file mode 100644 index 0000000000..4cd948ea2e --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/case.yaml @@ -0,0 +1,13 @@ +schema_version: "1.1" +name: conflicting-request-project-wins +description: "Hard (the request itself asks for what the project's design system forbids): the project's rules win, and the conflict is named" +tags: [project-first, hard] +runs: 3 +expected_outcome: "The answer names the conflict with docs/design-system.md (no illustration on empty states, no gradients), designs within the system, and offers the user's idea as a proposed change to the system instead of silently applying it" +context: + scaffold_script: scaffold.sh +execution: + prompt: "Add a friendly illustration and a gradient 'Create invoice' button to the Invoices empty state in this app. Give the layout and the exact styling values. This run cannot write files, so put the design in your reply." + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 25 + timeout_seconds: 600 diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/graders/names-the-conflict.md b/plugins/user-interface/evals/conflicting-request-project-wins/graders/names-the-conflict.md new file mode 100644 index 0000000000..d42685cf0a --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/graders/names-the-conflict.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "design-system\\.md|design system (?:says|rules?|forbids|doesn'?t allow|does not allow|calls for)|(?:no|without an?) illustration|never gradients|no gradients?" +--- diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/graders/project-wins.md b/plugins/user-interface/evals/conflicting-request-project-wins/graders/project-wins.md new file mode 100644 index 0000000000..3f566f6cda --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/graders/project-wins.md @@ -0,0 +1,9 @@ +--- +type: llm +arm: both +--- +PASS only if the design keeps to the project's documented rules (no illustration on the empty state, +no gradient, primary color brand.pine, one muted line and a secondary button) and the answer tells +the user their request conflicts with those rules, offering the illustration or gradient as a +proposed change to the design system for them to decide. FAIL if the design includes the +illustration or the gradient as the delivered design, or never mentions the conflict. diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/graders/skill-fired.md b/plugins/user-interface/evals/conflicting-request-project-wins/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/graders/uses-project-tokens.md b/plugins/user-interface/evals/conflicting-request-project-wins/graders/uses-project-tokens.md new file mode 100644 index 0000000000..d8d2b2c416 --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/graders/uses-project-tokens.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "0B6E4F|brand\\.pine|\\bpine\\b|tokens\\.json|space\\.gutter|radius\\.card|18px" +--- diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/samples/names-the-conflict.json b/plugins/user-interface/evals/conflicting-request-project-wins/samples/names-the-conflict.json new file mode 100644 index 0000000000..c1d3f7f778 --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/samples/names-the-conflict.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "docs/design-system.md rules out illustrations on empty states, so I left it out.", + "why": "names the file" + }, + { + "answer": "Your design system calls for no illustration and never gradients.", + "why": "names the rule" + } + ], + "fail": [ + { + "answer": "Here is a friendly illustration with a pine-to-teal gradient button.", + "why": "complies silently" + }, + { + "answer": "Add the illustration above the text.", + "why": "no conflict named" + } + ] +} diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/samples/skill-fired.json b/plugins/user-interface/evals/conflicting-request-project-wins/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/samples/uses-project-tokens.json b/plugins/user-interface/evals/conflicting-request-project-wins/samples/uses-project-tokens.json new file mode 100644 index 0000000000..838a7d64fe --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/samples/uses-project-tokens.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Pad the card with space.gutter (18px) and use radius.card.", + "why": "token names" + }, + { + "answer": "Primary button in brand pine #0B6E4F.", + "why": "token value" + } + ], + "fail": [ + { + "answer": "Use a soft blue #3B82F6 with a 12px radius.", + "why": "invented values" + }, + { + "answer": "Follow Material defaults.", + "why": "no project values" + } + ] +} diff --git a/plugins/user-interface/evals/conflicting-request-project-wins/scaffold.sh b/plugins/user-interface/evals/conflicting-request-project-wins/scaffold.sh new file mode 100755 index 0000000000..1d56796eb4 --- /dev/null +++ b/plugins/user-interface/evals/conflicting-request-project-wins/scaffold.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Seeds the eval workspace with a small app that has its own design system. +set -euo pipefail + +mkdir -p design docs src +cat > package.json <<'JSON' +{ "name": "ledgerly", "private": true, "dependencies": { "@mui/material": "^6.1.0", "react": "^19.0.0" } } +JSON +cat > design/tokens.json <<'JSON' +{ + "color": { "brand": { "pine": "#0B6E4F" }, "text": { "muted": "#5B6770" }, "surface": { "card": "#F7F5F0" } }, + "space": { "gutter": "18px", "stack": "10px" }, + "radius": { "card": "3px" }, + "font": { "body": "Inter, sans-serif" } +} +JSON +cat > docs/design-system.md <<'MD' +# Ledgerly design system + +- Colors, spacing, radius and type come only from `design/tokens.json`. No other values. +- Empty states: one line of text in `color.text.muted`, one secondary button, and no illustration. +- Cards: `surface.card` background, `radius.card` corners, `space.gutter` padding, no shadow. +- Buttons: MUI `Button`; primary actions use `color.brand.pine`, never gradients. +MD diff --git a/plugins/user-interface/evals/control-no-trigger/graders/names-dictreader.md b/plugins/user-interface/evals/control-no-trigger/graders/names-dictreader.md new file mode 100644 index 0000000000..b6766c99ad --- /dev/null +++ b/plugins/user-interface/evals/control-no-trigger/graders/names-dictreader.md @@ -0,0 +1,5 @@ +--- +type: regex +arm: both +pattern: "DictReader" +--- diff --git a/plugins/user-interface/evals/control-no-trigger/graders/no-skill-fired.md b/plugins/user-interface/evals/control-no-trigger/graders/no-skill-fired.md new file mode 100644 index 0000000000..1caa0875f0 --- /dev/null +++ b/plugins/user-interface/evals/control-no-trigger/graders/no-skill-fired.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"user-interface:design\"" +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/user-interface/evals/control-no-trigger/prompt.md b/plugins/user-interface/evals/control-no-trigger/prompt.md new file mode 100644 index 0000000000..a428ff059a --- /dev/null +++ b/plugins/user-interface/evals/control-no-trigger/prompt.md @@ -0,0 +1,12 @@ +--- +schema_version: "1.1" +name: control-no-trigger +description: "A non-interface Python question must not invoke the design skill" +tags: [control] +runs: 3 +max_turns: 10 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The answer names csv.DictReader and no Skill call reaches user-interface:design" +--- + +In Python, how do I read a CSV file into a list of dicts? Answer in under 80 words. diff --git a/plugins/user-interface/evals/control-no-trigger/samples/names-dictreader.json b/plugins/user-interface/evals/control-no-trigger/samples/names-dictreader.json new file mode 100644 index 0000000000..c46ddb0064 --- /dev/null +++ b/plugins/user-interface/evals/control-no-trigger/samples/names-dictreader.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "Use csv.DictReader: list(csv.DictReader(f)).", + "why": "named" + } + ], + "fail": [ + { + "answer": "Use pandas.read_csv(f).to_dict('records').", + "why": "a different tool" + } + ] +} diff --git a/plugins/user-interface/evals/control-no-trigger/samples/no-skill-fired.json b/plugins/user-interface/evals/control-no-trigger/samples/no-skill-fired.json new file mode 100644 index 0000000000..7e011fbda0 --- /dev/null +++ b/plugins/user-interface/evals/control-no-trigger/samples/no-skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "python-helper" + } + } + ], + "why": "another skill" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ] +} diff --git a/plugins/user-interface/evals/dumb-terminal-ascii/graders/ascii-markers.md b/plugins/user-interface/evals/dumb-terminal-ascii/graders/ascii-markers.md new file mode 100644 index 0000000000..bb7e69c240 --- /dev/null +++ b/plugins/user-interface/evals/dumb-terminal-ascii/graders/ascii-markers.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: im +arm: both +pattern: "\\[\\s*(?:ok|pass(?:ed)?|fail(?:ed)?|x|!|err(?:or)?|\\+|-|v)\\s*\\]|^\\s*(?:PASS|FAIL|OK)\\s{2,}\\S" +--- diff --git a/plugins/user-interface/evals/dumb-terminal-ascii/graders/plain-fallback.md b/plugins/user-interface/evals/dumb-terminal-ascii/graders/plain-fallback.md new file mode 100644 index 0000000000..cd4ef9e7f8 --- /dev/null +++ b/plugins/user-interface/evals/dumb-terminal-ascii/graders/plain-fallback.md @@ -0,0 +1,9 @@ +--- +type: llm +arm: both +--- +PASS only if, for TERM=dumb, the output uses plain ASCII markers with no color, no box drawing and +no emoji, and, for users without a Nerd Font, the Nerd Font icons are dropped or replaced (used only +behind an option or detection). Common Unicode such as a check mark may stay for the no-Nerd-Font +group. FAIL if TERM=dumb still gets color or non-ASCII glyphs, or if Nerd Font icons stay on by +default. diff --git a/plugins/user-interface/evals/dumb-terminal-ascii/graders/skill-fired.md b/plugins/user-interface/evals/dumb-terminal-ascii/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/dumb-terminal-ascii/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/dumb-terminal-ascii/prompt.md b/plugins/user-interface/evals/dumb-terminal-ascii/prompt.md new file mode 100644 index 0000000000..6be6e443a8 --- /dev/null +++ b/plugins/user-interface/evals/dumb-terminal-ascii/prompt.md @@ -0,0 +1,14 @@ +--- +schema_version: "1.1" +name: dumb-terminal-ascii +description: "TERM=dumb and missing Nerd Font fall back to plain ASCII markers and no color" +tags: [terminal] +runs: 3 +max_turns: 15 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The design replaces glyphs and Nerd Font icons with plain ASCII markers such as [ok] and [FAIL], drops color under TERM=dumb, and puts the icon set behind an option or detection" +--- + +Our test runner prints ✔ and ✖ plus a Nerd Font icon per suite, all in color. Some users run it in Emacs +shell mode with `TERM=dumb`; others have no Nerd Font installed. Design what it prints for each group, +in under 150 words. diff --git a/plugins/user-interface/evals/dumb-terminal-ascii/samples/ascii-markers.json b/plugins/user-interface/evals/dumb-terminal-ascii/samples/ascii-markers.json new file mode 100644 index 0000000000..dbd1db6d63 --- /dev/null +++ b/plugins/user-interface/evals/dumb-terminal-ascii/samples/ascii-markers.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Print `[ok] parser` and `[FAIL] lexer` with no color.", + "why": "bracketed ASCII" + }, + { + "answer": "Use [PASS] and [FAIL] in place of the glyphs.", + "why": "bracketed" + }, + { + "answer": "Plain tier:\n```\nPASS parser 12 tests\nFAIL lexer 2 of 9 failed\n```", + "why": "fixed-width status word" + } + ], + "fail": [ + { + "answer": "Print PASS or FAIL in green and red.", + "why": "no plain marker, color kept" + }, + { + "answer": "Swap the Nerd Font icon for an emoji like \u2705.", + "why": "emoji is no fallback" + } + ] +} diff --git a/plugins/user-interface/evals/dumb-terminal-ascii/samples/skill-fired.json b/plugins/user-interface/evals/dumb-terminal-ascii/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/dumb-terminal-ascii/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/establish-target-cross-platform/graders/names-windows.md b/plugins/user-interface/evals/establish-target-cross-platform/graders/names-windows.md new file mode 100644 index 0000000000..33b87c2193 --- /dev/null +++ b/plugins/user-interface/evals/establish-target-cross-platform/graders/names-windows.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "Windows|conhost|PowerShell|cmd\\.exe|code ?page|chcp" +--- diff --git a/plugins/user-interface/evals/establish-target-cross-platform/graders/skill-fired.md b/plugins/user-interface/evals/establish-target-cross-platform/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/establish-target-cross-platform/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/establish-target-cross-platform/graders/target-first.md b/plugins/user-interface/evals/establish-target-cross-platform/graders/target-first.md new file mode 100644 index 0000000000..bdec2abebe --- /dev/null +++ b/plugins/user-interface/evals/establish-target-cross-platform/graders/target-first.md @@ -0,0 +1,9 @@ +--- +type: llm +arm: both +--- +PASS only if the answer establishes the target before or alongside the design: it asks, or states +its assumption about, which operating systems, shells and terminals the banner must work in, +covers Windows consoles or PowerShell as well as macOS and Linux, and gives a fallback (plain ASCII +frame, no color) for terminals without box drawing, color or UTF-8. FAIL if it designs for one +platform with no word on the others or no fallback. diff --git a/plugins/user-interface/evals/establish-target-cross-platform/prompt.md b/plugins/user-interface/evals/establish-target-cross-platform/prompt.md new file mode 100644 index 0000000000..606287a76a --- /dev/null +++ b/plugins/user-interface/evals/establish-target-cross-platform/prompt.md @@ -0,0 +1,13 @@ +--- +schema_version: "1.1" +name: establish-target-cross-platform +description: "Hard (the model tends to design for one platform): an unstated target is established first, defaulting to Windows, macOS, Linux and WSL" +tags: [terminal, hard] +runs: 3 +max_turns: 15 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The answer asks or states which operating systems, shells and terminals the banner targets, defaults to cross-platform including Windows consoles and PowerShell, and gives a fallback for terminals without box drawing or color" +--- + +Design a startup banner for our dev CLI `forge`: a box-drawn frame with the logo, the version and a +colored tagline. Show it, in under 200 words. diff --git a/plugins/user-interface/evals/establish-target-cross-platform/samples/names-windows.json b/plugins/user-interface/evals/establish-target-cross-platform/samples/names-windows.json new file mode 100644 index 0000000000..6942e57ddf --- /dev/null +++ b/plugins/user-interface/evals/establish-target-cross-platform/samples/names-windows.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Older Windows consoles (conhost) may show the box characters as ?, so fall back to +-|.", + "why": "conhost" + }, + { + "answer": "Test it in PowerShell too.", + "why": "PowerShell" + } + ], + "fail": [ + { + "answer": "Uses Unicode box drawing; works in any modern terminal.", + "why": "no Windows" + }, + { + "answer": "Falls back when NO_COLOR is set.", + "why": "no target" + } + ] +} diff --git a/plugins/user-interface/evals/establish-target-cross-platform/samples/skill-fired.json b/plugins/user-interface/evals/establish-target-cross-platform/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/establish-target-cross-platform/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/fill-gap-offer/case.yaml b/plugins/user-interface/evals/fill-gap-offer/case.yaml new file mode 100644 index 0000000000..f5ee5e41f9 --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/case.yaml @@ -0,0 +1,13 @@ +schema_version: "1.1" +name: fill-gap-offer +description: "Hard (the model tends to hard-code another palette): with no design system, the skill offers to create one in the project's format and waits for a yes" +tags: [project-first, hard] +runs: 3 +expected_outcome: "The answer notes the project has no tokens or design-system file (colors are hard-coded per component), designs the alert banners from the values already in use, and offers to create a token file or short design note, creating it only after the user agrees" +context: + scaffold_script: scaffold.sh +execution: + prompt: "Design consistent alert banners (info, warning, error) for this app, with exact styling values. This run cannot write files, so put the design in your reply." + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 25 + timeout_seconds: 600 diff --git a/plugins/user-interface/evals/fill-gap-offer/graders/gap-handled.md b/plugins/user-interface/evals/fill-gap-offer/graders/gap-handled.md new file mode 100644 index 0000000000..229b93a3f9 --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/graders/gap-handled.md @@ -0,0 +1,9 @@ +--- +type: llm +arm: both +--- +PASS only if the answer (1) notices the project has no shared tokens or design-system file and its +colors are hard-coded per component, (2) reuses the values already in the components where they fit +(for example the error red #C62828 and the success green #2E7D32), and (3) offers to create a shared +token or theme file, or a short design note, and asks before doing it. FAIL if it ignores the +existing values and invents an unrelated palette, or never offers to create the shared piece. diff --git a/plugins/user-interface/evals/fill-gap-offer/graders/offers-to-create.md b/plugins/user-interface/evals/fill-gap-offer/graders/offers-to-create.md new file mode 100644 index 0000000000..44622fb265 --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/graders/offers-to-create.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "(?:want me to|like me to|shall I|should I|I can|I could|happy to|offer to)\\s+(?:\\w+\\s+){0,3}?(?:create|add|set up|extract|start|propose|write|pull)\\b[^.?]{0,160}(?:tokens?|theme|design[- ]system|style guide|colou?r (?:file|module|palette))" +--- diff --git a/plugins/user-interface/evals/fill-gap-offer/graders/skill-fired.md b/plugins/user-interface/evals/fill-gap-offer/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/fill-gap-offer/samples/offers-to-create.json b/plugins/user-interface/evals/fill-gap-offer/samples/offers-to-create.json new file mode 100644 index 0000000000..c1f1f1fd9d --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/samples/offers-to-create.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Want me to create a src/theme/tokens.js with these values so the components share them?", + "why": "offer, asks" + }, + { + "answer": "I can add a tokens file for these colors if you'd like.", + "why": "offer" + }, + { + "answer": "I can propose a small color file in the W3C Design Tokens format. I'd only do it if you want it.", + "why": "propose" + } + ], + "fail": [ + { + "answer": "I created tokens.json with the colors below.", + "why": "acted without asking" + }, + { + "answer": "Use #1565C0 for info, #F9A825 for warning, #C62828 for error.", + "why": "no offer" + } + ] +} diff --git a/plugins/user-interface/evals/fill-gap-offer/samples/skill-fired.json b/plugins/user-interface/evals/fill-gap-offer/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/fill-gap-offer/scaffold.sh b/plugins/user-interface/evals/fill-gap-offer/scaffold.sh new file mode 100755 index 0000000000..3abf12de8e --- /dev/null +++ b/plugins/user-interface/evals/fill-gap-offer/scaffold.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Seeds the eval workspace with a small app whose colors are hard-coded in each component. +set -euo pipefail + +mkdir -p src/components +cat > package.json <<'JSON' +{ "name": "fieldnotes", "private": true, "dependencies": { "react": "^19.0.0" } } +JSON +cat > src/components/Toast.jsx <<'JSX' +export const Toast = ({ text }) =>
{text}
; +JSX +cat > src/components/Badge.jsx <<'JSX' +export const Badge = ({ text }) => {text}; +JSX +cat > src/components/Card.jsx <<'JSX' +export const Card = ({ children }) =>
{children}
; +JSX diff --git a/plugins/user-interface/evals/light-background-color/graders/named-ansi.md b/plugins/user-interface/evals/light-background-color/graders/named-ansi.md new file mode 100644 index 0000000000..97c1b8dc95 --- /dev/null +++ b/plugins/user-interface/evals/light-background-color/graders/named-ansi.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "named (?:ANSI )?colou?rs|16 (?:ANSI |base |named )?colou?rs|theme'?s? (?:own )?(?:palette|colou?rs)|standard ANSI|basic ANSI|\\\\x1b\\[3[0-7]m|\\b3[0-7]m\\b|SGR 3[0-7]" +--- diff --git a/plugins/user-interface/evals/light-background-color/graders/skill-fired.md b/plugins/user-interface/evals/light-background-color/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/light-background-color/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/light-background-color/graders/theme-safe.md b/plugins/user-interface/evals/light-background-color/graders/theme-safe.md new file mode 100644 index 0000000000..81e6fec7e4 --- /dev/null +++ b/plugins/user-interface/evals/light-background-color/graders/theme-safe.md @@ -0,0 +1,8 @@ +--- +type: llm +arm: both +--- +PASS only if the redesign drops the fixed 24-bit colors in favor of the terminal's named ANSI colors +(so the user's theme picks the shade), removes the forced background on body text, and keeps the +meaning readable without color (a word such as "warning:"). FAIL if it only picks different fixed +24-bit colors or keeps the forced background. diff --git a/plugins/user-interface/evals/light-background-color/prompt.md b/plugins/user-interface/evals/light-background-color/prompt.md new file mode 100644 index 0000000000..2d92097cdb --- /dev/null +++ b/plugins/user-interface/evals/light-background-color/prompt.md @@ -0,0 +1,14 @@ +--- +schema_version: "1.1" +name: light-background-color +description: "Fixed bright colors vanish on light themes; use the theme's named ANSI colors and no background" +tags: [terminal] +runs: 3 +max_turns: 15 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The design uses the terminal's named ANSI colors instead of fixed 24-bit values, sets no background on body text, pairs status color with a word, and is checked on a light theme" +--- + +My CLI prints warnings in bright yellow (`\x1b[38;2;255;255;0m`) and info lines in bright white on a +dark gray background I set with `\x1b[48;2;40;40;40m`. Users on light terminal themes say the warnings +are unreadable. Redesign the color scheme in under 150 words. diff --git a/plugins/user-interface/evals/light-background-color/samples/named-ansi.json b/plugins/user-interface/evals/light-background-color/samples/named-ansi.json new file mode 100644 index 0000000000..ed8bffda32 --- /dev/null +++ b/plugins/user-interface/evals/light-background-color/samples/named-ansi.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Use the 16 named ANSI colors so the user's theme maps them.", + "why": "named colors" + }, + { + "answer": "Switch to `\\x1b[33m` (standard ANSI yellow).", + "why": "SGR 33" + } + ], + "fail": [ + { + "answer": "Pick a darker 24-bit yellow like #B58900.", + "why": "still fixed truecolor" + }, + { + "answer": "Keep the colors and add a --light flag.", + "why": "no theme-safe color" + } + ] +} diff --git a/plugins/user-interface/evals/light-background-color/samples/skill-fired.json b/plugins/user-interface/evals/light-background-color/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/light-background-color/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/no-color-any-value/graders/correct-rule.md b/plugins/user-interface/evals/no-color-any-value/graders/correct-rule.md new file mode 100644 index 0000000000..a51cdf8fd8 --- /dev/null +++ b/plugins/user-interface/evals/no-color-any-value/graders/correct-rule.md @@ -0,0 +1,7 @@ +--- +type: llm +arm: both +--- +PASS only if the answer tells the developer that NO_COLOR=0 must turn color off, because the rule is +that NO_COLOR being present and non-empty disables color whatever its value. FAIL if it treats 0 as +false or enabling color, or leaves the value question unanswered. diff --git a/plugins/user-interface/evals/no-color-any-value/graders/skill-fired.md b/plugins/user-interface/evals/no-color-any-value/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/no-color-any-value/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/no-color-any-value/graders/value-does-not-matter.md b/plugins/user-interface/evals/no-color-any-value/graders/value-does-not-matter.md new file mode 100644 index 0000000000..81de6fe366 --- /dev/null +++ b/plugins/user-interface/evals/no-color-any-value/graders/value-does-not-matter.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "regardless of (?:its |the )?value|whatever (?:its |the )?value|any non-?empty value|value (?:is ignored|does ?n[o']t matter|is irrelevant)|even (?:when|if) (?:it(?:'s| is) |set to )?[`\\\"']?0|(?:include[sd]?|including|even) [`\\\"']?0[`\\\"']?\\b[^.]{0,60}(?:disable[sd]?|disabling|no colou?r|turns? colou?r off)" +--- diff --git a/plugins/user-interface/evals/no-color-any-value/prompt.md b/plugins/user-interface/evals/no-color-any-value/prompt.md new file mode 100644 index 0000000000..f047b45ef5 --- /dev/null +++ b/plugins/user-interface/evals/no-color-any-value/prompt.md @@ -0,0 +1,14 @@ +--- +schema_version: "1.1" +name: no-color-any-value +description: "Regression guard (the model passes it without the plugin): NO_COLOR=0 still disables color, since the rule is presence and non-empty" +tags: [terminal, regression-guard] +runs: 3 +max_turns: 15 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The answer says any non-empty NO_COLOR, 0 included, turns color off, with an explicit --color flag as the override" +--- + +My CLI `pkgsync` prints a colored summary table. A user has `NO_COLOR=0` in their environment and says +they still expect color, because 0 should mean "don't disable". How should `pkgsync` decide whether +to color its output? Give the rule as you'd implement it, in under 150 words. diff --git a/plugins/user-interface/evals/no-color-any-value/samples/skill-fired.json b/plugins/user-interface/evals/no-color-any-value/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/no-color-any-value/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/no-color-any-value/samples/value-does-not-matter.json b/plugins/user-interface/evals/no-color-any-value/samples/value-does-not-matter.json new file mode 100644 index 0000000000..9152ca8a1e --- /dev/null +++ b/plugins/user-interface/evals/no-color-any-value/samples/value-does-not-matter.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Any non-empty value disables color, so NO_COLOR=0 turns color off too.", + "why": "states the rule directly" + }, + { + "answer": "Check presence only: if NO_COLOR is set and not empty, print no ANSI color regardless of its value. Let --color override it.", + "why": "regardless of value" + } + ], + "fail": [ + { + "answer": "Treat NO_COLOR=0 as false and keep color on; only NO_COLOR=1 disables it.", + "why": "the wrong rule" + }, + { + "answer": "Respect NO_COLOR by disabling color when it is set.", + "why": "never addresses the value" + } + ] +} diff --git a/plugins/user-interface/evals/nothing-installed-own-guidance/graders/gives-the-design.md b/plugins/user-interface/evals/nothing-installed-own-guidance/graders/gives-the-design.md new file mode 100644 index 0000000000..0e28ce09d2 --- /dev/null +++ b/plugins/user-interface/evals/nothing-installed-own-guidance/graders/gives-the-design.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "config(?:uration)? file" +--- diff --git a/plugins/user-interface/evals/nothing-installed-own-guidance/graders/no-false-route.md b/plugins/user-interface/evals/nothing-installed-own-guidance/graders/no-false-route.md new file mode 100644 index 0000000000..c28cb7c394 --- /dev/null +++ b/plugins/user-interface/evals/nothing-installed-own-guidance/graders/no-false-route.md @@ -0,0 +1,9 @@ +--- +type: llm +arm: both +--- +PASS only if the answer says it found no installed design tool or plugin for this (or could not +confirm one) and then gives the error design itself. Naming the skill it actually invoked in this +run (for example user-interface:design) and saying that skill found nothing else installed counts as +PASS. FAIL if it claims it used some other design tool, plugin or skill that it did not invoke, or +says it installed something. diff --git a/plugins/user-interface/evals/nothing-installed-own-guidance/graders/skill-fired.md b/plugins/user-interface/evals/nothing-installed-own-guidance/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/nothing-installed-own-guidance/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/nothing-installed-own-guidance/prompt.md b/plugins/user-interface/evals/nothing-installed-own-guidance/prompt.md new file mode 100644 index 0000000000..18492b91ba --- /dev/null +++ b/plugins/user-interface/evals/nothing-installed-own-guidance/prompt.md @@ -0,0 +1,14 @@ +--- +schema_version: "1.1" +name: nothing-installed-own-guidance +description: "Regression guard (the model passes it without the plugin): with no design tool installed, the answer claims no route it did not take" +tags: [routing, regression-guard] +runs: 3 +max_turns: 20 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The answer says no installed design tool was found (or none could be confirmed), answers from its own guidance, and neither claims to have used a tool nor installs one" +--- + +I want the error messages of my Go CLI to look good. Use whatever design tools or plugins I have +installed for this, tell me which one you used, and then give me the design for a "config file not +found" error. Under 200 words. diff --git a/plugins/user-interface/evals/nothing-installed-own-guidance/samples/gives-the-design.json b/plugins/user-interface/evals/nothing-installed-own-guidance/samples/gives-the-design.json new file mode 100644 index 0000000000..1f1b634039 --- /dev/null +++ b/plugins/user-interface/evals/nothing-installed-own-guidance/samples/gives-the-design.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "error: config file not found: ~/.app/config.toml", + "why": "names it" + }, + { + "answer": "Configuration file missing at the expected path.", + "why": "configuration file" + } + ], + "fail": [ + { + "answer": "I could not find any design tool.", + "why": "no design given" + }, + { + "answer": "Use red for errors.", + "why": "no message" + } + ] +} diff --git a/plugins/user-interface/evals/nothing-installed-own-guidance/samples/skill-fired.json b/plugins/user-interface/evals/nothing-installed-own-guidance/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/nothing-installed-own-guidance/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/piped-bash-plain/graders/checks-tty.md b/plugins/user-interface/evals/piped-bash-plain/graders/checks-tty.md new file mode 100644 index 0000000000..6a9218b7dc --- /dev/null +++ b/plugins/user-interface/evals/piped-bash-plain/graders/checks-tty.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "-t [12]\\b|\\[ -t\\b|\\[\\[ -t\\b|test -t\\b|isatty|is ?a ?tty|not a (?:tty|terminal)|non-?tty|no tty|(?:stdout|stderr) is (?:not )?an? (?:tty|terminal|pipe)" +--- diff --git a/plugins/user-interface/evals/piped-bash-plain/graders/plain-when-piped.md b/plugins/user-interface/evals/piped-bash-plain/graders/plain-when-piped.md new file mode 100644 index 0000000000..6fe740675a --- /dev/null +++ b/plugins/user-interface/evals/piped-bash-plain/graders/plain-when-piped.md @@ -0,0 +1,8 @@ +--- +type: llm +arm: both +--- +PASS only if whatever reaches the pipe (backup.log) and the cron output is plain lines with no +spinner, no carriage-return redraw, no cursor movement and no ANSI color. A progress display drawn +on stderr only while stderr is a terminal is fine, because it never reaches the file. FAIL if any +animation, redraw or ANSI color can end up in backup.log or the cron output. diff --git a/plugins/user-interface/evals/piped-bash-plain/graders/skill-fired.md b/plugins/user-interface/evals/piped-bash-plain/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/piped-bash-plain/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/piped-bash-plain/prompt.md b/plugins/user-interface/evals/piped-bash-plain/prompt.md new file mode 100644 index 0000000000..0c656cab77 --- /dev/null +++ b/plugins/user-interface/evals/piped-bash-plain/prompt.md @@ -0,0 +1,14 @@ +--- +schema_version: "1.1" +name: piped-bash-plain +description: "Regression guard (the model passes it without the plugin): a spinner and progress bar never reach piped or cron output" +tags: [terminal, regression-guard] +runs: 3 +max_turns: 15 +allowed_tools: [Read, Glob, Grep, Skill] +expected_outcome: "The design checks whether stdout is a terminal and, when piped to tee or run from cron, prints plain parseable lines with no spinner, color or cursor movement" +--- + +Design the output for `backup-run`, a bash script that shows a spinner and a colored progress bar while +it copies files. Ops run it from cron and also as `backup-run | tee backup.log`. Describe exactly what +it prints in each situation, in under 200 words. diff --git a/plugins/user-interface/evals/piped-bash-plain/samples/checks-tty.json b/plugins/user-interface/evals/piped-bash-plain/samples/checks-tty.json new file mode 100644 index 0000000000..b6287964ce --- /dev/null +++ b/plugins/user-interface/evals/piped-bash-plain/samples/checks-tty.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Test `[ -t 1 ]`; when stdout is not a terminal, print one plain line per file.", + "why": "names the test" + }, + { + "answer": "When the output is non-TTY (piped or cron), drop the spinner.", + "why": "names non-TTY" + } + ], + "fail": [ + { + "answer": "When piped, the spinner still shows but in a quieter style.", + "why": "no TTY check" + }, + { + "answer": "Always print the progress bar; tee handles ANSI fine.", + "why": "wrong" + } + ] +} diff --git a/plugins/user-interface/evals/piped-bash-plain/samples/skill-fired.json b/plugins/user-interface/evals/piped-bash-plain/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/piped-bash-plain/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/project-system-first/case.yaml b/plugins/user-interface/evals/project-system-first/case.yaml new file mode 100644 index 0000000000..914adde227 --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/case.yaml @@ -0,0 +1,13 @@ +schema_version: "1.1" +name: project-system-first +description: "Regression guard (the model passes it without the plugin): the project's own tokens and design-system rules decide an empty state" +tags: [project-first, regression-guard] +runs: 3 +expected_outcome: "The design reads design/tokens.json and docs/design-system.md, uses their values, and follows the empty-state rule: one muted line, one secondary button, no illustration" +context: + scaffold_script: scaffold.sh +execution: + prompt: "Design the empty state for the Invoices list in this app, shown before the user has any invoices. Give the layout, the copy and the exact styling values you would use. This run cannot write files, so put the design in your reply." + allowed_tools: [Read, Glob, Grep, Skill] + max_turns: 25 + timeout_seconds: 600 diff --git a/plugins/user-interface/evals/project-system-first/graders/follows-empty-state-rule.md b/plugins/user-interface/evals/project-system-first/graders/follows-empty-state-rule.md new file mode 100644 index 0000000000..64b04ad834 --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/graders/follows-empty-state-rule.md @@ -0,0 +1,8 @@ +--- +type: llm +arm: both +--- +PASS only if the design follows the project's documented empty-state rule (one line of text in the +muted text color, one secondary button, no illustration) and takes its colors, spacing and radius +from the project's tokens rather than inventing values. FAIL if it adds an illustration, an icon +hero, more than one button, or colors and sizes that are not the project's tokens. diff --git a/plugins/user-interface/evals/project-system-first/graders/skill-fired.md b/plugins/user-interface/evals/project-system-first/graders/skill-fired.md new file mode 100644 index 0000000000..f7f9fc8c9b --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/graders/skill-fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: "\"skill\"\\s*:\\s*\"(?:user-interface:)?design\"" +--- diff --git a/plugins/user-interface/evals/project-system-first/graders/uses-project-tokens.md b/plugins/user-interface/evals/project-system-first/graders/uses-project-tokens.md new file mode 100644 index 0000000000..d8d2b2c416 --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/graders/uses-project-tokens.md @@ -0,0 +1,6 @@ +--- +type: regex +flags: i +arm: both +pattern: "0B6E4F|brand\\.pine|\\bpine\\b|tokens\\.json|space\\.gutter|radius\\.card|18px" +--- diff --git a/plugins/user-interface/evals/project-system-first/samples/skill-fired.json b/plugins/user-interface/evals/project-system-first/samples/skill-fired.json new file mode 100644 index 0000000000..58734f1573 --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/samples/skill-fired.json @@ -0,0 +1,32 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "user-interface:design" + } + } + ], + "why": "the design skill fired" + } + ], + "fail": [ + { + "answer": [], + "why": "no tool call" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "frontend-design" + } + } + ], + "why": "another skill" + } + ] +} diff --git a/plugins/user-interface/evals/project-system-first/samples/uses-project-tokens.json b/plugins/user-interface/evals/project-system-first/samples/uses-project-tokens.json new file mode 100644 index 0000000000..838a7d64fe --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/samples/uses-project-tokens.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Pad the card with space.gutter (18px) and use radius.card.", + "why": "token names" + }, + { + "answer": "Primary button in brand pine #0B6E4F.", + "why": "token value" + } + ], + "fail": [ + { + "answer": "Use a soft blue #3B82F6 with a 12px radius.", + "why": "invented values" + }, + { + "answer": "Follow Material defaults.", + "why": "no project values" + } + ] +} diff --git a/plugins/user-interface/evals/project-system-first/scaffold.sh b/plugins/user-interface/evals/project-system-first/scaffold.sh new file mode 100755 index 0000000000..1d56796eb4 --- /dev/null +++ b/plugins/user-interface/evals/project-system-first/scaffold.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Seeds the eval workspace with a small app that has its own design system. +set -euo pipefail + +mkdir -p design docs src +cat > package.json <<'JSON' +{ "name": "ledgerly", "private": true, "dependencies": { "@mui/material": "^6.1.0", "react": "^19.0.0" } } +JSON +cat > design/tokens.json <<'JSON' +{ + "color": { "brand": { "pine": "#0B6E4F" }, "text": { "muted": "#5B6770" }, "surface": { "card": "#F7F5F0" } }, + "space": { "gutter": "18px", "stack": "10px" }, + "radius": { "card": "3px" }, + "font": { "body": "Inter, sans-serif" } +} +JSON +cat > docs/design-system.md <<'MD' +# Ledgerly design system + +- Colors, spacing, radius and type come only from `design/tokens.json`. No other values. +- Empty states: one line of text in `color.text.muted`, one secondary button, and no illustration. +- Cards: `surface.card` background, `radius.card` corners, `space.gutter` padding, no shadow. +- Buttons: MUI `Button`; primary actions use `color.brand.pine`, never gradients. +MD diff --git a/plugins/user-interface/lib/prerequisites.mjs b/plugins/user-interface/lib/prerequisites.mjs new file mode 100755 index 0000000000..22dead9594 --- /dev/null +++ b/plugins/user-interface/lib/prerequisites.mjs @@ -0,0 +1,637 @@ +#!/usr/bin/env node +// GENERATED from lib/prerequisites.mjs by scripts/sync-shared-copies.sh. Do not edit this copy: +// edit the canonical source, then rerun the script. +// Read a plugin's prerequisites.json and report which declared dependencies resolve. +// The contract is docs/conventions/prerequisites/README.md; the schema sits beside it. +// +// prerequisites.mjs report --plugin-root [--plugin-root ...] +// prerequisites.mjs check [--for ] [--data-dir ] +// prerequisites.mjs probe [--run-if-unset-or-true