diff --git a/integrations/docs/kiro-powers.md b/integrations/docs/kiro-powers.md index 43f501e..c4592ae 100644 --- a/integrations/docs/kiro-powers.md +++ b/integrations/docs/kiro-powers.md @@ -21,7 +21,6 @@ The `integrations/kiro-powers/` folder is a [Kiro power](https://kiro.dev/docs/p | `kane-cli run` steering | `integrations/kiro-powers/steering/kane-cli-run.md` | Full reference for one-shot `kane-cli run`: objective patterns + checkpoint analyze methods (Visual / Textual-DOM / URL / Title / DevTools→Network/Console/Performance/Cookies/localStorage), full flag table, NDJSON parsing (including `project_folder_auto_defaulted`), results presentation, failure diagnosis, parallel execution, project/folder management (`projects`/`folders list|create`, auto-default gate). | | `kane-cli testmd` steering | `integrations/kiro-powers/steering/kane-cli-testmd.md` | Full reference for `kane-cli testmd`: file format, frontmatter (incl. `tags:`), `@import`, replay/author cache, `Result.md`, CI patterns, parse errors, generate → testmd pipeline. | | `kane-cli testrun` + evidence steering | `integrations/kiro-powers/steering/kane-cli-testrun.md` | Full reference for `kane-cli testrun run` (selection, preflight, parallel, dry-run, typed `testrun_*` NDJSON events, exit codes) and the evidence-pack surface (`evidence serve/validate/merge`, pack locations, the stderr view hint). | -| `kane-cli generate` steering | `integrations/kiro-powers/steering/kane-cli-generate.md` | Full reference for `kane-cli generate`: the three modes (new / refine / save), attaching files (`--files`), clarification round-trips, the refine→save→run loop, typed NDJSON event schema (`generate_*`), result presentation, the generate → testmd handoff. | | `kane-cli fair-evaluation` steering | `integrations/kiro-powers/steering/kane-cli-fair-evaluation.md` | How to compare Kane CLI to other approaches honestly: like-for-like lifecycle phases (authoring vs the alternative's script *creation*; replay vs execution; plus locator-break repair + ongoing maintenance), the comparison traps (replay is zero-LLM, the sunk-cost trap, phase mismatch), and what Kane CLI is purpose-built for. | | `kane-cli mobile` steering | `integrations/kiro-powers/steering/kane-cli-mobile.md` | Full reference for driving a native mobile app (locally on macOS Apple Silicon, or on the cloud grid from any machine): the `--target desktop\|emulator\|simulator` axis (desktop/browser is the default), `--device-name` / `--os-version` / `--app` selection, the required app under test (build or `APP…` id), one-time local setup + `kane-cli doctor` / `devices list`, the flat `_test.md` `target:` + `app:` (+ `device_name:`/`os_version:`) frontmatter keys, mobile members in `testrun` and `testrun run --remote` (grid rules and preflight codes), and the web-only checkpoint carve-out. | | `kane-cli assurance` steering | `integrations/kiro-powers/steering/kane-cli-assurance.md` | Full reference for the assurance flow: requirement docs → `context ingest`/`extract` → review checkpoints → `design tests` → author → `testrun` → `cover`, the `--mode agent` pause loop (exit 3 = resumable), the review checkpoints, and `maintain reconcile`. | @@ -41,18 +40,16 @@ The canonical skill is a thin `SKILL.md` (7 sections, ~300 lines) plus on-demand | `references/agent-config.md`: saved preferences location, schema, shell read and write, hard-case rules, changing preferences later | `steering/kane-cli-first-run.md` → Saved preferences **and** `POWER.md` → Every session (the read command, first-session detection, watch mode values) | | `references/cards.md`: rules for every card, passed, failed, didn't start, stopped early, possible product bug, saved test, suite (local and cloud grid) | `steering/kane-cli-run.md` → Presenting results (rules + the five one-shot run cards) **and** `steering/kane-cli-testmd.md` → The saved test card **and** `steering/kane-cli-testrun.md` → Presenting results (suite card, failed-tests table, cloud grid rows) | | `references/live-strip.md`: the live status strip (Claude Code only) | One sentence in `POWER.md` → Every session saying the strip is Claude Code only and not offered in Kiro. Do **not** document the strip commands in the power | -| `SKILL.md` §2 Decision tree | `steering/kane-cli-run.md` → Decision tree, `steering/kane-cli-testmd.md` → When to recommend `testmd`, and `steering/kane-cli-generate.md` → When to recommend `generate` | -| `SKILL.md` §3 Building a `run` command — flags, exit codes, examples, bare-objective guardrail | `POWER.md` → Command reference (condensed) **and** `steering/kane-cli-run.md` → Full flag reference | +| `SKILL.md` §2 Decision tree | `steering/kane-cli-run.md` → Decision tree, `steering/kane-cli-testmd.md` → When to recommend `testmd`, and `steering/kane-cli-assurance.md` for every test-case authoring ask | +| `SKILL.md` §3 Building a `run` command — flags, exit codes, examples, bare-objective guardrail, unresolved variables and the fill-before-any-run recipe | `POWER.md` → Command reference (condensed) **and** `steering/kane-cli-run.md` → Full flag reference + Variables and secrets | | `SKILL.md` §4 Writing objectives — three patterns, "store as", do/don't | `steering/kane-cli-run.md` → Writing objectives — three patterns | | `SKILL.md` §5 Parsing `--agent` output — essentials (includes `project_folder_auto_defaulted` in typed events) | `steering/kane-cli-run.md` → Parsing the NDJSON output (Event types + Parsing strategy summary) | -| `SKILL.md` §6 Generate test cases (authoring — no browser) | `POWER.md` → Overview (the third "way Kiro uses it") **and** all of `steering/kane-cli-generate.md` | +| `SKILL.md` §6 Test cases from a description (no requirement document) | `POWER.md` → Overview (the third "way Kiro uses it") **and** `steering/kane-cli-testmd.md` → Authoring from a description **and** `steering/kane-cli-assurance.md` | | `SKILL.md` §7 When to read which reference | Kiro analogue: `POWER.md`'s steering-file mapping (POWER.md tells Kiro when to load each steering file) | | `references/objectives-cookbook.md` — analyze methods (Visual / Textual-DOM / URL / Title / DevTools→Network/Console/Performance/Cookies/localStorage), operators, chaining, pitfalls, worked examples | `steering/kane-cli-run.md` → Analyze methods — picking the right checkpoint (plus the existing Combining patterns, Assertion specificity, and Do / Don't sections) | -| `references/testmd.md`: testmd file format, replay & cascade, `@import`, commands, parse errors, gate-fires-before-launch note, the saved-test stream table (`test_md_step_start` through `test_md_done`) | All of `steering/kane-cli-testmd.md` (the stream table lives under Replay policy and completion → The saved-test stream) | -| `references/generate.md` — generate modes, attaching files (`--files`), refine→save→run loop, clarification handling, Functional-only save, generate→testmd handoff | All of `steering/kane-cli-generate.md` | -| `references/generate-parsing.md` — typed `generate_*` event schema (incl. `generate_upload`), terminal `generate_done`, exit codes | `steering/kane-cli-generate.md` → Parsing the NDJSON output + Terminal `generate_done` + Exit codes | +| `references/testmd.md`: testmd file format, replay & cascade, `@import`, commands, parse errors, gate-fires-before-launch note, the saved-test stream table (`test_md_step_start` through `test_md_done`) | All of `steering/kane-cli-testmd.md` (the stream table lives under Replay policy and completion → The saved-test stream); authoring from a description points at the assurance steering | | `references/parsing.md`: the 0.8.17+ stream contract (`stream_start`, `v`, `ts`, ignore unknown fields and event types, typeless step lines, completion event last, `events.ndjson` and the active-run pointer file), full NDJSON event schemas (`project_folder_auto_defaulted`, `bifurcation`, `child_agent_*`, `ask_user`, complete `run_end` fields incl. `credits_consumed`) | `steering/kane-cli-run.md` → Parsing the NDJSON output (The stream contract + full event-type list + Terminal `run_end` event) | -| `references/test-manager.md`: project/folder agent surface (`projects list|create`, `folders list|create` with the **required** `--project `, NDJSON wire shape, pagination), run-startup auto-default gate, `project_folder_auto_defaulted` event, self-healing for stale IDs, the global "changing where results go" flow | `POWER.md` → Step 3 (project/folder onboarding) **and** `steering/kane-cli-run.md` → Browsing / creating projects and folders + The run-startup auto-default gate + Changing where results go **and** `steering/kane-cli-testmd.md` → Quick start (gate note) **and** `steering/kane-cli-generate.md` → Configuration surface (gate note) | +| `references/test-manager.md`: project/folder agent surface (`projects list|create`, `folders list|create` with the **required** `--project `, NDJSON wire shape, pagination), run-startup auto-default gate, `project_folder_auto_defaulted` event, self-healing for stale IDs, the global "changing where results go" flow | `POWER.md` → Step 3 (project/folder onboarding) **and** `steering/kane-cli-run.md` → Browsing / creating projects and folders + The run-startup auto-default gate + Changing where results go **and** `steering/kane-cli-testmd.md` → Quick start (gate note) | | `references/debug.md` — log layout, debugging flow, common failure patterns (incl. "did you mean" subcommand and self-healing rows), bug-report heuristic | `steering/kane-cli-run.md` → Failure handling & log inspection + Bug-report heuristic | | `references/parallel.md` — when to split, agent prompt template, batch summary | `steering/kane-cli-run.md` → Parallel execution | | `references/testrun.md`: batch runs: selection (paths/`--match`/`--tags`), preflight reasons, `--remote` (HyperExecute jobs, `remote_*` events, remote preflight codes, the 0.8.17+ remote additions), mobile members, `testrun_*` event schema (incl. `session_id`, `log_path`, `failure`, the authored member pair, `testrun_progress`), each test's own log and the `--stream-members` rule, suite card presentation, exit codes | All of `steering/kane-cli-testrun.md` | diff --git a/integrations/kiro-powers/POWER.md b/integrations/kiro-powers/POWER.md index e796911..ef2936a 100644 --- a/integrations/kiro-powers/POWER.md +++ b/integrations/kiro-powers/POWER.md @@ -1,7 +1,7 @@ --- name: "kane-cli" displayName: "Kane CLI — Browser Automation & AI Test Authoring" -description: "Drive a real browser from natural-language objectives, AND author structured test cases / scenarios from a feature description. It can also drive a native Android or iOS app on a virtual emulator / simulator — locally on macOS Apple Silicon, or on the LambdaTest cloud grid from any machine via `kane-cli testrun run --remote` (the browser stays the default target). Kane CLI manages Chrome, an AI agent, and a TestMu AI session, then emits structured NDJSON Kiro can parse. Use for any task that needs a real browser (navigation, form fills, search, UI checks, screenshots, deploy verification), a native mobile app (locally on macOS Apple Silicon, or on the cloud grid from any machine), to author test cases / scenarios from a requirement (`kane-cli generate`), or for committable, replay-cached browser tests (`kane-cli testmd`)." +description: "Drive a real browser from natural-language objectives, AND design requirement-linked test suites from a PRD, a spec, or a description the user gives (assurance). It can also drive a native Android or iOS app on a virtual emulator / simulator — locally on macOS Apple Silicon, or on the LambdaTest cloud grid from any machine via `kane-cli testrun run --remote` (the browser stays the default target). Kane CLI manages Chrome, an AI agent, and a TestMu AI session, then emits structured NDJSON Kiro can parse. Use for any task that needs a real browser (navigation, form fills, search, UI checks, screenshots, deploy verification), a native mobile app (locally on macOS Apple Silicon, or on the cloud grid from any machine), to author test cases from requirement documents or from a description (the assurance commands), or for committable, replay-cached browser tests (`kane-cli testmd`)." keywords: - "kane-cli" - "kaneai" @@ -75,7 +75,7 @@ npm install -g @testmuai/kane-cli Then re-verify with `kane-cli --version`. Requirements: **Node.js 18+** and **Google Chrome** installed locally (Kane CLI auto-launches Chrome over CDP on ports 9222–9230). -**Local mobile app testing is scoped to macOS on Apple Silicon (arm64); the cloud grid lifts that.** The default browser (desktop) target has no host requirement. To drive a native Android or iOS app *locally* the user needs macOS Apple Silicon plus **Xcode 16+** (iOS) or **Android Studio** with one `arm64-v8a` AVD (Android), then `kane-cli login` and `kane-cli doctor --target emulator|simulator --install`. From **any other machine**, a saved mobile `_test.md` suite runs on a HyperExecute emulator/simulator with `kane-cli testrun run … --remote --device-name "" --os-version ` (needs a HyperExecute plan with macOS runners and `kane-cli plugin install remote-execution`). Load the **`kane-cli-mobile`** steering file for both flows. +**Local mobile app testing is scoped to macOS on Apple Silicon (arm64); the cloud grid lifts that.** The default browser (desktop) target has no host requirement. To drive a native Android or iOS app *locally* the user needs macOS Apple Silicon plus **Xcode 16+** (iOS) or **Android Studio** with one `arm64-v8a` AVD (Android), then `kane-cli login` and `kane-cli doctor --target emulator|simulator --install`. From **any other machine**, a saved mobile `_test.md` suite runs on a HyperExecute emulator/simulator with `kane-cli testrun run … --remote --device-name "" --os-version < /dev/null` (needs a HyperExecute plan with macOS runners and `kane-cli plugin install remote-execution`). Load the **`kane-cli-mobile`** steering file for both flows. Common install failures: @@ -271,9 +271,9 @@ Three ways Kiro uses it: 1. **Ad-hoc browser tasks — `kane-cli run`.** One-shot natural-language objectives. The process exits when the agent reports `run_end`. Use this for navigation, form fills, search, verification, screenshots, data extraction, and deploy smoke tests. 2. **Committable, replayable tests — `kane-cli testmd`.** Tests live as `_test.md` files in the repo. The first run authors each step (agent figures the page out); every later run replays from a cache — no agent, no LLM cost, much faster. Use this for regression suites, CI gates, and validation hooks. -3. **AI test-case authoring — `kane-cli generate`.** Turns a plain-language description of *what to test* into structured Test Scenarios + typed Test Cases (Positive / Negative / Edge). **No browser is launched.** Reach for it whenever a task needs test cases written — don't hand-draft them in chat or a scratch file. `--save` writes Functional cases as runnable `_test.md`, which feeds straight into `kane-cli testmd run` (the generate → testmd pipeline). +3. **Test-case authoring — the assurance commands.** Turns requirement documents, or a description the user gives in chat, into designed, requirement-linked `_test.md` tests: a document is ingested (a chat description is first written to `requirements/.md` in the user's words), the extracted use-cases are reviewed, `kane-cli design tests` designs them, the designed tests are reviewed, the variables design declared are filled, and `kane-cli testrun run … < /dev/null` authors and replays them. **No browser is launched by design.** Never hand-draft cases in chat or a scratch file. Load `steering/kane-cli-assurance.md`. -Use `--agent` for `run`, `testmd run`, and `generate`. `testrun run` has no `--agent`: it emits NDJSON when **stdin** is not a TTY (use `< /dev/null` for terminal automation). Assurance conversational commands use `--mode agent`. +Use `--agent` for `run` and `testmd run`. `testrun run` has no `--agent`: it emits NDJSON only when **stdin** is not a TTY, so every `testrun run` line ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. Other capabilities to know about: @@ -297,11 +297,11 @@ When the user's task makes one of these patterns relevant, load the matching ste - **`steering/kane-cli-mobile.md`**: any time the user wants to drive a **native mobile app** (Android or iOS) instead of the browser — locally on **macOS Apple Silicon**, or on the **cloud grid from any machine**. Covers the `--target desktop|emulator|simulator` axis (desktop / browser stays the default), selecting a device (`--device-name` + `--os-version`, `kane-cli devices list [--remote]`) and the required app under test (`--app `, `kane-cli apps list`), one-time local setup (Xcode / Android Studio plus `kane-cli doctor --target … --install`), the flat `_test.md` `target:` + `app:` (+ `device_name:`/`os_version:`) frontmatter keys, mobile members in `kane-cli testrun`, and `testrun run --remote` — the grid rules (one platform per job; simulator `.zip` builds auto-upload on real runs, or use uploaded `APP…` ids) and its preflight codes. - **`steering/kane-cli-testmd.md`** — any time the user wants a committable test, or is reading / editing / running a `_test.md` file. Covers the `kane-cli testmd` commands, `_test.md` file format and frontmatter (including `tags:`), `@import` composition, the replay-vs-author cache model, `Result.md`, lock conflicts, and CI patterns. - **`steering/kane-cli-testrun.md`** — any time the user wants to run **several** saved `_test.md` tests as one batch ("run the suite", "run all the smoke tests", "nightly regression"), or asks about evidence packs (viewing, sharing, validating a run's `.evidence` file). Covers `kane-cli testrun run` (selection by paths / `--match` / `--tags`, preflight, `--parallel`, `--dry-run`), its typed NDJSON events, exit codes, and the `kane-cli evidence` commands. -- **`steering/kane-cli-generate.md`** — any time the user wants test cases or scenarios **written** (no browser action). Covers the three generate modes (new / refine / save), clarification round-trips, the refine→save→run loop, the typed NDJSON event schema, and the generate → testmd handoff. +- Test cases or scenarios **written** (no browser action) → **`steering/kane-cli-assurance.md`**: a requirement document, or a chat description written to `requirements/.md`, is ingested and designed from; no other authoring path exists. - **`steering/kane-cli-fair-evaluation.md`** — any time the user asks you to compare, evaluate, benchmark, or justify Kane CLI against another tool or approach (cost, tokens, effort, ROI). Covers the like-for-like lifecycle comparison method (Kane authoring vs the alternative's script *creation*; replay vs execution; plus locator-break repair and ongoing maintenance), the common comparison traps, and what Kane CLI is purpose-built for. - **`steering/kane-cli-assurance.md`** — any time the user has **requirement documents** (a PRD, a spec, acceptance notes) and wants tests designed from them, coverage accounting ("what exactly is covered?"), or the suite reconciled after a requirements change. Covers the assurance journey (`context ingest`/`extract` → review → `design tests` → author → `testrun` → `cover`), the pause loop (`--mode agent`, exit 3 = resumable), the review checkpoints, and `maintain reconcile`. -Default to `kane-cli-run.md` for one-shot browser tasks. Switch to `kane-cli-mobile.md` when the user wants to test a **native Android or iOS app** rather than the browser (locally on macOS Apple Silicon, or on the cloud grid from any machine with `testrun run --remote`). Desktop / browser stays the default target, so only load it once the user asks for a mobile app. Switch to `kane-cli-testmd.md` the moment the user says anything like "save this test", "commit this", "regression / smoke test", "make this replayable", or "run in CI" — or asks about a `_test.md` file by name. Switch to `kane-cli-testrun.md` when the user wants **two or more** saved tests run together, or asks to see/share run evidence. Switch to `kane-cli-generate.md` when the user says anything like "write test cases for", "give me a test suite for", "generate tests for", "what edge cases should we cover" — or when the task needs cases authored but no browser. **Don't hand-draft test cases in chat or a scratch file** — load the generate steering and use `kane-cli generate`. Switch to `kane-cli-assurance.md` when the user brings **requirement documents** (a PRD, a spec) and wants a designed suite, requirement-linked coverage, or the suite reconciled after a requirements change — that flow uses `--mode agent` (not `--agent`) and its own exit-code meanings. Load **`kane-cli-fair-evaluation.md`** whenever the user asks whether Kane CLI is worth it, or to compare its cost / effort / ROI against generating Playwright scripts or any other approach — comparisons are only honest like-for-like across the test lifecycle. +Default to `kane-cli-run.md` for one-shot browser tasks. Switch to `kane-cli-mobile.md` when the user wants to test a **native Android or iOS app** rather than the browser (locally on macOS Apple Silicon, or on the cloud grid from any machine with `testrun run --remote`). Desktop / browser stays the default target, so only load it once the user asks for a mobile app. Switch to `kane-cli-testmd.md` the moment the user says anything like "save this test", "commit this", "regression / smoke test", "make this replayable", or "run in CI" — or asks about a `_test.md` file by name. Switch to `kane-cli-testrun.md` when the user wants **two or more** saved tests run together, or asks to see/share run evidence. Switch to `kane-cli-assurance.md` when the user says anything like "write test cases for", "give me a test suite for", "generate tests for", "what edge cases should we cover", when the task needs cases authored but no browser (a chat description is written to `requirements/.md` and designed from; **don't hand-draft test cases in chat or a scratch file**), or when the user brings **requirement documents** (a PRD, a spec) and wants a designed suite, requirement-linked coverage, or the suite reconciled after a requirements change — that flow uses `--mode agent` (not `--agent`) and its own exit-code meanings. Load **`kane-cli-fair-evaluation.md`** whenever the user asks whether Kane CLI is worth it, or to compare its cost / effort / ROI against generating Playwright scripts or any other approach — comparisons are only honest like-for-like across the test lifecycle. # Command reference (condensed) @@ -347,7 +347,7 @@ kane-cli config chrome-profile kane-cli feedback --test-id --feedback-type --details "..." # Batch runs — many _test.md files, one execution, one evidence pack -kane-cli testrun run [paths...] --agent [--match ] [--tags ] [--parallel ] [--dry-run] +kane-cli testrun run [paths...] [--match ] [--tags ] [--parallel ] [--dry-run] < /dev/null # Evidence packs kane-cli evidence validate --json # exit 0 valid / 1 invalid / 2 not found @@ -395,7 +395,7 @@ kane-cli run "Go to {{app_url}}/dashboard, store the welcome message as 'welcome **Suite run** — several saved tests as one execution: ```bash -kane-cli testrun run --tags smoke --parallel 4 --headless --agent +kane-cli testrun run --tags smoke --parallel 4 --headless < /dev/null ``` One summary, one exit code (`0` all passed · `1` any failure · `2` invalid plan · `3` cancelled), one sealed evidence pack in `.testmuai/evidence/`. @@ -405,7 +405,7 @@ For `_test.md` examples and the full `kane-cli testmd` reference, load **`kane-c # Best practices - **Start every session with the ready check and end every run with its result card** (see **Every session**). Put `KANE_CLI_USER_AGENT=kiro` inline on every command. -- Use `--agent` on `run`, `testmd run`, and `generate`; use non-TTY stdin for `testrun`, and `--mode agent` for conversational Assurance commands. +- Use `--agent` on `run` and `testmd run`; end every `testrun run` line in `< /dev/null` (stdin not a TTY selects NDJSON, and the first stdout line must be `stream_start`); use `--mode agent` for conversational Assurance commands. - **Include the starting URL in the objective.** Don't assume the agent knows where to start. - **Use imperative verbs:** "go to", "click", "type", "store as", "assert". - **Use the `store … as ''` pattern** for any value you want back. Vague phrasing won't persist. diff --git a/integrations/kiro-powers/steering/kane-cli-assurance.md b/integrations/kiro-powers/steering/kane-cli-assurance.md index cb30992..771bbfd 100644 --- a/integrations/kiro-powers/steering/kane-cli-assurance.md +++ b/integrations/kiro-powers/steering/kane-cli-assurance.md @@ -1,6 +1,6 @@ # Kane CLI — assurance steering (requirements → designed suite → coverage → upkeep) -Load this file when the user has **requirement documents** (a PRD, a spec, acceptance notes) and wants tests designed from them, coverage accounting ("what exactly is covered?"), or the suite kept current when requirements change. For quick test cases from a one-line description, use `kane-cli generate` instead (load `kane-cli-generate.md`). Never write test cases by hand. +Load this file when the user has **requirement documents** (a PRD, a spec, acceptance notes) and wants tests designed from them, coverage accounting ("what exactly is covered?"), or the suite kept current when requirements change. A description given in chat with no document is written to `requirements/.md` in the user's words and follows the same path. Never write test cases by hand. Requires kane-cli 0.6.1+ — on an older CLI, `kane-cli context …` fails as an *unknown command* (exit 2). That means the CLI is too old, not a typo: confirm with `kane-cli --version`, have the user update, and stop rather than improvising the workflow with other commands. Flags marked with a release (0.7.1+ … 0.8.6+) need at least that release — an *unknown option* error on one of them means the same thing. @@ -12,7 +12,7 @@ kane-cli context review --verdicts --json # 2. CHECKPOINT: us kane-cli design tests --use-case --mode agent --max 8 # 3. design ACs, scenarios, tests kane-cli context review --verdicts --json # 4. CHECKPOINT: user approves the design kane-cli testmd run .testmuai/tests/_test.md --agent # 5. author each kept test once (real browser) -kane-cli testrun run --match 't-' # 6. batch replays from then on +kane-cli testrun run --match 't-' < /dev/null # 6. batch replays from then on kane-cli cover gaps # 7. the coverage ribbon; drill in with `cover gaps ` for commands ``` @@ -30,7 +30,7 @@ The conversational assurance commands (`context ingest`/`extract`, `design tests 2. Resume in **plain words**: `kane-cli context extract --resume --mode agent --message ""` — or, on 0.7.1+, by id (`--answer q1=2 --answer q2=""`) or by landing the source the agent needs (`--with-source ` — the pending questions defer while it reads, then only what's still open is re-asked). 3. Sessions are durable on 0.7.1+ — a crash that left a checkpoint exits 3 with the resume command. They expire in 24 h; `kane-cli context sessions --json` lists them; `sessions clean ` removes one you abandoned. -This exit-3 meaning applies to these assurance commands only — `run`/`testmd`/`testrun`/`generate` keep 3 = timeout/cancelled. +This exit-3 meaning applies to these assurance commands only — `run`/`testmd`/`testrun` keep 3 = timeout/cancelled. # Design rules @@ -38,7 +38,7 @@ Gates return commands, not dead ends (0.7.1+): designing an unreviewed or alread # The authoring bridge -On 0.8.4+, batch designed tests directly with `kane-cli testrun run` — unauthored members classify as `author`, the run authors them, and the evidence consolidates afterwards (best-effort); `--from-context` selects members by assurance test ids. On pre-0.8.4 CLIs, `testrun` refuses never-authored members (`missing_meta`) — author each once with `kane-cli testmd run` first. Designed tests may carry `{{variables}}` for values the requirements never pinned. On 0.8.2+, `kane-cli cover gaps` is the **coverage ribbon** (one row per use-case, no commands); drill into `cover gaps ` for every AC plus a `next` actions block, or `--json` for `ready_command` rows (`--rollup lenient|strict` picks the proven formula; `--flat` and gaps' `--from` are removed and refuse; `--mode agent` delivers the same data as one event plus `next[]`). A failing row leads with the evidence command — surface its warning that re-running an authored test against a broken app may heal the test around the failure. +On 0.8.4+, batch designed tests directly with `kane-cli testrun run` — unauthored members classify as `author`, the run authors them, and the evidence consolidates afterwards (best-effort); `--from-context` selects members by assurance test ids. On pre-0.8.4 CLIs, `testrun` refuses never-authored members (`missing_meta`) — author each once with `kane-cli testmd run` first. Designed tests may carry `{{variables}}` for values the requirements never pinned: design declares each name it invents as an empty stub in `.testmuai/variables/assurance.json` and reports them (`variables_declared` during the run, `variables_summary` on a clean completion); surface them as a to-do and fill them before any run, per **Fill the variables before any run** in the `kane-cli-run` steering file, or the run types the placeholder as written. On 0.8.2+, `kane-cli cover gaps` is the **coverage ribbon** (one row per use-case, no commands); drill into `cover gaps ` for every AC plus a `next` actions block, or `--json` for `ready_command` rows (`--rollup lenient|strict` picks the proven formula; `--flat` and gaps' `--from` are removed and refuse; `--mode agent` delivers the same data as one event plus `next[]`). A failing row leads with the evidence command — surface its warning that re-running an authored test against a broken app may heal the test around the failure. # When requirements change diff --git a/integrations/kiro-powers/steering/kane-cli-generate.md b/integrations/kiro-powers/steering/kane-cli-generate.md deleted file mode 100644 index fc9f718..0000000 --- a/integrations/kiro-powers/steering/kane-cli-generate.md +++ /dev/null @@ -1,307 +0,0 @@ -# Kane CLI — `kane-cli generate` steering - -Load this steering file whenever the user wants **test cases or test scenarios written** — because they asked, or because the task needs them — rather than a browser action. `kane-cli generate` turns a plain-language description of *what to test* into structured Test Scenarios (logical groupings), each containing typed Test Cases (Positive / Negative / Edge). It does **not** drive a browser. The result is a tree of scenarios + cases you present to the user, refine conversationally, and optionally save as runnable `_test.md` files. - -The single rule governing this framework: **every invocation is one turn, then exit.** There is no interactive session — continuity across turns is carried by a request id that the previous turn hands back. Treat each `kane-cli generate` call as one round-trip with the AI Test Case Generator. - -`kane-cli run` is for browser actions. `kane-cli testmd` is for running saved tests. `kane-cli generate` is for **authoring** test cases — when a task says "write tests for X" or "I need a regression suite for the checkout flow", reach for this, not for hand-drafting cases in chat or a scratch file. - ---- - -# When to recommend `generate` - -The decision is about the **deliverable**: does the user want browser action, a saved test to run, or a *suite of test cases to write*? - -## User-phrase triggers - -| User says | Use | -|---|---| -| "write test cases for the login flow" | ✅ `generate` | -| "give me a test suite for checkout" | ✅ `generate` | -| "what edge cases should we cover for the password reset?" | ✅ `generate` | -| "generate tests for the new feature spec" | ✅ `generate` | -| "more negative cases for this scenario" | ✅ `generate` with `--refine` | -| "save these as runnable tests" | ✅ `generate` with `--save`, then `kane-cli testmd run` | -| "run a smoke test on the checkout page" | ❌ `kane-cli run` | -| "verify the dashboard loads after login" | ❌ `kane-cli run` | -| "save THIS browser session as a regression test" | ❌ `kane-cli run --name`, then `kane-cli testmd` | - -## When the intent is unclear - -Ask, don't guess: - -> "Do you want me to write test cases for this (no browser), or run a browser check?" - -Default to `kane-cli generate` when the task needs cases authored. **Don't hand-draft test cases in chat or a scratch file** — generate them so they come out structured, refinable, and runnable as `_test.md`. - ---- - -# The three modes — one turn per invocation, then exit - -There is **no interactive session**. Each invocation runs exactly one generation turn and exits. Continuity across turns is carried by a **request id** (`--req `) that the previous turn's terminal line hands back. - -| Mode | Command | Notes | -|---|---|---| -| **New** | `kane-cli generate "" --agent` | Starts a fresh request. Capture the request id from the terminal line. | -| **Refine** | `kane-cli generate "" --refine --req --agent` | Adjusts an existing request. `--refine` **and** `--req` required; needs a change description. | -| **Save** | `kane-cli generate --save --req [--out ] --agent` | Writes the request's Functional cases to `_test.md`. No new turn, takes no objective. | - -`--refine` and `--save` always run headless (even from a terminal). - -## Flag reference - -| Flag | Purpose | -|---|---| -| `--agent` | **Required for Kiro.** Typed NDJSON on stdout. Auto-on when stdin is not a TTY, but pass it explicitly. | -| `--req ` | The request id to `--refine` or `--save`. | -| `--out ` | Save target — **only** with `--save`; default `/.testmuai/tests`. | -| `--name ` | Names the run and the saved suite folder. | -| `--scenario-limit ` / `--per-scenario-limit ` | Cap scenarios / cases-per-scenario. | -| `--memory` | Use the memory layer — reuse relevant existing cases, reduce duplicates. | -| `--files ` | Comma-separated local files to attach as context (new / refine only — see "Attaching files"). | -| `--project ` / `--folder ` | Test Manager project / folder. | -| `--username` / `--access-key` | Auth (same as `run`). | - -If neither `--project`/`--folder` nor a saved project/folder is set when generation starts, Kane CLI auto-resolves one headlessly and emits a `project_folder_auto_defaulted` event before `generate_start` — surface as a one-line note and continue parsing. - -## Attaching files - -Pass local files as extra context with `--files ` on a **new** generation or a **`--refine`** (not `--save`). The generator reads them and reflects them in the scenarios + cases — a spec, a UI screenshot, a PDF / Word doc, or a CSV of inputs. - -```bash -kane-cli generate "test the login flow described in the attached spec" --files ./login-spec.pdf,./mockup.png --agent -``` - -- **Supported types** — documents (`.txt .json .xml .csv .pdf .docx .xlsx`), images (`.jpg .jpeg .png .gif .bmp .webp`), audio (`.mp3 .wav .m4a`), video (`.mp4 .mov .webm .mpeg .mpga`). -- **Limits** — up to **10 files**, each **≤ 50 MB**. -- **Validated as a set, up front** — if any path is missing, an unsupported type, too large, or over the count, the whole command is rejected (exit `2`) **before anything is sent** and the offending paths are listed; fix and re-run. Files outside the working directory are allowed but flagged with a warning on stderr. -- **`new` / `--refine` only** — combining `--files` with `--save` exits `2`. - -Each attached file emits a `generate_upload` line (`status` `uploading` → `done`) **before** `generate_start` (see "Parsing the NDJSON output"). The `@`-mention palette is the interactive-TUI way to attach — **not** relevant to Kiro/agent usage; pass `--files`. - ---- - -# The refine → save → run loop - -```bash -# 1. New request -kane-cli generate "checkout flow on a shopping site" --agent -# → terminal line carries request id (e.g. 23271) + Refine/Save hints - -# 2. Refine (repeat as needed) -kane-cli generate "also cover an expired card and an out-of-stock item" --refine --req 23271 --agent - -# 3. Save the Functional cases as runnable _test.md -kane-cli generate --save --req 23271 --agent -# → /.testmuai/tests///_test.md - -# 4. Run / replay them — switches to the `kane-cli-testmd` steering file -kane-cli testmd run .testmuai/tests///_test.md --agent -``` - -**Save is Functional-only.** `--save` writes only test cases whose category is **Functional** — those are the ones runnable as `_test.md`. Non-functional cases (Security, Performance, etc.) are generated and shown in the result but are **not** written; saving a request with no Functional cases writes nothing and says so. Saved files are ordinary `_test.md` tests — see the **`kane-cli-testmd`** steering file for running, editing, and replay. This is the **generate → testmd** pipeline: author cases here, run them there. - ---- - -# Clarifications — act on them, never drop them - -If a turn ends with a **clarification question**, that is **success (exit 0)**, not an error — the generator needs an answer before it can continue. You must act on it: - -1. Read the question. -2. **Decide** — answer it yourself from context, **or** surface it to your user and get an answer. -3. **Re-invoke** with the answer as a refine: - - ```bash - kane-cli generate "" --refine --req --agent - ``` - -Never drop a clarification. A turn that ended awaiting one is not "done" — it's blocked on a single round-trip. - ---- - -# Parsing the NDJSON output - -> **Internal reference only.** Never echo these field names (`generate_snapshot`, `request_id`, `generate_done`, `generate_clarification`, `_meta`) back to the user. Translate them into plain language. - -Unlike `run` (which has untyped `step`/`status` progress lines), **every `generate` line is typed** — it always has a `type` field. That makes parsing simpler: - -``` -for each line on stdout: - parse JSON, switch on obj.type - if obj.type === "generate_done" → terminal event, stop parsing - if obj.type === "generate_snapshot" → the deliverable (full scenarios + cases) - if obj.type === "generate_clarification" → turn ended awaiting an answer (still exit 0) - else → progress / informational -``` - -Build post-turn logic on **`generate_done`** (terminal, stable schema) and read the result from **`generate_snapshot`** (emitted once, at turn end). - -## Event types - -Generate-specific (typed `generate_*`): - -| `type` | Key fields | Meaning → what to do | -|---|---|---| -| `generate_upload` | `file`, `index`, `total`, `status` | Only when `--files` is used. One per attached file, **before `generate_start`**; `status` goes `"uploading"` → `"done"` (or `"failed"`). Progress only — narrate "attaching …" or ignore. A failed upload surfaces as an `error` + non-zero exit. | -| `generate_start` | `request_id`, `objective_chars`, `scenario_limit`, `per_scenario_limit`, `is_refine` | Turn began. **Capture `request_id`** — it's the handle for every later `--refine` / `--save`. `is_refine` distinguishes a new request from a continuation. | -| `generate_thinking` | `took_ms` | Liveness only. Narrate "thinking…" or ignore. | -| `generate_progress` | `pct` | Milestone (25 / 50 / 75 / 100). Optional progress display — not a completion signal. | -| `generate_snapshot` | `scenario_count`, `case_count`, `scenarios[]` | **The deliverable** — full scenarios, each with its cases. Each case carries `title`, `polarity` (`"p"`/`"n"`/`"e"` = Positive / Negative / Edge), `category` (`"Functional"`, `"Security"`, …), `priority`. Emitted exactly once. Present per "Presenting results" below. | -| `generate_clarification` | `text` | The generator needs an answer; the turn ended (exit 0) awaiting it. **Not an error** — answer via `--refine --req`. | -| `generate_chat` | `text` | The model's prose reply (e.g. what a refine changed). Show as info. | -| `generate_save_result` | `suite_dir`, `saved`, `fell_back`, `warning?` | `--save` wrote files. `saved` = files written; `fell_back` = cases written as prose because they couldn't be expanded; `warning` e.g. `"no functional test cases"`. | -| `generate_done` | `request_id`, `status`, `scenario_count`, `case_count`, `refine_hint`, `save_hint`, `suite_dir?` | **Terminal line.** Branch on `status`; use `refine_hint` / `save_hint` **verbatim** as the next command. `suite_dir` present only after a `--save`. | - -Shared with `run` (not generate-specific): - -| `type` | Fields | Meaning | -|---|---|---| -| `project_folder_auto_defaulted` | resolved project + folder (id, name) | Run-startup gate auto-resolved a project/folder when none was configured. Fires **before** `generate_start`. Translate to a one-line note ("Kane CLI auto-selected project X / folder Y for this turn"). | -| `error` | `message` | A failure occurred; pair with the exit code to decide retry vs abort. | -| `update_available` | `current`, `latest`, `severity` | A newer Kane CLI exists. Informational; emitted before `generate_start`. | - -## Terminal `generate_done` - -Example after a **`--save`** run (note `suite_dir` — a new/refine turn omits it): - -```json -{ - "type": "generate_done", - "request_id": "23271", - "status": "completed", - "scenario_count": 3, - "case_count": 11, - "refine_hint": "kane-cli generate \"\" --refine --req 23271", - "save_hint": "kane-cli generate --save --req 23271", - "suite_dir": ".testmuai/tests/checkout-23271" -} -``` - -`status` values: - -| Value | Exit | Meaning | -|---|---|---| -| `completed` | `0` | Turn finished cleanly (includes turns that ended with a clarification). | -| `failed` / `ended` | `1` | Generation failed or stopped without producing the deliverable. | -| `stopped` | `3` | Stopped / cancelled. | - -The hints carry no `--agent` flag, but it is auto-on for non-TTY callers (agents/pipes), so re-invoking a hint verbatim still yields NDJSON. - -## Exit codes - -| Code | Meaning | -|---|---| -| `0` | Turn completed (including a turn that ended with a clarification). | -| `1` | Generation failed or stopped without producing the deliverable. | -| `2` | Error (auth / setup / transport, or an invalid flag combination). | -| `3` | Stopped / cancelled. | -| `130` | Interrupted (Ctrl-C). | - -Invalid flag combinations exit `2` with a message on stderr. The full set: - -- `--refine` and `--save` together -- `--refine` without `--req` -- `--refine` without a change description -- `--refine` combined with `--out` (`--out` is save-only) -- `--save` without `--req` -- `--save` with a description (it takes none) -- `--out` without `--save` -- `--files` with `--save` (files attach to a new generation or a refine, not a save) -- `--req` without `--refine` or `--save` -- a new generation with no description - ---- - -# Presenting results - -Present the result **adaptively** based on size, and always offer the next-step commands the terminal line hands back. - -## ≤ ~30 cases → a nested tree - -Scenario, then each case with its type tag: - -``` -✓ Generated 3 scenarios · 11 cases (request 23271) - -▸ Login - - Valid credentials [Positive] - - Wrong password [Negative] - - Empty fields [Edge] -▸ Checkout - - Guest checkout [Positive] - - Expired card [Negative] - - Cart total mismatch [Edge] -▸ Cart management - ... -``` - -## More than ~30 cases → summary + scenario list - -Cases on request: - -``` -✓ Generated 6 scenarios · 84 cases (request 23271) - - • Login (12 cases) - • Checkout (20 cases) - • Cart management (14 cases) - • ... -``` - -Expand a scenario's cases only when the user asks. - -## Always end with next-step hints - -The terminal `generate_done` event carries `refine_hint` and `save_hint` — present them **verbatim** (they already carry the request id). Don't hand-build them. - -> Want to refine? `kane-cli generate "" --refine --req 23271` -> Ready to save the Functional cases? `kane-cli generate --save --req 23271` - -## Field names are internal - -Never expose `generate_snapshot`, `request_id`, `generate_done`, `polarity`, `category`, `_meta` to the user. Translate `"p"` → "Positive", `"n"` → "Negative", `"e"` → "Edge". Translate scenarios as "scenarios" and cases as "test cases" — that's the user-facing vocabulary. - ---- - -# Failure handling - -| Symptom | Likely cause | Fix | -|---|---|---| -| Exit `2` with "invalid flag combination" | Mixed `--refine` + `--save`, or missing `--req` | Re-invoke with the correct mode (see "The three modes" above). | -| Exit `2` listing bad `--files` paths | A path is missing / unsupported type / >50 MB / >10 files | Fix the offending paths (the message lists them); nothing was uploaded. Don't pair `--files` with `--save`. | -| Exit `2` with "auth/setup" | Credentials missing or environment unset | `kane-cli whoami`; re-run `kane-cli login` if needed. | -| Turn ended with `generate_clarification` | Generator needs an answer — **this is exit 0, not a failure** | Read the question, decide on an answer (yourself or via the user), re-invoke `kane-cli generate "" --refine --req --agent`. | -| `generate_save_result` with `warning: "no functional test cases"` | The request has no Functional cases to save | Either refine to add Functional cases, or accept that the suite is non-functional (Security / Performance / etc.) and won't produce `_test.md` files. | -| Bare-objective shortcut: `kane-cli ""` exits `2` with "did you mean" | The `generate` subcommand is mandatory | Re-invoke as `kane-cli generate "" --agent`. Same rule for `run` / `testmd run`. | - ---- - -# Configuration surface (reference) - -Generate uses the same auth and project/folder configuration as `run` and `testmd`: - -```bash -kane-cli whoami -kane-cli config show -kane-cli config project # or the interactive picker in TTY (OAuth + basic both work) -kane-cli config folder # or the interactive picker in TTY -kane-cli projects list [--search ] [--limit ] [--offset ] --agent -kane-cli projects create "" [--description ""] --agent -kane-cli folders list --project [--search ] [--limit ] [--offset ] --agent # --project is required -kane-cli folders create "" --project [--description ""] --agent # --project is required -``` - -If nothing is configured, the run-startup gate auto-defaults a project/folder before `generate_start` and emits `project_folder_auto_defaulted`. Pre-configure only when the user wants generated cases filed in a specific place. See the **`kane-cli-run`** steering file for full project/folder mechanics. - ---- - -# Handoff to `kane-cli-testmd` - -Once `--save` has written `_test.md` files, every later interaction with those files is a `kane-cli testmd` workflow: - -- Run them: `kane-cli testmd run --agent` -- List them: `kane-cli testmd list` -- Inspect: `kane-cli testmd status ` -- Re-export code: `kane-cli testmd export --language python|javascript` - -**Switch to the `kane-cli-testmd` steering file** once the user moves from authoring to running. Generate's job ends at `--save`; testmd's job picks up from there. diff --git a/integrations/kiro-powers/steering/kane-cli-mobile.md b/integrations/kiro-powers/steering/kane-cli-mobile.md index 5ccb842..fa4cc4f 100644 --- a/integrations/kiro-powers/steering/kane-cli-mobile.md +++ b/integrations/kiro-powers/steering/kane-cli-mobile.md @@ -11,7 +11,7 @@ The single rule that scopes this file: **mobile is opt-in.** A run with no `--ta | Where the device runs | Host | Commands | |---|---|---| | **Local** — a simulator/emulator on this machine | **macOS on Apple Silicon (arm64) only.** Not Intel Macs, Linux, or Windows | `kane-cli run --target …`, `kane-cli testmd run`, `kane-cli testrun run` | -| **Cloud grid** — a virtual device on a LambdaTest HyperExecute macOS host | **Any machine**; no Xcode or Android Studio needed. The account needs a HyperExecute plan with macOS runners | `kane-cli testrun run --remote --device-name "" --os-version ` — see "Running a mobile suite on the cloud grid" below | +| **Cloud grid** — a virtual device on a LambdaTest HyperExecute macOS host | **Any machine**; no Xcode or Android Studio needed. The account needs a HyperExecute plan with macOS runners | `kane-cli testrun run --remote --device-name "" --os-version < /dev/null` — see "Running a mobile suite on the cloud grid" below | - **Desktop stays the default.** The `--target` flag is what selects mobile. Leave it off and every run drives the browser, unchanged. - If the user is not on a mac-arm64 machine, local mobile is not an option — **offer the grid**: save the objective as a `_test.md` (`target: emulator|simulator` + `app:`) and run it with `--remote`. Do not tell them mobile is unavailable. @@ -129,7 +129,7 @@ A `_test.md` selects its surface through the **`target:`** frontmatter key — * Author it once (real device, like any first run), then replay from cache. Everything else about the `_test.md` format is unchanged. Load the **`kane-cli-testmd`** steering file. Run a mobile test with `kane-cli testmd run --agent`. -**Mobile members in `kane-cli testrun`.** A mobile `_test.md` is a normal batch member (0.8.7+). Locally (mac-arm64 with the setup above): `kane-cli testrun run --device-name "" --os-version `. On the cloud grid, from any machine: add `--remote` (next section). Load the **`kane-cli-testrun`** steering file for the flags, events, and rollup. +**Mobile members in `kane-cli testrun`.** A mobile `_test.md` is a normal batch member (0.8.7+). Locally (mac-arm64 with the setup above): `kane-cli testrun run --device-name "" --os-version < /dev/null`. On the cloud grid, from any machine: add `--remote` (next section). Load the **`kane-cli-testrun`** steering file for the flags, events, and rollup. --- @@ -140,9 +140,9 @@ One command turns a mobile suite into a LambdaTest HyperExecute job on a macOS h ```bash kane-cli plugin install remote-execution # once; `kane-cli plugin doctor remote-execution` checks it kane-cli devices list --target emulator --remote --agent # grid catalog: name + os_versions per row -kane-cli testrun run tests/app/ --remote --device-name "Pixel 7" --os-version 14 --dry-run # validate + resolve device, no job -kane-cli testrun run tests/app/ --remote --device-name "Pixel 7" --os-version 14 -kane-cli testrun run tests/ios/ --remote --device-name "iPhone 15" --os-version 17.5 +kane-cli testrun run --match '^tests/app/' --remote --device-name "Pixel 7" --os-version 14 --dry-run < /dev/null # validate + resolve device, no job +kane-cli testrun run --match '^tests/app/' --remote --device-name "Pixel 7" --os-version 14 < /dev/null +kane-cli testrun run --match '^tests/ios/' --remote --device-name "iPhone 15" --os-version 17.5 < /dev/null ``` **Always `--dry-run` first** — it runs the remote preflight and resolves the device against the catalog at no cost. Allow several minutes for the real run. diff --git a/integrations/kiro-powers/steering/kane-cli-run.md b/integrations/kiro-powers/steering/kane-cli-run.md index 1361d83..cb2bfa9 100644 --- a/integrations/kiro-powers/steering/kane-cli-run.md +++ b/integrations/kiro-powers/steering/kane-cli-run.md @@ -20,7 +20,7 @@ When a dependency check fails, run the fix yourself. Only ask the user when the - Test / verify something → same, with assertion phrasing. - Extract data from a page → same, using the `store … as ''` pattern. - Save / re-run / commit a test → switch to `kane-cli testmd`. Load the **`kane-cli-testmd`** steering file. -- **Test cases or scenarios written** — because the user asked, or because the task needs them (no browser action) → **don't hand-draft them**; load the **`kane-cli-generate`** steering file and use `kane-cli generate`. Trigger phrases: "write tests for", "test cases for", "test suite for", "what edge cases", "generate tests for". +- **Test cases or scenarios written** — because the user asked, or because the task needs them (no browser action) → **don't hand-draft them**; load the **`kane-cli-assurance`** steering file: a requirement document is ingested and designed from, and a description given in chat is written to `requirements/.md` in the user's words, ingested, then designed from. Trigger phrases: "write tests for", "test cases for", "test suite for", "what edge cases", "generate tests for". - Browse / create a Test Manager project or folder, or interpret a `project_folder_auto_defaulted` event → use `kane-cli projects list|create` / `kane-cli folders list|create --project ` (NDJSON under `--agent`). The run-startup gate auto-defaults a project/folder when nothing is configured and emits `project_folder_auto_defaulted` before the first progress event. - The user wants results saved somewhere else ("change project") → follow **Changing where results go** below. The change is global, and the question must say so. - The user wants to change how runs behave ("kane preferences": watch or quiet, one-off or suite), or asks what Kane CLI can do or for the tour again ("kane tour") → load the **`kane-cli-first-run`** steering file. @@ -223,6 +223,15 @@ Loading order (later wins): **Always parameterize:** credentials, API keys, tokens, environment-specific URLs. **OK to hardcode:** one-off URLs, static UI text, navigation paths. +**Unresolved variables (0.8.15+).** kane-cli checks every `{{name}}` in the objective before the run starts. A name with no value is a warning: the run goes ahead and types the name as written, unless a step sets it first. With `--agent` the warning is one `{"type":"warning","code":"unresolved_variables", ...}` event before the first progress frame, carrying `suggested_file` and `variables[]` (`name`; `reason`: `value_missing` = the key exists in `file` with no value · `not_declared` = the key is in no file, add it to `suggested_file`; `used_by[]`). Never checked: an explicit `{{global.*}}` (it resolves from Test Manager at run time), `{{smart.*}}`, `{{environment.*}}`, `{{secrets.*}}`, `{{totp.*}}`, and names an earlier step stores (`store … as 'x'`). Numbers in a variable file count as values (loaded as strings); booleans do not. + +**Fill the variables before any run.** Do this before `run`, `testmd run` and `testrun run`, and always before `--remote`, which books a grid job. A missing value is asked for before the run; the first-session rule that nothing is asked before the first result does not cover it. + +1. Collect the names with no value: the `warning`, a dry-run plan's `unresolved[]` rows, or a design run's `variables_declared` and `variables_summary` rows. Design rows list what design declared, not every missing value, and a dry run's `valid: true` says nothing about values: the dry run's `warning` is the check. +2. A value you already have, because the user said it or the requirement document states it, you write into that key in the file the event names (for `run`, `--variables '{"name":{"value":"…"}}'` also works), and you tell the user what you filled and where. +3. The rest you ask for once, in one message, using each variable's `description` when design gave one. Plain values (a URL, an email, a user name) the user gives you here or adds to the file, their choice. Secrets, which are any row with `secret: true`, any description that names a credential, and any name containing `password`, `secret`, `token` or `key`, the user fills in the file and tells you when done; you never ask for the value in chat, never echo it, and report names and file paths only. +4. Then a fresh `--dry-run` of the exact selection: a valid plan with no `warning` is the check. A member whose rows are all filled may run while the others wait. Never fill a placeholder just to silence the check; a throwaway value is right only when the description asks for one, such as a deliberately wrong password. A frontmatter declaration with an empty value also silences the check, so look at the values a step relies on, not only at the warning. A name still empty is typed into the page as written; if a run then failed at that step, say so. + ## Context files The agent picks up two Markdown context files automatically: @@ -291,8 +300,9 @@ The same stream is also written to disk line by line, as `events.ndjson` in the | `bifurcation` | `flows[]`, `count` | Agent split the objective into sub-flows | | `child_agent_start` | `child_id`, `objective`, `parent_step` | Child agent spawned | | `child_agent_end` | `child_id`, `success`, `steps_taken`, `summary` | Child agent finished | -| `ask_user` | `question`, `step_index`, `options?` | Agent needs input (auto-disabled when stdin is non-TTY) | +| `ask_user` | `question`, `step_index`, `options?` | Agent needs input | | `error` | `message` | Error | +| `warning` | `code`, `message`, `suggested_file`, `variables[]` | *(0.8.15+)* Before any progress: `code: "unresolved_variables"` — a `{{name}}` had no value; the run continues and types the name as written unless a step sets it first. See Variables and secrets. | | `test_md_evidence_ingest` | `status: "ok"\|"failed"`, `evidence_id`, `stage?` | `testmd run` only: a replay's evidence pack published to the dashboard. Informational. | | `test_md_bundle_sync` | `status: "ok"\|"failed"`, `commit_id`, `bytes?`/`stage?` | `testmd run`/`testmd sync`: test bundle pushed to cloud after an authored commit. Informational. | | `testrun_*` family | see the **`kane-cli-testrun`** steering file | Only from `kane-cli testrun run`; its terminal event is `testrun_done`, not `run_end`. | @@ -301,7 +311,6 @@ The `run` stream has no `run_start` event. On kane-cli 0.8.17+ the first line is **The evidence hint is not an event.** After a run, Kane CLI prints `` evidence: view locally with `kane-cli evidence serve ` `` on **stderr**. Never look for it on stdout. -`ask_user` is auto-disabled when stdin is not a TTY. Since Kiro runs Kane CLI as a subprocess, `ask_user` events will not be emitted — write objectives that don't require interactive input. ## Parsing strategy @@ -412,6 +421,7 @@ Every result is an emoji table. Rules for every card: - **Never show internals:** no event names, no field names, no paths the user does not own. File names they own (`checkout_test.md`, `output-checkout/`) are fine. - **`🟡 Didn't start` is not `🔴 Failed`.** When nothing ran, say what to fix. - **Secret-looking values never go in chat.** For a missing value whose name contains `password`, `secret`, `token` or `key`, add an empty entry to the variables file for the user to fill. Ask in chat only for plain values (a URL, a user name). +- **Variables with no value go on the card.** When the run's output carried the `unresolved_variables` warning, add a `⚠️ **Variables**` row before ➡️ Next naming each one. - If the run's output carried an update notice, add one quiet last line under the card: `kane-cli is available.` **Successful run:** @@ -466,7 +476,7 @@ Then extract the failing-step screenshot from the run's evidence pack (`unzip

` starts a local-only ser | 🎯 Agent clicks the wrong element | Ambiguous UI, multiple similar elements | Be more specific ("click the **blue** Submit button in the **checkout form**") | | 👁️ Agent says "done" but nothing happened | Objective too vague | Add a concrete assertion ("assert the confirmation page shows an order number") | | 💀 Exit `2`, no steps | Auth, TMS credential exchange, or Chrome failure | Check `kane-cli whoami`; ensure Chrome is installed | -| ❓ Exit `2` with "did you mean …" | Missing `run` subcommand — agent invoked `kane-cli ""` instead of `kane-cli run ""` | Re-invoke with `run` (same rule for `testmd run` / `generate`) | +| ❓ Exit `2` with "did you mean …" | Missing `run` subcommand — agent invoked `kane-cli ""` instead of `kane-cli run ""` | Re-invoke with `run` (same rule for `testmd run`) | | 📤 Upload silently fails after setting a project/folder by hand | Saved ID is invalid (typo, deleted, no access) | No action needed — the next run detects the 4xx and auto-defaults a working project/folder. To rebind: `kane-cli config project` or `kane-cli projects list` → `kane-cli config project ` | | ⏱️ Exit `3` | Timeout or cancelled | Raise `--timeout`, raise `--max-steps`, or split the objective | | 🚫 `CDP endpoint not reachable` | Chrome not running | Drop `--cdp-endpoint` and let Kane CLI auto-launch Chrome | @@ -577,7 +587,7 @@ Offer the visual route: `kane-cli evidence serve ` starts a local-only ser For multiple independent browser tasks, decompose and run in parallel. -> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, don't hand-roll parallelism — `kane-cli testrun run --parallel N` gives isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Load the **`kane-cli-testrun`** steering file. This section is for **ad-hoc `run` objectives** only. +> **Saved tests? Use testrun instead.** If the tasks are committed `_test.md` files, don't hand-roll parallelism — `kane-cli testrun run --parallel N < /dev/null` gives isolated Chromes, a pooled scheduler, one rollup, and one evidence pack. Load the **`kane-cli-testrun`** steering file. This section is for **ad-hoc `run` objectives** only. ## When to split @@ -685,7 +695,7 @@ NDJSON wire shape: each result row is `{id, name}`, terminated by `{_meta: "page ## The run-startup auto-default gate -Every `run`, `testmd run`, and `generate` validates the cached project/folder before launching. Three outcomes: +Every `run` and `testmd run` validates the cached project/folder before launching. Three outcomes: 1. Cached project/folder still valid → run proceeds, no event. 2. Nothing configured **or** cached IDs are gone / invalid / inaccessible → Kane CLI auto-resolves (find-or-create) and emits `project_folder_auto_defaulted` on stdout before any progress event. Surface as a one-liner ("Kane CLI auto-selected project X / folder Y for this run"). @@ -724,7 +734,7 @@ Project-local overrides live in `./.testmuai/` (`context.md`, `variables/*.json` ## Command-specific completion -The `run_end` parsing strategy applies to one-shot `run` only. For `testmd run`, collect `test_md_done.overall_status`, `duration_s`, `session_id`, and optional `share_url`; embedded `run_end` events can finish individual steps. Local suites emit `testrun_done`; dispatched remote suites then emit `remote_done` (retain `status`, `exit`, `sessions_path`). `generate` emits `generate_done`. Assurance conversational agent streams end in `done`; review/read verbs have their own contracts. Always check process exit too: early refusal, invalid plan or dry-run can exit without the normal completion event. +The `run_end` parsing strategy applies to one-shot `run` only. For `testmd run`, collect `test_md_done.overall_status`, `duration_s`, `session_id`, and optional `share_url`; embedded `run_end` events can finish individual steps. Local suites emit `testrun_done`; dispatched remote suites then emit `remote_done` (retain `status`, `exit`, `sessions_path`). Assurance conversational agent streams end in `done`; review/read verbs have their own contracts. Always check process exit too: early refusal, invalid plan or dry-run can exit without the normal completion event. Progress is for live display: count only `done`/`failed` completions, retaining child and execution context when step indices repeat. diff --git a/integrations/kiro-powers/steering/kane-cli-testmd.md b/integrations/kiro-powers/steering/kane-cli-testmd.md index dc0a704..c236cb5 100644 --- a/integrations/kiro-powers/steering/kane-cli-testmd.md +++ b/integrations/kiro-powers/steering/kane-cli-testmd.md @@ -215,27 +215,11 @@ On exit, Kane CLI writes the session to `/.testmuai/tests/amazon-search_tes Slug rules: `[a-zA-Z0-9_-]+`. Pick a slug that reads as a filename — e.g. `login-smoke`, `checkout-happy-path`, `dashboard-regression`. -## Authoring `_test.md` from a description — the generate → testmd pipeline +## Authoring `_test.md` from a description — through assurance -When the user wants test cases written from scratch — from a feature spec, a requirement, or just a description of what to test — **don't hand-draft them and don't capture them via a live `--name` session.** Use `kane-cli generate` to author structured Functional cases first, save them as runnable `_test.md`, then run them here: +When the user wants test cases written from scratch — from a feature spec, a requirement, or just a description of what to test — **don't hand-draft them and don't capture them via a live `--name` session.** Every test case comes from the assurance commands: a requirement document is ingested; a description given in chat is written to `requirements/.md` in the user's words and ingested; then the extracted use-cases are reviewed, `kane-cli design tests …` designs them, the designed tests are reviewed, the variables design declared are filled (the `kane-cli-run` steering file, **Fill the variables before any run**), and `kane-cli testrun run … < /dev/null` authors and replays them. -```bash -# 1. Author scenarios + cases from a description -kane-cli generate "checkout flow on a shopping site" --agent -# → capture the request id (e.g. 23271) - -# 2. Refine if needed (repeat as often as needed) -kane-cli generate "also cover an expired card and an out-of-stock item" --refine --req 23271 --agent - -# 3. Save the Functional cases as runnable _test.md files -kane-cli generate --save --req 23271 --agent -# → /.testmuai/tests///_test.md - -# 4. Run them — back in testmd territory -kane-cli testmd run .testmuai/tests///_test.md --agent -``` - -Full mechanics (the three modes, clarification round-trips, presenting scenarios + cases, the typed event schema) live in the **`kane-cli-generate`** steering file. Load that file the moment the user asks for cases to be written; come back here to run them. +Load the **`kane-cli-assurance`** steering file the moment the user asks for cases to be written; come back here for single-file `testmd run` work. --- @@ -464,7 +448,7 @@ Why each flag: - `--on-lock-conflict wait` — block instead of failing if a teammate is editing the same test. - Adaptive healing recovers replay failures by default; `--no-adaptive-heal` opts out. -Two non-TTY behaviors to know (Kiro always runs Kane CLI non-TTY): interactive `ask_user` prompts are disabled, so a step that would wait for input fails cleanly instead of hanging; and the first step needs a resolvable start URL (the `url:` frontmatter key, the `--url` flag, or a site named in the step's prose) or the run fails — pass `--allow-missing-url` to start from the current page instead. +One non-TTY behavior to know (Kiro always runs Kane CLI non-TTY): the first step needs a resolvable start URL (the `url:` frontmatter key, the `--url` flag, or a site named in the step's prose) or the run fails — pass `--allow-missing-url` to start from the current page instead. Capturing the exit code in shell: diff --git a/integrations/kiro-powers/steering/kane-cli-testrun.md b/integrations/kiro-powers/steering/kane-cli-testrun.md index 8a2729d..8600d2a 100644 --- a/integrations/kiro-powers/steering/kane-cli-testrun.md +++ b/integrations/kiro-powers/steering/kane-cli-testrun.md @@ -10,19 +10,19 @@ Load this file when the user wants to run **several** saved `_test.md` tests as - One test → `kane-cli testmd run` (load `kane-cli-testmd.md`). - Multiple ad-hoc `run` objectives (not saved tests) → the parallel-execution section of `kane-cli-run.md` still applies. - **Mobile** `_test.md` members (`target: emulator|simulator`) are normal members (0.8.7+): locally on mac-arm64 with `--device-name`/`--os-version`; on the **cloud grid from any machine** with `--remote` (section below and `kane-cli-mobile.md`). -- The user wants the suite on the **cloud grid** (no local Chrome, or a mobile suite from a non-Mac) → `kane-cli testrun run … --remote`. +- The user wants the suite on the **cloud grid** (no local Chrome, or a mobile suite from a non-Mac) → `kane-cli testrun run … --remote < /dev/null`. # Command ```bash -kane-cli testrun run [paths...] [flags] # NDJSON is automatic when stdin is not a TTY (use < /dev/null in terminal automation) — there is NO --agent flag on testrun +kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON only when stdin is not a TTY: every line you write ends in < /dev/null — there is NO --agent flag on testrun ``` `[paths...]` is optional — omit it to auto-discover every `*_test.md` under the cwd. Explicit paths must end in `_test.md`. | Flag | Purpose | Default | |---|---|---| -| `--match ` | Filter candidates by project-relative path regex | — | +| `--match ` | Filter candidates by project-relative path regex | The path is as the OS writes it: `tests/app/` on macOS and Linux, `tests\app\` on Windows. Quote the regex with double quotes in cmd.exe; single quotes are literal there. | | `--tags ` | ANY-match on frontmatter `tags:` (repeatable or comma-separated, case-insensitive) | — | | `--parallel ` | Worker count; each desktop worker gets an isolated Chrome with a fresh temp profile | `1` | | `--on-failure ` | `continue` (run everything) \| `fail-fast` (stop dispatching new members after a failure) | `continue` | @@ -45,16 +45,16 @@ All members must share one org + project. *(0.8.4+)* Members need **not** be aut | `org_mismatch` | Different organisation than the other tests | "Check `kane-cli testmd status ` — it belongs to another org" | | `project_mismatch` | Different project than the other tests | "Run it separately or per-project" | -If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. Suggest `--dry-run` to preview the plan cheaply before a big run. +If **any** member fails preflight, the plan is invalid: nothing runs, exit `2`. Suggest `--dry-run` to preview the plan cheaply before a big run. *(0.8.15+)* Preflight also checks variables. A `{{name}}` with no value is a warning: the member stays in the plan (`valid` does not look at variables), and the run types the name as written unless a step sets it first. The check covers every step of the member, replayed ones included. `testrun run` has no `--variables` flag, so fill the pool file (`.testmuai/variables/*.json`) or the member's own `variables:` frontmatter first, per **Fill the variables before any run** in the `kane-cli-run` steering file. # Remote: the suite as one HyperExecute job (`--remote`) -`kane-cli testrun run --remote` ships the cwd as the job payload, provisions a grid runtime on a **HyperExecute macOS runner** — Chrome for web members, a **virtual Android emulator or iOS simulator** for mobile members — runs every member there as its own headless `testmd run`, and brings the recordings (`output-/`) and the sealed evidence pack back into the project. Works **from any machine** with nothing local but Node and the plugin (no Chrome needed); the account needs a HyperExecute plan with macOS runners and the plugin (`kane-cli plugin install remote-execution`; `kane-cli plugin doctor remote-execution` checks it). Auth is a LambdaTest username + access key — an OAuth profile is exchanged automatically. Not the same as `--ws-endpoint`, which attaches a remote browser to a run that still executes locally. +`kane-cli testrun run --remote < /dev/null` ships the cwd as the job payload, provisions a grid runtime on a **HyperExecute macOS runner** — Chrome for web members, a **virtual Android emulator or iOS simulator** for mobile members — runs every member there as its own headless `testmd run`, and brings the recordings (`output-/`) and the sealed evidence pack back into the project. Works **from any machine** with nothing local but Node and the plugin (no Chrome needed); the account needs a HyperExecute plan with macOS runners and the plugin (`kane-cli plugin install remote-execution`; `kane-cli plugin doctor remote-execution` checks it). Auth is a LambdaTest username + access key — an OAuth profile is exchanged automatically. Not the same as `--ws-endpoint`, which attaches a remote browser to a run that still executes locally. ```bash -kane-cli testrun run --tags smoke --remote --dry-run # validate + resolve, dispatch nothing -kane-cli testrun run tests/app/ --remote --device-name "Pixel 7" --os-version 14 # Android suite on the grid -kane-cli testrun run tests/ios/ --remote --device-name "iPhone 15" --os-version 17.5 # iOS suite on the grid +kane-cli testrun run --tags smoke --remote --dry-run < /dev/null # validate + resolve, dispatch nothing +kane-cli testrun run --match '^tests/app/' --remote --device-name "Pixel 7" --os-version 14 < /dev/null # Android suite on the grid +kane-cli testrun run --match '^tests/ios/' --remote --device-name "iPhone 15" --os-version 17.5 < /dev/null # iOS suite on the grid ``` - **Always `--dry-run` first** — it runs the remote preflight and resolves the device against the grid catalog (`kane-cli devices list --target emulator|simulator --remote --agent`) without creating a job. @@ -73,7 +73,7 @@ All typed; stdout; one JSON object per line. **Local completion: `testrun_done`; | `type` | Payload | Notes | |---|---|---| -| `testrun_plan` | `members: [{path, test_id?, tags, failure?}]`, `valid`, `parallel`, `parallel_clamped?` | If `valid: false`, treat as immediate failure — report each member's `failure` reason and expect exit `2`. | +| `testrun_plan` | `members: [{path, test_id?, tags, failure?, unresolved?}]`, `valid`, `parallel`, `parallel_clamped?` | If `valid: false`, treat as immediate failure — report each member's `failure` reason and expect exit `2`; a `warning` may still follow before the process exits. *(0.8.15+)* `unresolved[]` lists the member's `{{name}}`s with no value; `valid` does not look at it. One `warning` event with `code: "unresolved_variables"` follows the plan and the run goes ahead. | | `testrun_start` | `execution_id`, `members` (paths), `parallel` | | | `testrun_member_start` | `path`, `test_id?`, *(0.8.17+)* `session_id`, `log_path` | A saved test started. `log_path` is the absolute path of that test's own event log (see **Each test's own log** below). | | `testrun_member_end` | `path`, `test_id?`, `status`, `duration_s`, *(0.8.17+)* `session_id`, `log_path`, `failure?: {message, step_index?}` | `status` ∈ `passed \| failed \| broken \| interrupted`. `failure` is present when the test did not pass: use it for the "where" and "why" of the failed-tests table. | @@ -213,7 +213,7 @@ Local suites containing any mobile member require `--parallel 1`; larger values Healing is enabled by default (three shrinking replay windows, then re-authoring of authorable steps). `--no-adaptive-heal` disables it. Retired `--retry`/`--retry-count` only print a notice and have no effect. Replay-only recorded steps retain their recordings even during healing. -NDJSON selection uses stdin, not stdout: run `kane-cli testrun run < /dev/null` for automation launched from a terminal. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. +NDJSON selection uses stdin, not stdout: every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. ### Remote behavior still requiring verification