From 8d55a80e0b8081532f45d4be220377e8ee8874f5 Mon Sep 17 00:00:00 2001 From: Shravan Mahajan Date: Mon, 5 Oct 2026 18:34:12 +0530 Subject: [PATCH 1/2] docs + skill: maintain evolve --mode agent and testrun run --agent (kane-cli 0.8.21) maintain evolve can be driven headless from 0.8.21. With --mode agent the blast-radius confirmation becomes a question on the stream: evolve_target states the blast radius, evolve_paused carries the question, and the run ends paused with exit 3. The answer goes on the next call with --answer evolve:=1 (proceed) or =2 (decline). On proceed the design run's own events arrive between child_start and child_end, and evolve_diff follows a clean design run. The skill, its parsing reference, the maintain and automation guides and the Kiro power describe the stream, what a partial answer and a paused design session mean, and the two refusal codes. testrun run takes --agent from 0.8.21, selecting the same NDJSON stream a non-TTY stdin selects. The stdin redirect stays the documented rule because it works on every version. Co-Authored-By: Claude Fable 5.1 --- .agents/skills/kane-cli/SKILL.md | 4 ++-- .../kane-cli/references/assurance-parsing.md | 4 ++++ .agents/skills/kane-cli/references/assurance.md | 8 +++++--- .agents/skills/kane-cli/references/testrun.md | 4 ++-- .claude/skills/kane-cli/SKILL.md | 4 ++-- .../kane-cli/references/assurance-parsing.md | 4 ++++ .claude/skills/kane-cli/references/assurance.md | 8 +++++--- .claude/skills/kane-cli/references/testrun.md | 4 ++-- README.md | 2 +- docs/user-guide/assurance/automation.md | 16 ++++++++++++++++ docs/user-guide/assurance/maintain.md | 16 +++++++++++++++- docs/user-guide/testrun.md | 3 ++- integrations/kiro-powers/POWER.md | 2 +- .../kiro-powers/steering/kane-cli-assurance.md | 2 +- .../kiro-powers/steering/kane-cli-testrun.md | 2 +- skill-installer/skills/SKILL.md | 4 ++-- .../skills/references/assurance-parsing.md | 4 ++++ skill-installer/skills/references/assurance.md | 8 +++++--- skill-installer/skills/references/testrun.md | 4 ++-- 19 files changed, 76 insertions(+), 27 deletions(-) diff --git a/.agents/skills/kane-cli/SKILL.md b/.agents/skills/kane-cli/SKILL.md index 81c6e03..f76ed73 100644 --- a/.agents/skills/kane-cli/SKILL.md +++ b/.agents/skills/kane-cli/SKILL.md @@ -5,7 +5,7 @@ description: Browser automation + AI test authoring via kane-cli - run browser o # Kane CLI — Browser Automation Skill -Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. `testrun run` has no `--agent`: it emits NDJSON only when **stdin** is not a TTY, so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. +Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. On every version `testrun run` emits NDJSON when **stdin** is not a TTY (0.8.21+ also takes `--agent`), so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. **Authoring test cases or scenarios?** Never write them by hand: every test case comes from the **assurance** commands — Read `references/assurance.md` first. @@ -321,7 +321,7 @@ For one-shot `run`, build post-run logic on `run_end` and process exit. Saved te For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`. -The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. +The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `maintain evolve` on 0.8.21+, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. `kane-cli testrun run` also emits its own typed stream (`testrun_plan` … terminal `testrun_done`) — schema in `references/testrun.md`. `kane-cli testmd run` may additionally emit `test_md_evidence_ingest` (replay evidence published) and `test_md_bundle_sync` (test bundle synced) — informational; describe in plain language, never surface raw names. The post-run evidence hint (`` evidence: view locally with `kane-cli evidence serve ` ``) is a **stderr** text line, not a stdout event — don't try to parse it from the NDJSON stream; see `references/evidence.md` for how to act on it. diff --git a/.agents/skills/kane-cli/references/assurance-parsing.md b/.agents/skills/kane-cli/references/assurance-parsing.md index 9562a82..d2d71a6 100644 --- a/.agents/skills/kane-cli/references/assurance-parsing.md +++ b/.agents/skills/kane-cli/references/assurance-parsing.md @@ -81,6 +81,10 @@ Use `text` + `options[].label` + `recommended_index` + `rationale` to decide or *(0.7.2+)* the stream opens with a minimal `run_start` (`session` only — no `trace`) and the re-extract child rides the SAME stream: extract-vocabulary events (`source_start`, `agent_activity`, `plan`, `commit`, …) interleave between the `reconcile_*` events, all stamped `verb: "reconcile"` — dispatch on `type`, never on position. The reconcile family: `reconcile_plan` `{source_id, plan_path, rows[], archive[]}` → per row `reconcile_row_start` `{kind, ref, stale?, direct?}` + `reconcile_row_end` `{kind, ref, outcome: applied|failed|skipped|plan-only|paused, exit_code?, detail?}` → `reconcile_paused` `{plan_path, pending[]}` when ARCHIVE rows remain (exit 3; a human resumes with `--apply ` in a terminal) → `reconcile_summary` `{applied, skipped, deferred, plan_only, failed, paused, stale_created}` on every path → `done` always last. Validation refusals ride the stream as `error` + `done` (exit 2), never stderr alone. +## Evolve events (verb `evolve`) *(0.8.21+)* + +`kane-cli maintain evolve --mode agent` (or `--from-stale`) shares the envelope with `verb: "evolve"`. `evolve_batch` `{stale_nodes, use_cases}` opens a `--from-stale` run. `evolve_target` `{ref?, use_case, pairs, because?}` states the blast radius once per designed use-case, before any question. `evolve_paused` `{pending_questions, resume, next[]}` ends the run at exit `3` while a question has no answer: question ids are `evolve:`, options `proceed` / `decline`; answer on the next call with `--answer evolve:=1` (or `=proceed`) or `--answer evolve:=2` (or `=decline`), which needs `--mode agent`. After a proceed: `child_start` `{child: "design", subject}`, the design run's own events stamped `verb: "design"` (every event of `design tests --mode agent` except its `done`; a paused design session arrives as `session_paused` with `sid`, `resume` and `next[]`), `child_end` `{ok, dur_s, exit_code?}`. After a clean design run: `evolve_diff` `{use_case, generation: {before, after}, superseded[], minted[], retained[], moves[]}`, pairs as `{scenario, test}` and moves as `{ac, before[], after[]}`. `evolve_skipped` `{use_case, reason: "declined" | "no-design"}` marks a use-case not evolved; `no-design` appears in `--from-stale` batches only, and an explicit never-designed target refuses with exit `2`. `error` `{message, code?}`: `MODE_USAGE` (only `interactive` and `agent` are accepted), `ANSWER_USAGE` (an answer to a question the run does not ask, even when nothing is stale, or a value that is neither option); other refusals carry the message only, and a refused answer is `error` + `done` with nothing before it. `done` `{status, exit_code, next?}`: `complete` 0, `refused` 2, `paused` 3 (evolve's question or the design session), `error` with the design run's own code; `next[]` carries every paused session's resume, uncapped, whatever the status. A `--from-stale` batch runs every use-case whatever the earlier ones did: the first child code that is neither 0 nor 3 names the exit, else 3 when any child paused, else 0. + ## Sync events (verb `sync`) *(0.8.14+)* `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` under `--mode agent` (and `kane-cli context sync add`, `kane-cli context sync list`, `kane-cli context sync remove`, `kane-cli context sync status`, `kane-cli context sync doctor`) share the envelope with `verb: "sync"` and their own `sync_*` family — `sync_probe_started`/`sync_probe`, `sync_status`, `sync_pull_done`/`sync_push_done`, `sync_rebase_started`/`sync_rebase_decision`/`sync_rebase_done`/`sync_rebase_open`, `sync_error{code, detail, remedy}` — then `done`. Two of them cross into the assurance streams: `sync_behind` (an advisory on `extract`/`design`/`reconcile` streams: the location has records this store has not pulled — one line, not a failure) and `sync_error` code `SYNC_REBASE_PENDING` (the fence: a sync rebase is open, so the write verb refused; `verb` carries that verb's own name). Schema, refusal codes and the `--answer` loop: `references/context-sync.md`. diff --git a/.agents/skills/kane-cli/references/assurance.md b/.agents/skills/kane-cli/references/assurance.md index 05b188a..47991ed 100644 --- a/.agents/skills/kane-cli/references/assurance.md +++ b/.agents/skills/kane-cli/references/assurance.md @@ -28,13 +28,13 @@ On 0.7.1+, `context ingest --mode agent` lands the files and runs the extraction Two hard rules: 1. **The two checkpoints are the user's, not yours.** Everything the extract and design agents emit is *derived* (unreviewed) until a human decision promotes it — see §4. Do not start authoring (step 5) until the user has approved the design output or explicitly told you to proceed. -2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). +2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`, `evolve`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). Extract, design, and reconcile call the KaneAI service and consume credits; everything else is local and free. ## 2. The pause loop — exit 3 is a pause, NOT a failure -**Scope: this rule applies to `context extract`/`context ingest`, `design tests`, and `maintain reconcile` (§11).** (For `run`/`testmd`/`testrun`, exit 3 still means timeout/cancelled.) +**Scope: this rule applies to `context extract`/`context ingest`, `design tests`, `maintain reconcile` (§11) and, on 0.8.21+, `maintain evolve`.** (For `run`/`testmd`/`testrun`, exit 3 still means timeout/cancelled.) These commands take **`--mode agent`** — not `--agent`; they reject that flag, and a bare non-TTY invocation exits `2` asking for an explicit mode. In `--mode agent`, **every question pauses the run** (0.8.8+ — earlier CLIs auto-answered low/medium-risk questions with their recommended defaults and paused only on high risk): @@ -227,7 +227,9 @@ Never `context ingest` the new version first — reconcile does its own re-inges - **`--from` takes URLs.** The same remote URLs ingest takes (§3): Jira/Confluence (0.7.2+), web pages (0.8.3+), Linear issues and documents (0.8.6+). `--source-id` is then optional — the URL carries its own id, and a contradicting `--source-id` refuses. Kind continuity holds both ways: a URL can't version a file-backed source and a file can't version a remote one — each refusal names the correct `--from`. A source ingested under a custom id (`--as`) is maintained by re-running that ingest with the same `--as`. The first reconcile of a Jira issue last ingested pre-0.7.2 may report the one-time upgrade re-version (§3) — not a content edit. - **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` — parse per `references/assurance-parsing.md`. -For staleness that arrived outside a reconcile, `kane-cli maintain evolve` re-designs a use-case — it is interactive-only (the blast-radius confirmation is the point); suggest the user run it in a terminal rather than scripting around it. +For staleness that arrived outside a reconcile, `kane-cli maintain evolve ` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). A fresh target refuses with exit `2` unless `--because ""` gives one, and a use-case that was never designed refuses too (`design tests` first). In a terminal evolve states the blast radius and asks. + +*(0.8.21+)* `kane-cli maintain evolve --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:"`, options `proceed` / `decline`), `resume` and `next[]`, then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:=1` (or `=proceed`) or `--answer evolve:=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and `--because` rides the resume. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined or never-designed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. ## Sharing a context store diff --git a/.agents/skills/kane-cli/references/testrun.md b/.agents/skills/kane-cli/references/testrun.md index 4545deb..59697bd 100644 --- a/.agents/skills/kane-cli/references/testrun.md +++ b/.agents/skills/kane-cli/references/testrun.md @@ -15,7 +15,7 @@ ## Command ```bash -kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON only when stdin is not a TTY: every line you write ends in < /dev/null — there is NO --agent flag on testrun +kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not a TTY, on every version: end every line in < /dev/null (0.8.21+ also takes --agent) ``` `[paths...]` is optional — omit it to auto-discover every `*_test.md` under the cwd. Explicit paths must end in `_test.md`. @@ -165,7 +165,7 @@ Local suites containing any mobile member require `--parallel 1`; larger values Healing is enabled by default (three shrinking replay windows, then re-authoring of authorable steps). `--no-adaptive-heal` disables it. Retired `--retry`/`--retry-count` only print a notice and have no effect. Replay-only recorded steps retain their recordings even during healing. -NDJSON selection uses stdin, not stdout: every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. +NDJSON selection uses stdin, not stdout (on 0.8.21+ `--agent` selects the same stream): every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. ### Remote behavior still requiring verification diff --git a/.claude/skills/kane-cli/SKILL.md b/.claude/skills/kane-cli/SKILL.md index 81c6e03..f76ed73 100644 --- a/.claude/skills/kane-cli/SKILL.md +++ b/.claude/skills/kane-cli/SKILL.md @@ -5,7 +5,7 @@ description: Browser automation + AI test authoring via kane-cli - run browser o # Kane CLI — Browser Automation Skill -Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. `testrun run` has no `--agent`: it emits NDJSON only when **stdin** is not a TTY, so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. +Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run` and `testmd run`. On every version `testrun run` emits NDJSON when **stdin** is not a TTY (0.8.21+ also takes `--agent`), so every `testrun run` line you write ends in `< /dev/null` (bash and zsh: macOS, Linux, Git Bash; in cmd.exe write `< NUL`; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it shows the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own; stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Assurance conversational commands use `--mode agent`. **Authoring test cases or scenarios?** Never write them by hand: every test case comes from the **assurance** commands — Read `references/assurance.md` first. @@ -321,7 +321,7 @@ For one-shot `run`, build post-run logic on `run_end` and process exit. Saved te For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`. -The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. +The assurance conversational commands (`context ingest`/`context extract`, `design tests`, `maintain reconcile`, `maintain evolve` on 0.8.21+, `cover`) do NOT take `--agent` — they take **`--mode agent`** and speak their own typed stream ending in `done` (open vocabulary — tolerate unknown event types; on 0.7.2+ the stream is strict — every stdout line parses, stderr silent — while on 0.7.1 a merged `context ingest` prints a few prose receipt lines BEFORE the stream — skip to the first `{` line, harmless on 0.7.2+ — and a landing-phase ingest failure ends with prose + exit 1/2 and no stream at all: a refusal, not a crash; on 0.7.2+ those failures ride the stream as `error` + `done`); **for those commands only, exit `3` means paused-and-resumable, not timeout** — schema in `references/assurance-parsing.md`, behavior in `references/assurance.md`. The context sync verbs (`kane-cli context sync`, `kane-cli context push`, `kane-cli context pull`, `kane-cli context clone`; kane-cli 0.8.14+) take `--mode agent` the same way and speak a `sync_*` family ending in `done`; there too exit `3` means a decision or a pull is needed, not a failure — `references/context-sync.md`. `kane-cli context sync setup` refuses without a terminal (`TTY_REQUIRED`): agents use `kane-cli context sync add` and `kane-cli context clone`. `kane-cli testrun run` also emits its own typed stream (`testrun_plan` … terminal `testrun_done`) — schema in `references/testrun.md`. `kane-cli testmd run` may additionally emit `test_md_evidence_ingest` (replay evidence published) and `test_md_bundle_sync` (test bundle synced) — informational; describe in plain language, never surface raw names. The post-run evidence hint (`` evidence: view locally with `kane-cli evidence serve ` ``) is a **stderr** text line, not a stdout event — don't try to parse it from the NDJSON stream; see `references/evidence.md` for how to act on it. diff --git a/.claude/skills/kane-cli/references/assurance-parsing.md b/.claude/skills/kane-cli/references/assurance-parsing.md index 9562a82..d2d71a6 100644 --- a/.claude/skills/kane-cli/references/assurance-parsing.md +++ b/.claude/skills/kane-cli/references/assurance-parsing.md @@ -81,6 +81,10 @@ Use `text` + `options[].label` + `recommended_index` + `rationale` to decide or *(0.7.2+)* the stream opens with a minimal `run_start` (`session` only — no `trace`) and the re-extract child rides the SAME stream: extract-vocabulary events (`source_start`, `agent_activity`, `plan`, `commit`, …) interleave between the `reconcile_*` events, all stamped `verb: "reconcile"` — dispatch on `type`, never on position. The reconcile family: `reconcile_plan` `{source_id, plan_path, rows[], archive[]}` → per row `reconcile_row_start` `{kind, ref, stale?, direct?}` + `reconcile_row_end` `{kind, ref, outcome: applied|failed|skipped|plan-only|paused, exit_code?, detail?}` → `reconcile_paused` `{plan_path, pending[]}` when ARCHIVE rows remain (exit 3; a human resumes with `--apply ` in a terminal) → `reconcile_summary` `{applied, skipped, deferred, plan_only, failed, paused, stale_created}` on every path → `done` always last. Validation refusals ride the stream as `error` + `done` (exit 2), never stderr alone. +## Evolve events (verb `evolve`) *(0.8.21+)* + +`kane-cli maintain evolve --mode agent` (or `--from-stale`) shares the envelope with `verb: "evolve"`. `evolve_batch` `{stale_nodes, use_cases}` opens a `--from-stale` run. `evolve_target` `{ref?, use_case, pairs, because?}` states the blast radius once per designed use-case, before any question. `evolve_paused` `{pending_questions, resume, next[]}` ends the run at exit `3` while a question has no answer: question ids are `evolve:`, options `proceed` / `decline`; answer on the next call with `--answer evolve:=1` (or `=proceed`) or `--answer evolve:=2` (or `=decline`), which needs `--mode agent`. After a proceed: `child_start` `{child: "design", subject}`, the design run's own events stamped `verb: "design"` (every event of `design tests --mode agent` except its `done`; a paused design session arrives as `session_paused` with `sid`, `resume` and `next[]`), `child_end` `{ok, dur_s, exit_code?}`. After a clean design run: `evolve_diff` `{use_case, generation: {before, after}, superseded[], minted[], retained[], moves[]}`, pairs as `{scenario, test}` and moves as `{ac, before[], after[]}`. `evolve_skipped` `{use_case, reason: "declined" | "no-design"}` marks a use-case not evolved; `no-design` appears in `--from-stale` batches only, and an explicit never-designed target refuses with exit `2`. `error` `{message, code?}`: `MODE_USAGE` (only `interactive` and `agent` are accepted), `ANSWER_USAGE` (an answer to a question the run does not ask, even when nothing is stale, or a value that is neither option); other refusals carry the message only, and a refused answer is `error` + `done` with nothing before it. `done` `{status, exit_code, next?}`: `complete` 0, `refused` 2, `paused` 3 (evolve's question or the design session), `error` with the design run's own code; `next[]` carries every paused session's resume, uncapped, whatever the status. A `--from-stale` batch runs every use-case whatever the earlier ones did: the first child code that is neither 0 nor 3 names the exit, else 3 when any child paused, else 0. + ## Sync events (verb `sync`) *(0.8.14+)* `kane-cli context sync`, `kane-cli context push`, `kane-cli context pull` and `kane-cli context clone` under `--mode agent` (and `kane-cli context sync add`, `kane-cli context sync list`, `kane-cli context sync remove`, `kane-cli context sync status`, `kane-cli context sync doctor`) share the envelope with `verb: "sync"` and their own `sync_*` family — `sync_probe_started`/`sync_probe`, `sync_status`, `sync_pull_done`/`sync_push_done`, `sync_rebase_started`/`sync_rebase_decision`/`sync_rebase_done`/`sync_rebase_open`, `sync_error{code, detail, remedy}` — then `done`. Two of them cross into the assurance streams: `sync_behind` (an advisory on `extract`/`design`/`reconcile` streams: the location has records this store has not pulled — one line, not a failure) and `sync_error` code `SYNC_REBASE_PENDING` (the fence: a sync rebase is open, so the write verb refused; `verb` carries that verb's own name). Schema, refusal codes and the `--answer` loop: `references/context-sync.md`. diff --git a/.claude/skills/kane-cli/references/assurance.md b/.claude/skills/kane-cli/references/assurance.md index 05b188a..47991ed 100644 --- a/.claude/skills/kane-cli/references/assurance.md +++ b/.claude/skills/kane-cli/references/assurance.md @@ -28,13 +28,13 @@ On 0.7.1+, `context ingest --mode agent` lands the files and runs the extraction Two hard rules: 1. **The two checkpoints are the user's, not yours.** Everything the extract and design agents emit is *derived* (unreviewed) until a human decision promotes it — see §4. Do not start authoring (step 5) until the user has approved the design output or explicitly told you to proceed. -2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). +2. **One step at a time, one store-writer at a time.** Never run two store-mutating commands concurrently (`ingest`, `extract`, `review` with verdicts, `design`, `reconcile`, `evolve`) — the store is single-writer (§8), and extraction additionally holds a store-wide lock (a concurrent extract refuses `EXTRACT_LOCKED`, exit 2 — wait, never delete locks). Extract, design, and reconcile call the KaneAI service and consume credits; everything else is local and free. ## 2. The pause loop — exit 3 is a pause, NOT a failure -**Scope: this rule applies to `context extract`/`context ingest`, `design tests`, and `maintain reconcile` (§11).** (For `run`/`testmd`/`testrun`, exit 3 still means timeout/cancelled.) +**Scope: this rule applies to `context extract`/`context ingest`, `design tests`, `maintain reconcile` (§11) and, on 0.8.21+, `maintain evolve`.** (For `run`/`testmd`/`testrun`, exit 3 still means timeout/cancelled.) These commands take **`--mode agent`** — not `--agent`; they reject that flag, and a bare non-TTY invocation exits `2` asking for an explicit mode. In `--mode agent`, **every question pauses the run** (0.8.8+ — earlier CLIs auto-answered low/medium-risk questions with their recommended defaults and paused only on high risk): @@ -227,7 +227,9 @@ Never `context ingest` the new version first — reconcile does its own re-inges - **`--from` takes URLs.** The same remote URLs ingest takes (§3): Jira/Confluence (0.7.2+), web pages (0.8.3+), Linear issues and documents (0.8.6+). `--source-id` is then optional — the URL carries its own id, and a contradicting `--source-id` refuses. Kind continuity holds both ways: a URL can't version a file-backed source and a file can't version a remote one — each refusal names the correct `--from`. A source ingested under a custom id (`--as`) is maintained by re-running that ingest with the same `--as`. The first reconcile of a Jira issue last ingested pre-0.7.2 may report the one-time upgrade re-version (§3) — not a content edit. - **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` — parse per `references/assurance-parsing.md`. -For staleness that arrived outside a reconcile, `kane-cli maintain evolve` re-designs a use-case — it is interactive-only (the blast-radius confirmation is the point); suggest the user run it in a terminal rather than scripting around it. +For staleness that arrived outside a reconcile, `kane-cli maintain evolve ` re-designs the parent use-case of the entity you name (`--from-stale` takes every use-case with stale designs). A fresh target refuses with exit `2` unless `--because ""` gives one, and a use-case that was never designed refuses too (`design tests` first). In a terminal evolve states the blast radius and asks. + +*(0.8.21+)* `kane-cli maintain evolve --mode agent` puts that question on the stream. A call on a stale target with no answer ends paused, exit `3`: `evolve_target` carries the blast radius (`use_case`, the `pairs` it affects, `because?`), then `evolve_paused` carries `pending_questions` (one per use-case, `id: "evolve:"`, options `proceed` / `decline`), `resume` and `next[]`, then `done{paused}`. Present the blast radius with the question, as with any pause (the pause loop above). Answer on the next call with `--answer evolve:=1` (or `=proceed`) or `--answer evolve:=2` (or `=decline`), repeatable; `--answer` needs `--mode agent`, and `--because` rides the resume. Nothing runs until every question is answered. A partial set pauses again on the rest, and the `next` commands then carry the answers already given, so a "decline the rest" command still re-designs what was answered proceed: read a `next` command before running it. On proceed, the design run's own events arrive between `child_start` and `child_end` stamped `verb: "design"`. After a clean design run comes `evolve_diff` (superseded, minted, retained, criteria whose tests moved). A design session that pauses arrives as its own `session_paused`: resume that session, then run evolve again for the diff. `done.next[]` carries every paused session's resume, all of them, whatever the exit code. In a `--from-stale` batch every use-case runs whatever the earlier ones did, and a declined or never-designed one is reported as `evolve_skipped`. Schema: `references/assurance-parsing.md`. On a CLI before 0.8.21 evolve has no `--mode`: suggest the user run it in a terminal rather than scripting around it. ## Sharing a context store diff --git a/.claude/skills/kane-cli/references/testrun.md b/.claude/skills/kane-cli/references/testrun.md index 4545deb..59697bd 100644 --- a/.claude/skills/kane-cli/references/testrun.md +++ b/.claude/skills/kane-cli/references/testrun.md @@ -15,7 +15,7 @@ ## Command ```bash -kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON only when stdin is not a TTY: every line you write ends in < /dev/null — there is NO --agent flag on testrun +kane-cli testrun run [paths...] [flags] < /dev/null # NDJSON when stdin is not a TTY, on every version: end every line in < /dev/null (0.8.21+ also takes --agent) ``` `[paths...]` is optional — omit it to auto-discover every `*_test.md` under the cwd. Explicit paths must end in `_test.md`. @@ -165,7 +165,7 @@ Local suites containing any mobile member require `--parallel 1`; larger values Healing is enabled by default (three shrinking replay windows, then re-authoring of authorable steps). `--no-adaptive-heal` disables it. Retired `--retry`/`--retry-count` only print a notice and have no effect. Replay-only recorded steps retain their recordings even during healing. -NDJSON selection uses stdin, not stdout: every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. +NDJSON selection uses stdin, not stdout (on 0.8.21+ `--agent` selects the same stream): every `kane-cli testrun run` line ends in `< /dev/null` (bash and zsh on macOS, Linux and Git Bash; `< NUL` in cmd.exe; from PowerShell run it through cmd: `cmd /c "kane-cli testrun run … < NUL"`). Check the first stdout line: on 0.8.17+ it is `{"type":"stream_start"…}`. A prose plan there instead means the CLI is in terminal mode: it prints the human view and, after a real run, opens an evidence table that waits until `q` or Esc, so the process never exits on its own. Stop it and rerun with the redirect. An empty stdout with a message on stderr is a usage error: read it. Dry-run validates a plan, not runtime authentication or browser/device readiness. Always observe process exit, including paths without a normal completion event. ### Remote behavior still requiring verification diff --git a/README.md b/README.md index 225ddfd..b88bb1b 100644 --- a/README.md +++ b/README.md @@ -201,7 +201,7 @@ kane-cli run " and …'>" --ag Three rules: -1. Use `--agent` for `run` and `testmd run`. `testrun run` has no `--agent`: it emits NDJSON only when **stdin** is not a TTY, so every `testrun run` line an agent writes ends in `< /dev/null` (`< NUL` in cmd.exe; from PowerShell, `cmd /c "… < NUL"`); the first stdout line is `stream_start` (0.8.17+); a prose plan there means terminal mode, which after a real run waits on an evidence table until `q` or Esc. Assurance conversational commands use `--mode agent`. Parse command-specific completion events and process exit. +1. Use `--agent` for `run` and `testmd run`. On every version `testrun run` emits NDJSON when **stdin** is not a TTY (0.8.21+ also takes `--agent`), so every `testrun run` line an agent writes ends in `< /dev/null` (`< NUL` in cmd.exe; from PowerShell, `cmd /c "… < NUL"`); the first stdout line is `stream_start` (0.8.17+); a prose plan there means terminal mode, which after a real run waits on an evidence table until `q` or Esc. Assurance conversational commands use `--mode agent`. Parse command-specific completion events and process exit. 2. **Use the "store as" pattern** for extraction. `"go to example.com, store the page title as 'page_title'"` — never `"read the page title"`. 3. **One objective = one task.** If a flow has more than ~15 steps, split it into multiple `kane-cli run` calls and run them in parallel. diff --git a/docs/user-guide/assurance/automation.md b/docs/user-guide/assurance/automation.md index 9c7bd48..3759e1a 100644 --- a/docs/user-guide/assurance/automation.md +++ b/docs/user-guide/assurance/automation.md @@ -96,6 +96,22 @@ Two more parsing rules: Validation failures (bad inputs, unknown source, the fork guard) ride the stream as `error` + `done` with exit `2` — never stderr alone. +### Evolve's stream + +`maintain evolve --mode agent` *(0.8.21)* speaks the same envelope with `verb: "evolve"`. The first call states the blast radius and pauses on a question; the next call answers it with `--answer evolve:=1` (proceed) or `--answer evolve:=2` (decline); the words `proceed` and `decline` work too. + +| type | payload highlights | +|---|---| +| `evolve_batch` | `--from-stale` only, first: `stale_nodes`, `use_cases` | +| `evolve_target` | once per designed use-case, before any question: `ref?`, `use_case`, `pairs` (the blast radius), `because?` | +| `evolve_paused` | a question has no answer yet; the run ends here with exit `3`: `pending_questions[]` (id `evolve:`, options `proceed` / `decline`), `resume`, `next[]` | +| `evolve_skipped` | a use-case not evolved: `use_case`, `reason` (`declined` \| `no-design`). `no-design` appears in `--from-stale` batches only; an explicit target that was never designed refuses with exit `2` | +| `child_start` / `child_end` | around the design run: `child: "design"`, `subject` / `ok`, `dur_s`, `exit_code?` | +| *(the design run's events)* | between the brackets, stamped `verb: "design"`: every event of `design tests --mode agent` except its `done`; a paused design session arrives as `session_paused` | +| `evolve_diff` | after a clean design run: `use_case`, `generation` (`before`, `after`), `superseded[]`, `minted[]`, `retained[]` (pairs as `scenario` + `test`), `moves[]` (`ac`, `before[]`, `after[]`) | +| `error` | a refusal or failure: `message`, `code?` (`MODE_USAGE`, `ANSWER_USAGE`) | +| `done` | always last: `complete` (0), `refused` (2), `paused` (3, evolve's question or the design session), `error` with the design run's own code; `next[]` carries every paused session's resume, all of them, whatever the status | + ## The pause → answer → resume loop This is the heart of driving assurance from an agent. A real exchange (events abridged, payloads shortened): diff --git a/docs/user-guide/assurance/maintain.md b/docs/user-guide/assurance/maintain.md index 5f3c434..b856b67 100644 --- a/docs/user-guide/assurance/maintain.md +++ b/docs/user-guide/assurance/maintain.md @@ -9,6 +9,7 @@ kane-cli maintain reconcile --apply [path] # continue a kane-cli maintain reconcile --from --source-id --mode agent # headless — see Automation kane-cli maintain evolve [--because ""] # re-design one stale use-case (interactive) kane-cli maintain evolve --from-stale # …or every use-case with stale designs +kane-cli maintain evolve --mode agent # headless (0.8.21): the blast-radius question pauses the run ``` @@ -139,7 +140,7 @@ kane-cli maintain evolve [--because ""] # any designed entity kane-cli maintain evolve --from-stale # every use-case with stale designed entities ``` -Evolve re-designs the **parent use-case** of whatever you point it at — a test, scenario, criterion, or the use-case itself. It is interactive-only, and the blast radius is always stated before anything runs; declining is a clean exit. +Evolve re-designs the **parent use-case** of whatever you point it at — a test, scenario, criterion, or the use-case itself. The blast radius is always stated before anything runs; declining is a clean exit. - **Staleness-gated:** a fresh target refuses. `--because ""` is the sanctioned override — your reason becomes the change context the re-design sees, on the record. - `--from-stale` collects every use-case with stale designed entities and walks them one confirm at a time. @@ -147,6 +148,19 @@ Evolve re-designs the **parent use-case** of whatever you point it at — a test - After a clean run, evolve reports the diff between the two design generations — what was superseded, what was minted, what was **retained** unchanged, and which criteria's verifying tests moved. A re-design doesn't break what it didn't change. - Reconcile's MODIFY rows route here automatically — reach for evolve directly when staleness arrived outside a reconcile (an older change, a retired source). [`kane-cli cover gaps`](./coverage.md) lists stale designed entities in its ranked worklist. +### Headless: `--mode agent` *(0.8.21)* + +`--mode agent` speaks the same typed stream as `design tests` and `reconcile`: one JSON object per line on stdout, `verb: "evolve"`, nothing on stderr, `done` last. Agent mode never answers for you. The blast-radius confirmation becomes a question, the run pauses with exit `3`, and the answer arrives on the next call: + +```bash +kane-cli maintain evolve t-4 --mode agent +# evolve_target (the blast radius), evolve_paused (question id evolve:uc-3, options proceed / decline), done paused, exit 3 +kane-cli maintain evolve t-4 --mode agent --answer evolve:uc-3=1 # 1 = proceed, 2 = decline +# evolve_target, child_start, the design run's own events (verb: design), child_end, evolve_diff, done complete, exit 0 +``` + +`--because` rides the resume command. With `--from-stale` there is one question per use-case and nothing runs until every one is answered; a partial set pauses again on the rest and keeps the answers given. A design session that pauses arrives as its own `session_paused`, and `done` carries its resume command: resume it, then run evolve again for the diff. After a partial answer the suggested commands carry the answers already given, so declining the rest still re-designs what you answered proceed. `--mode` accepts `interactive` and `agent` only (`MODE_USAGE` otherwise); an answer to a question the run does not ask refuses `ANSWER_USAGE`. The event table is in [Automation](./automation.md#evolves-stream). + ## Exit codes | Code | Meaning | diff --git a/docs/user-guide/testrun.md b/docs/user-guide/testrun.md index 56f05ba..e2415ee 100644 --- a/docs/user-guide/testrun.md +++ b/docs/user-guide/testrun.md @@ -60,7 +60,7 @@ warning: 2 variables have no value A step that sets a name first binds it; anything else is typed as written. Fill the values to bind them. ``` -`testrun run` has no `--variables` flag: fill the pool file or the member's own `variables:` frontmatter. In agent mode (stdin not a TTY) the same content arrives as one `warning` event with `code: "unresolved_variables"` right after `testrun_plan`, whose members carry their rows — see [Running tests](./running-tests.md#unresolved-variables) for the shape. +`testrun run` has no `--variables` flag: fill the pool file or the member's own `variables:` frontmatter. In agent mode (stdin not a TTY, or `--agent` on 0.8.21+) the same content arrives as one `warning` event with `code: "unresolved_variables"` right after `testrun_plan`, whose members carry their rows — see [Running tests](./running-tests.md#unresolved-variables) for the shape. ## Mobile members @@ -84,6 +84,7 @@ kane-cli testrun run --match '^tests/app/' --remote --device-name "Pixel 7" --os | `--on-failure ` | `continue` \| `fail-fast` | `continue` | | `--name