skill: triage a run from its evidence pack

.agents/skills/kane-cli/SKILL.md

@@ -136,7 +136,7 @@ For exit code 1 (or `status: "failed"` in `run_end`), present the failure card.
 | ➡️ **Next** | <a retry you can run now> · <walk through the failing step> |
 ```
 
-The failing step's screenshot lives inside the run's evidence pack (the stderr hint names the pack path): extract it with `unzip <pack> "tests/*/steps/*/screenshot.png" -d <tmpdir>`, Read it, and show it **under** the card. For the pack layout and deeper diagnosis, see `references/debug.md`.
+The failing step's screenshot lives inside the run's evidence pack (the stderr hint names the pack path): extract it with `unzip <pack> "tests/*/steps/*/screenshot.*" -d <tmpdir>`, Read it, and show it **under** the card. For the pack layout and deeper diagnosis, see `references/debug.md`.
 
 Exit code 2 means nothing ran: that is a `🟡 Didn't start` card, not a failure (`references/cards.md` §4).
 
@@ -168,6 +168,7 @@ When the user's request involves a browser — or writing test cases:
 - Multiple independent browser tasks → Read `references/parallel.md` first
 - View, share, or validate run evidence (`.evidence` packs) → Read `references/evidence.md`
 - Debug a failed run → Read `references/debug.md`
+- The quick debug did not explain a failure, a run passed but looks wrong, or the person asks whose fault a failure is or whether the triage in a pack is right → Read `references/triage.md`
 - Configure kane-cli or check directory layout → Read `references/setup-and-config.md`
 - Browse / create / pick a Test Manager project or folder, or interpret the auto-default event → Read `references/test-manager.md`
 - The person wants results saved somewhere else ("change project") → Read `references/test-manager.md` §6. The change is global, and the question must say so
@@ -350,6 +351,7 @@ A draft in chat cannot be run, and a hand-written `_test.md` carries no requirem
 | Need the assurance NDJSON event schema (`--mode agent`) | `references/assurance-parsing.md` |
 | Share the context store with a team, join one, keep stores level, or resolve a sync conflict (kane-cli 0.8.14+) | `references/context-sync.md` |
 | Run failed, need to diagnose | `references/debug.md` |
+| Full triage of a pack: the step sheet, screenshots set against intent, a report with one cause | `references/triage.md` |
 | View, share, validate, or merge evidence packs | `references/evidence.md` |
 | Multiple independent browser tasks | `references/parallel.md` |
 | Need full NDJSON event schema (`run`) | `references/parsing.md` |

.agents/skills/kane-cli/references/debug.md

@@ -1,4 +1,4 @@
-<!-- Read this when a kane-cli run failed and you need to diagnose. Owns the evidence-pack layout (the only source of run logs), how to read a sealed pack, the debugging flow, how to read failure.yaml (and why a designed test is not fixed by rephrasing), the common-failure-patterns table, and when to file a bug report vs not. -->
+<!-- Read this when a kane-cli run failed and you need to diagnose. Owns the evidence-pack layout (the only source of run logs), step identity and the join rules, what a step folder holds, how to read a sealed pack, the debugging flow, how to read failure.yaml (and why a designed test is not fixed by rephrasing), the common-failure-patterns table, and when to file a bug report vs not. -->
 
 # Failure Handling & Log Inspection
 
@@ -17,7 +17,7 @@ unzip -l <pack>                                            # list entries
 unzip -p <pack> "tests/*/result.yaml"                      # verdict + per-step outcomes
 unzip -p <pack> "tests/*/steps/*/failure.yaml"             # failure records (failed steps only)
 unzip -p <pack> "tests/*/logs/0-console.ndjson"            # browser console, run 0
-unzip <pack> "tests/*/steps/*/screenshot.png" -d /tmp/ev   # extract screenshots to view
+unzip <pack> "tests/*/steps/*/screenshot.*" -d /tmp/ev     # extract screenshots to view
 ```
 
 Pack layout (per test):
@@ -33,18 +33,49 @@ tests/<test-id>/
 │   ├── <n>-actions.ndjson     # agent actions per run — the old runs/<n>/ actions log, now in-pack
 │   ├── <n>-console.ndjson     # browser console, per-step attributed
 │   └── <n>-network.har        # network traffic, per-step attributed
-├── steps/<ordinal>-<step-id>/ # screenshot.png, annotated.png, step.json,
+├── steps/<ordinal>-<step-id>/ # step.json, screenshot.* (png or jpg), annotated.png (some steps),
 │                              #   failure.yaml (failed steps only)
 ├── auteur/execution.json      # full execution trajectory
 └── v16-trajectory/            # per-run planning summaries
 ```
 
+### Step identity and joins
+
+One step has a different name in each file. Join on `action_id`, never on a step number.
+
+| Name | Where | Meaning |
+|---|---|---|
+| ordinal | leading number of the step folder; `result.yaml` `ordinal` | order in the pack |
+| id | rest of the folder name; `step.json` `id`; `result.yaml` `id` | `<run index>-<agent step>` for a step that ran |
+| action_id | `step.json`, the `result.yaml` row, the `<n>-actions.ndjson` line, `auteur/execution.json` `operations[].operation_id` | the join key across files |
+| step | `<n>-actions.ndjson` `step` | counted per file; never a join key |
+
+Join a step folder to the other files by `id`, then `action_id`, then ordinal. The run index `<n>` of a step id `<n>-<m>` is the prefix of its logs: `logs/<n>-run.log`, `logs/<n>-actions.ndjson`.
+
+Where the intent of a step lives:
+
+| Source | Fields | Notes |
+|---|---|---|
+| `auteur/execution.json` | `instruction`, `reasoning`, `status`, `error_message`; `operations[].intent` | The authored steps. In a replay pack this describes the run that authored the test, so its `status` is the authored outcome |
+| `logs/<n>-actions.ndjson` | `action_type`, `action_instruction`, `rationale`, `action_params`, `selectors`, `status`, `error`, `timestamp` | What this run did |
+| `step.json` | `kind`, `summary`, `status`, `duration_ms`, `url` | One-line view of the step |
+
+### What is and is not in a step folder
+
+- **The screenshot is taken before the step's action runs**, in an authoring run and in a replay. The effect of a step is in the next step's screenshot.
+- `annotated.png` draws the action's marker on that same before-the-action screenshot. Not every step has one.
+- Steps the run never reached are `skipped` rows in `result.yaml` and have no folder.
+- An action can have a line in `<n>-actions.ndjson` and no folder: it has no screenshot of its own.
+
+`scripts/evidence-map.mjs` prints all of this as one step sheet (`references/triage.md` §1).
+
 ## Debugging Flow
 
 1. **Parse the `run_end` event** from stdout — `status`, `reason`, `summary`, plus `session_dir`.
 2. **Open the pack**: the failed step's `failure.yaml` (error + page state), then that step's slice of `<n>-console.ndjson` / `<n>-network.har` — a 4xx/5xx or JS error usually explains the failure; cite it in plain language.
-3. **Look at the failing step's `annotated.png`** — it highlights the element the agent acted on; makes "clicked the wrong thing" / "element missing" obvious. Render it inline.
+3. **Look at the failing step's `screenshot.*` and `annotated.png`.** They show the page **before** the step's action ran, with a marker on what the agent was about to act on; that makes "aimed at the wrong thing", "element missing" and "something covers it" obvious. Whether an action took effect is in the **next** step's screenshot. Render the image inline.
 4. **Check `tui.log`** (in the pack's `logs/`, or `{session_dir}/tui.log`) for session-level issues (Chrome launch, auth, upload).
+5. **Still unexplained, a pass that looks wrong, or the person asks whose fault it is?** Do the full triage: `references/triage.md`.
 
 Offer the user the visual route too: `kane-cli evidence serve <pack>` opens it in the hosted viewer. If a pack won't open, `kane-cli evidence validate <pack>` tells you whether it's truncated/unsealed.
 

.agents/skills/kane-cli/references/evidence.md

@@ -18,7 +18,7 @@ A `.evidence` file is a standard zip: `unzip -l <pack>` lists it, `unzip -p <pac
     ├── logs/                         # meta.yaml, tui.log, and per run index n:
     │                                 #   <n>-run.log, <n>-actions.ndjson,
     │                                 #   <n>-console.ndjson, <n>-network.har
-    ├── steps/<ordinal>-<step-id>/    # screenshot.png, annotated.png, step.json
+    ├── steps/<ordinal>-<step-id>/    # screenshot.* (png or jpg), annotated.png, step.json
     │                                 #   (+ failure.yaml on failed steps)
     ├── auteur/execution.json         # full execution trajectory
     └── v16-trajectory/               # per-run planning summaries
@@ -74,7 +74,7 @@ Combines packs (execution ids or paths, order-significant) into one, sealed by d
 
 ## Debugging with a pack
 
-The pack is the first place to look on a failure: the failed step's **failure record** (error + page state), its **console/network slice** (4xx/5xx or JS errors usually explain it), and the **annotated screenshot** (what the agent actually acted on). Full failure workflow: `references/debug.md`.
+The pack is the first place to look on a failure: the failed step's **failure record** (error + page state), its **console/network slice** (4xx/5xx or JS errors usually explain it), and the **annotated screenshot** (what the agent was about to act on; a step's screenshot is taken before its action runs). Full failure workflow: `references/debug.md`. A step-by-step triage with a report: `references/triage.md`.
 
 ## Evidence merge identity
 

.agents/skills/kane-cli/references/triage.md

@@ -0,0 +1,79 @@
+<!-- Read this when the quick debugging flow in debug.md did not explain a failure, when a run passed but looks wrong, when the person asks whose fault a failure is, or when they ask whether the triage written inside a pack is right. Owns the step sheet script, the triage steps, the screen table, the triage report and the cause buckets. The pack layout and join rules stay in debug.md. -->
+
+# Triaging a run from its evidence pack
+
+A pack records three views of every step: what the screen showed, what the agent meant to do, and what the logs say happened. A triage is a claim that holds in all three. **The screen can hold facts no log mentions** (a dialog, a toast, a spinner, an error banner, a keyboard), so the screenshots are read every time.
+
+Pack layout, step identity and join rules: `references/debug.md`.
+
+## 1. Steps
+
+1. **Map the pack.** Run the step sheet. It joins step folders, the actions log, `auteur/execution.json` and `result.yaml` by `action_id`, prints `run.yaml` and the failure index, and lists the error lines of the run logs.
+
+   ```bash
+   node "<skill dir>/scripts/evidence-map.mjs" <pack.evidence> --out <new empty dir>
+   node "<skill dir>/scripts/evidence-map.mjs" <extracted pack dir> [--test <test id>]
+   ```
+
+   The first form unzips the pack into the folder you name, which must be empty. A merged or suite pack holds many tests: take them one at a time from the failure index, with `--test`.
+
+2. **Frame the run** from `run.yaml` and `result.yaml`: `run_kind` (`author` or `replay`), `origin`, the kane-cli version, and the environment. A replay repeats recorded actions, so a wrong choice of action belongs to the run that authored the test, not to this one.
+
+3. **Read every log signal** the sheet printed, in the log itself, with the lines around it. A sign-in, network or browser error here outranks any step-level story. The sheet only matches patterns: search the logs yourself for the failing step, by the action's `at=` time.
+
+4. **Pick the window.** The step carrying the failure record, the step before it, and the step after it when one exists.
+   - Check the sheet's `UNMAPPED` list first. An action with no step folder ran without a screenshot of its own, and it can be what ended the run.
+   - Add an earlier step of the same kind as the failing one, with the step after it. A before and after pair of the same action that worked is the cheapest comparison.
+
+5. **Fill one screen row per window step** (the table in §2). For each:
+   - Open `screenshot.*` with your image-reading tool. Open `annotated.*` too when present.
+   - Take the intent from the sheet: the authored `intent`, `instruction`, `reasoning` and `inside`, then the action's instruction, `params` and `rationale`. When the instruction text disagrees with the action type or the rationale, say so and trust the screenshot.
+   - Compare the screen to the intent.
+   - Prove the effect from the **next** step's screenshot: a step's own screenshot is taken before its action runs. If no later screenshot exists, write "not provable from the screen" and name the log line you used in its place.
+
+6. **Check the triage inside the pack.** A step `failure.yaml` with a `triage:` block is a claim written by an automatic investigation. State whether the screen table and the logs support it.
+
+7. **Name one cause** (§3) and report.
+
+## 2. Report
+
+The report has these six parts, in this order:
+
+1. **Verdict.** One or two sentences: what failed, and the cause.
+2. **Screen table.** Required, one row per window step:
+
+   | Step folder | Mapped by | Screenshot opened | What the screen shows | What the agent was doing | Screen vs intent | Effect seen in next screenshot |
+   |---|---|---|---|---|---|---|
+
+   `Mapped by` is `id`, `action_id` or `ordinal`. A step folder with no image names the earlier image it borrows. An action with no step folder gets one row with "no image": the last folder's screenshot was taken before it and is not its screen.
+3. **Log evidence.** Each claim with the file and line inside the pack.
+4. **Triage inside the pack.** Agree or disagree, and why. "None in pack" when absent.
+5. **Other findings.** Defects the pack shows that did not cause this failure, such as a step that passed without changing the screen. "None" when absent.
+6. **Not determined.** What the pack cannot show, and what would settle it (a re-run, a look at the app, a bug report).
+
+Label anything you have not tested as a hypothesis.
+
+Give the person the verdict in plain words first. Show the table and the rest when they ask for the detail, or when the cause is not the obvious one.
+
+## 3. Causes
+
+Pick the earliest fault the pack proves.
+
+| Cause | Typical sign | What to do next |
+|---|---|---|
+| The app under test | The screenshot shows the product misbehaving: an error banner, a wrong value, a button that does nothing | Tell the person it is a product bug and show the screenshot. Do not change the test to get around it |
+| The test or the objective | `test.md` asks for something ambiguous, impossible, or no longer true of the app | Fix the wording. For a designed test, read `references/assurance.md` §6.1 first |
+| The environment | Sign-in, network, browser or device errors in the logs around the failing step | Fix the setup and run again (`references/setup-and-config.md`) |
+| kane-cli | The screen was right and the step was clear, and the action was still wrong or did not take effect; or a replay fails a step that the authoring run passed on the same screen | Offer a bug report (`references/debug.md` §Filing a bug report) |
+
+## 4. Common mistakes
+
+| Mistake | Fix |
+|---|---|
+| Matching a screenshot to a step by step number | Step numbers are counted per file. Use the sheet's join |
+| Reading a screenshot as the result of its step | It is the screen before the action. Look one step later |
+| Trusting a `passed` step | Passed means the action returned, not that the screen changed |
+| Blaming the step that carries the failure record | Check the `UNMAPPED` list for actions after it first |
+| Repeating the triage inside the pack | Check it against the log signals and the screen table first |
+| Reading an off-target marker as a wrong click | The marker is drawn on a screenshot taken before the action. Confirm the click in the actions log |
+| Quoting `*.har` content | It can carry credentials. Cite the request, never the secret |