Skip to content

fix(SatisfactionCapture): numbered list items must not parse as a 1/10 rating - #1670

Open
asdf8675309 wants to merge 2 commits into
danielmiessler:mainfrom
asdf8675309:fix/explicit-rating-list-item-false-positive
Open

fix(SatisfactionCapture): numbered list items must not parse as a 1/10 rating#1670
asdf8675309 wants to merge 2 commits into
danielmiessler:mainfrom
asdf8675309:fix/explicit-rating-list-item-false-positive

Conversation

@asdf8675309

@asdf8675309 asdf8675309 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

parseExplicitRating() in LifeOS/install/hooks/SatisfactionCapture.hook.ts reads a user prompt for an explicit 1–10 satisfaction rating. It reads ordinary prose that begins with a number as a rating. Three shapes, each hidden behind the previous one:

shape example parsed as
list marker 1) what does this flag do? rating 1
bare digit + separator 2 - config.md -- add the do-not-edit header, and also back up the file first rating 2
natural English 1 thing that's worth noting… · 3 action items to take from this… rating 1 / 3

The first is a straightforward gap: the reject guard bailed only when the character after the digit matched [/.\dA-Za-z], so ) and ] passed. SENTENCE_STARTERS is a word list (items?|things?|steps?|…) with no defense against a list delimiter.

The other two cannot be fixed syntactically. - is an accepted rating separator, so 2 - <prose> is shape-identical to 8 - nice. And 1 thing that's worth noting… is just English. Only comment length separates them.

This is not cosmetic. A rating < 5 writes a learning note; ≤ 3 also calls captureFailure(), which copies the entire session transcript to disk as a permanent failure record; every rating writes a row to the satisfaction ledger that feeds goal tracking. One misparse contaminates three surfaces at once — and because the kebab-case bundle name is LLM-generated from the phantom rating, a plain question acquires a fabricated failure description that reads as authoritative to everything downstream.

Change

One file, one concern, two commits.

  • Extend the reject guard's character class: /^[/.\dA-Za-z]//^[/.)\]\dA-Za-z]/.
  • Add MAX_BARE_DIGIT_COMMENT = 80, capping the trailing comment on the bare-digit path only. The N/10 and word forms are unambiguous rating syntax and are deliberately left uncapped, so 8/10 <long comment> still works.
  • Export parseExplicitRating so it is unit-testable in isolation. No signature change.

. was already in the class, so 1. first thing was never affected.

Why it's safe

The failure mode is a false negative: an ambiguous input returns null and one rating goes unrecorded. That is strictly better than the alternative, where a false positive writes a permanent transcript copy, a learning note, and a phantom ledger row. The code comments state this so a future reader doesn't relax it back toward permissiveness.

Existing installs cannot break on upgrade. The change only ever converts a rating into null, and only for a digit followed by )/], or a bare digit trailed by more than 80 characters of prose. No currently-accepted rating form produces either shape.

Verification

No test runner exists in this repo, so the case table is the proof — and SatisfactionCapture.test.ts ships alongside so it becomes runnable the moment bun test is wired up.

I ran 34 inputs through the pre-change and post-change implementations side by side, extracting both versions of the function into isolated modules and diffing the outputs rather than reasoning about the regex. 6 changed, all numbered-list shapes. 0 legitimate rating forms changed.

Input Before After
1) install the CLI, 2) run the setup script {rating:1, comment:") install…"} null
1) what does this flag do? {rating:1, comment:") what does…"} null
2] bracket form {rating:2, comment:"] bracket form"} null
1) · 10) {rating:1} · {rating:10} null
2 - config.md -- add the do-not-edit header, and also back up the file first {rating:2, comment:"config.md --…"} null
1 thing that's worth noting, the weekly quota reset Friday evening… {rating:1, comment:"thing that's…"} null
1. first thing\n2. second thing null null (already correct)
8/10 really solid work here, especially the part where you caught the edge case {rating:8, comment:"really solid…"} unchanged — not capped
eight really solid work here, especially the part where you caught the edge case {rating:8, comment:"really solid…"} unchanged — not capped
8 · 10 · 7 · 1 · 0 · 11 unchanged unchanged
ten · nine · seven out of ten unchanged unchanged
Eight great work · 10/10, thank you · 9 / 10 · 8 out of 10 nice unchanged unchanged
8 - nice · 8: nice · 4 - could be better · 9 solid work · 5 meh unchanged unchanged
2/10 items · 2 items · 3 files changed · 8. null null

bun run SatisfactionCapture.test.ts23/23 passed, exit 0. Tested on macOS (Darwin 25.5.0, arm64), Bun 1.3.14.

Field evidence. On one real install, across 607 ledger rows, the explicit-rating path produced 5 rows in its entire history and every one was a false positive — zero genuine explicit ratings, ever. Three shapes accounted for them. The reason this went unnoticed for months is that the path is supposed to be the high-signal one, so its output was never suspected; the phantoms simply made the feature look alive. Ten failure bundles and five learning notes were written from those five misparses.

Found but NOT changed

  • 1 ) item — whitespace before the delimiter — still parses when the tail is under 80 characters. The guard inspects only the character immediately after the digit; skipping whitespace would also swallow 8 . nice, a legitimate-ish rating shape. Left alone and documented in the code.
  • The 80-char threshold is a judgment call. Observed phantom tails ran 100–258 characters; real rating comments are a few words. There is no principled boundary, only a wide empirical gap.
  • The fraction path (N/10, N out of 10) and SENTENCE_STARTERS are untouched. Neither interacts with this bug, and both were correct for every case in the table.
  • A positive-whitelist rewrite of the guard (enumerate allowed next-characters instead of blocking) would be more robust in the abstract. Rejected: it changes far more of the function's behavior surface than the bug warrants, and this diff should stay reviewable.
  • bun test cannot resolve yaml in a fresh checkout of this directory (SatisfactionCapture.hook.tslib/identity.tsyaml), because bun test uses stricter resolution than bun run's auto-install. Pre-existing and unrelated to this change — reproducible with a bare import { parse } from 'yaml'. I did not touch package.json; the test targets bun run for exactly this reason.

Scope

Nothing outside parseExplicitRating is touched. No dependencies added, no config changed, no other hook affected. Full diff: +22/−2 in the hook plus one new 113-line test file (135 insertions, 2 deletions total).

…0 rating

parseExplicitRating() treats a prompt beginning with a digit as an explicit
satisfaction rating. Its reject guard only bailed when the character after
the digit matched [/.\dA-Za-z], so a ")" or "]" passed straight through and
an ordinary numbered list parsed as rating 1:

  "1) install the CLI, 2) run the setup script"
    -> { rating: 1, comment: ") install the CLI, 2) run the setup script" }

That is not cosmetic. A rating <= 3 triggers captureFailure(), which copies
the entire session transcript to disk as a permanent failure record, and
writes a 1/10 row to the satisfaction ledger that feeds goal tracking. A
numbered question list therefore manufactures a failure that never happened.

Extends the existing guard's character class with ")" and "]". "." was
already covered, so "1. first thing" was never affected. Also exports the
predicate so it is unit-testable in isolation; no signature change.

Verified against every rating form the function currently accepts: 34 inputs
run through the pre-change and post-change implementations, 6 changed, all
of them numbered-list shapes, 0 changes to any legitimate rating form.

Adds SatisfactionCapture.test.ts (18 cases). The repo has no test runner, so
it is self-contained: `bun run SatisfactionCapture.test.ts` prints PASS/FAIL
per case and exits non-zero on failure.
… digit

The ")"/"]" guard in the previous commit closes the list-marker shape, but not
the more common one. "-" is an accepted rating separator, so:

  "2 - add the header, but also back the file up somewhere outside the install"

is shape-identical to "8 - nice" and still parses as rating 2. Sibling detection
does not help: two of the three observed cases have no second list marker.

Length separates them. Real rating comments are a few words ("nice", "solid
work"); these tails ran 138, 145 and 258 characters. Caps the comment at 80
chars for the BARE-DIGIT path only — "N/10" and word forms are unambiguous
rating syntax and stay uncapped, so "8/10 <long comment>" is unaffected.

Evidence from one real install: across 607 ledger rows, every row the explicit
path ever produced was a false positive — five for five, zero genuine ratings.
Three of the five are this shape; the other two were the ")" shape. The path is
supposed to be the high-signal one, which is why the failure went unnoticed.

Test file grows to 23 cases: the three long-tail shapes, plus two asserting the
unambiguous forms are NOT capped. Re-ran the 34-input pre/post comparison — 6
changed, all list shapes, 0 legitimate rating forms affected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant