Skip to content

test(descriptions): no advertised tool description carries leaked indentation - #380

Merged
PYDuquesnoy merged 1 commit into
masterfrom
test/advertised-text-is-clean
Sep 28, 2026
Merged

PYDuquesnoy merged 1 commit into
masterfrom
test/advertised-text-is-clean

Conversation

@PYDuquesnoy

Copy link
Copy Markdown
Contributor

The narrow, sound half of #378. Seven caller-facing messages were carrying runs of leaked
indentation — a Rust string continuation collapsed onto one source line keeps its whitespace — and all
seven are fixed in #376/#377. This is the gate that stops it recurring on the one surface whose
population is complete.

Why descriptions, and not "messages"

#378 records that there is no cheap sound gate over every string that can reach a caller:

candidate population why it does not work
a source scan cannot tell a collapsed continuation from a correct one without exclusion rules for whitespace fixtures (["", " "]), embedded ObjectScript whose indentation is deliberate (" Do ..RunUser()"), and aligned comments. Measured: 592 hits across 68 files, all false positives — it matched across lines, so it read the source layout rather than the runtime string
message constants sound but tiny — three in this crate, and they would have caught none of the seven, which were literals inside functions
messages built at call time only reachable by triggering their paths

Descriptions are the one population that is complete, wire-facing and free: every one goes to every
client on every tools/list, and advertised_tools() returns all of them with no IRIS connection. So
this asserts the surface it can assert exhaustively, and the doc comment says plainly that it is not
the whole problem.

Controls, because a clean result here is the easy thing to get wrong

  • the tool list must hold ≥ 20 entries — a zero from an empty list would otherwise read as clean;
  • ≥ 20 must actually carry a description, so the scan cannot be skipping them all as empty;
  • all four toolsets, since a description can be advertised by one and not another;
  • the detector has its own test, driven by the real text as it came back from the tool before the
    fix, plus the repaired form and a newline case that must not be flagged.

That last one matters here specifically: the first version of this scan produced 592 false positives,
and a version tuned the other way would have produced zero real ones with nothing to distinguish the
two.

Verification

  • cargo fmt --all -- --check clean, cargo clippy --workspace --all-targets -- -D warnings clean,
    cargo build --workspace, and the gate with IRIS env unset: 5 result lines, 0 FAILED, both tests
    confirmed by name, a fabricated name returning 0.

  • 2 mutations, each red and restored:

    mutation caught by
    a collapsed continuation injected into iris_execute's real description the gate, naming the tool and reporting a 22-space run
    the tool list truncated to empty the control — "advertised only 0 tools, so a clean result would prove nothing"

Zero descriptions are affected today, so this is green on the tree as it stands: it locks in a
property rather than fixing one. #378 stays open for the source-scan scoping decision.

…entation

The narrow, sound half of issue 378. Seven caller-facing messages were found carrying runs of leaked
indentation — a Rust string continuation collapsed onto one source line keeps its whitespace — and
all seven are fixed in #376/#377. This is the gate that stops the same thing recurring on the one
surface where the population is complete.

## Why descriptions and not "messages"

#378 records that there is no cheap sound gate over "every string that can reach a caller":

* A SOURCE scan cannot distinguish a collapsed continuation from a correct one without exclusion
  rules for whitespace test fixtures, embedded ObjectScript whose indentation is deliberate, and
  aligned trailing comments. Measured: the naive version reported 592 hits across 68 files, every one
  a false positive, because it matched across lines and so read the source layout rather than the
  runtime string.
* Message CONSTANTS are a sound population but a tiny one — three in this crate — and would have
  caught none of the seven, which were literals inside functions.
* Messages built at call time are only reachable by triggering their paths.

Descriptions are the one population that is complete, wire-facing and free: every one goes to every
client on every tools/list, and advertised_tools() returns all of them with no IRIS connection. So
this asserts what it can assert exhaustively and says plainly that it is not the whole problem.

## What it checks, and its controls

All four toolsets, because a description can be advertised by one and not another. Two controls guard
against a vacuous pass: the tool list must hold at least 20 entries, and at least 20 must actually
carry a description — a zero from an empty list would otherwise read as clean.

The detector has its own test, using the real text as it came back from the tool before the fix. That
matters here specifically: the first version of this scan produced 592 false positives, and a version
tuned the other way would have produced zero real ones with nothing to show the difference.

## Verification

fmt, clippy -D warnings, cargo build --workspace, and the gate with IRIS env unset: 5 result lines, 0
FAILED, both tests confirmed by name, a fabricated name returning 0.

Two mutations, each red and restored:

* a collapsed continuation injected into iris_execute's real description — caught, naming the tool
  and reporting a 22-space run.
* the tool list truncated to empty — caught by the control, "advertised only 0 tools, so a clean
  result would prove nothing", rather than passing as clean.

Currently zero descriptions are affected, so this commit is green on the tree as it stands: it locks
in a property rather than fixing one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PYDuquesnoy added a commit that referenced this pull request Sep 24, 2026
Ran this PR's test-after-put against a real instance for the first time. The behaviour is right in all
five cases — a green suite gives test_ok: true; a RED suite gives test_ok: false while the write's own
success stays true, which is the #310 separation working live; no compile gives test_skipped; a compile
failure gives COMPILE_ERROR with the write envelope untouched and no test block; and no `test`
parameter adds nothing at all.

What was wrong is the text. The skip message came back as:

  not run: `test` needs a COMPILED class and this call did not compile. Pass                  compile=true

Four messages, all in this PR: the three TestGate::skipped_reason arms and the unparseable-result
note. Same collapsed line-continuation as the seven fixed in #376/#377 — I fixed those on the branches
I happened to be working in and did not check the others.

## The scan that found the rest, and the one that did not

Running the earlier detector over every branch was useless: master-descended branches report 15-40 hits
each, because most matches are whitespace test fixtures, embedded ObjectScript whose indentation is
deliberate, and aligned doc-comment tables. No conclusion is available from those numbers.

Scoping it to the lines a branch ADDS (`git diff origin/master...origin/<br>`, `^+` only) makes the
population precise. Across all fifteen open PR branches that gives four real leaks, all four here.
Everything else the scan flags on those branches is intentional: measurement tables in doc comments,
the deliberate `"ERROR #5002:   "` truncation fixture, the real IRIS SQLCODE text in #368's fixtures
(which genuinely contains double spaces), and the leaked string the #380 detector test asserts on.

Verified at runtime after the fix by driving the built binary: the skip message contains no double
space.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@PYDuquesnoy

Copy link
Copy Markdown
Contributor Author

Recording the #378 linkage — nothing here references it, so merging would leave it open — but not adding Closes #378, because this PR covers a strict subset of that issue.

Measured against the diff:

files changed:  crates/iris-agentic-dev-core/tests/advertised_text_is_clean.rs   (1 file)
added lines mentioning `description`:        10
added lines touching err_json / fail_with:    0

#378 is about "caller-facing messages" — its reported instance was a compare_namespace refusal message, not a tool description. This PR guards the advertised descriptions. Both matter and the description half is genuinely done; the message half is untouched here.

The remaining half, and why it is blocked

I scoped it on the issue earlier: reconstruct each string literal as the compiler would, then flag only internal multi-space runs — which structurally excludes whitespace fixtures (trim-empty), deliberate ObjectScript indentation (line-leading) and json! padding (not inside a literal), with no ObjectScript marker list to maintain. Measured four scopings on master:

A  all literals, test mods included      35 hits
B  all literals, test mods stripped      18
C  regular strings only, tests included  25
D  regular strings only, tests stripped   8   <- 1 true positive, 7 legitimate

So the gate is variant D plus a justified seven-entry allowlist (verbatim PSQL/IRIS text, deliberate column alignment, embedded ObjectScript in a regular string).

It cannot land before #377. The single true positive in variant D is doc.rs:773 — the iris_doc put/names refusal with a collapsed \-continuation — and #377 is what repairs it. A message gate added now would red on that line until #377 merges.

So #378 needs this PR and a follow-up gate after #377. Closing it is a merge-time call.

PYDuquesnoy added a commit that referenced this pull request Sep 28, 2026
Ran this PR's test-after-put against a real instance for the first time. The behaviour is right in all
five cases — a green suite gives test_ok: true; a RED suite gives test_ok: false while the write's own
success stays true, which is the #310 separation working live; no compile gives test_skipped; a compile
failure gives COMPILE_ERROR with the write envelope untouched and no test block; and no `test`
parameter adds nothing at all.

What was wrong is the text. The skip message came back as:

  not run: `test` needs a COMPILED class and this call did not compile. Pass                  compile=true

Four messages, all in this PR: the three TestGate::skipped_reason arms and the unparseable-result
note. Same collapsed line-continuation as the seven fixed in #376/#377 — I fixed those on the branches
I happened to be working in and did not check the others.

## The scan that found the rest, and the one that did not

Running the earlier detector over every branch was useless: master-descended branches report 15-40 hits
each, because most matches are whitespace test fixtures, embedded ObjectScript whose indentation is
deliberate, and aligned doc-comment tables. No conclusion is available from those numbers.

Scoping it to the lines a branch ADDS (`git diff origin/master...origin/<br>`, `^+` only) makes the
population precise. Across all fifteen open PR branches that gives four real leaks, all four here.
Everything else the scan flags on those branches is intentional: measurement tables in doc comments,
the deliberate `"ERROR #5002:   "` truncation fixture, the real IRIS SQLCODE text in #368's fixtures
(which genuinely contains double spaces), and the leaked string the #380 detector test asserts on.

Verified at runtime after the fix by driving the built binary: the skip message contains no double
space.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PYDuquesnoy added a commit that referenced this pull request Sep 28, 2026
…ether anything compiled (#371)

* feat(iris_doc): run the named test after a compiled write, and say whether anything compiled

Items 2 and 4 of issue 327, measured over 31,009 parsed `tool_use` blocks from 364 transcripts.

`iris_doc(put)` is immediately followed by `iris_test` 579 times. A boolean would cover only
the 346 where the class written IS the class tested; a STRING covers all 579, because in the
other 233 the suite lives beside the class (`Hospital.BO.PatientDb` ->
`Hospital.Tests.BO.PatientDbTest`).

The decision is a pure `TestGate` with four states, not a boolean, because the ways a test does
NOT run need different fixes from the caller:

* `Run(pattern)` — the payload reports a compiled class.
* `NotRequested` — nothing added to the payload at all. A `test_skipped` on every ordinary put
  would be noise on the 2,979 measured puts that want none of this.
* `SkippedCallFailed` — the write did not succeed. Its own envelope is returned UNTOUCHED: it
  already carries the compile error, SCM elicitation or refusal the caller must act on, and
  rebuilding it to add a note would risk dropping the isError flag and hints that are the answer.
* `SkippedNotCompiled { compile_requested }` — two different sentences, because "you did not ask
  for a compile" and "the compile produced nothing" are different states and need different
  fixes.

Keyed off the payload's `compiled`, not off the mode, so it needs no list of modes to keep in
step with `DocMode`.

The write's verdict and the test's verdict stay separate (#310): `success` stays the WRITE's —
the document landed — and the test arrives as `test` with `test_ok` beside it. A red test must
not read as a failed write, and a failed write must not read as a red test.

Sequenced in the `iris_doc` tool method rather than inside `doc::handle_iris_doc`, because
running a suite is `iris_test`'s own 600-line job (polling, the result query, the per-case
shaping of #233/#273). Calling the tool means the two paths cannot disagree about what a red
test looks like.

89 of the measured `put` -> `iris_compile` pairs are a put that did not pass `compile`, followed
by a compile of the SAME class. Issue 327 records that as a 3% tail and proposes the cheap
option rather than flipping the default: have the put say so.

The no-compile branch carried NO `compiled` key at all, so "was it compiled?" was answerable
only by noticing an absence — and an absence reads as "does not apply here", not as "no". All
three write outcomes now report `compiled` AND `compile_requested`, so the states are distinct
from those two fields alone: (false,false) written and nobody asked, (false,true) the compile ran
and failed, (true,true) compiled clean.

fmt, clippy `-D warnings`, `cargo build --workspace`, and the gate with IRIS env unset. 15 new
tests. 10 mutations applied one at a time — including "a blank test counts as a request", "a red
test rewrites the write's own verdict", "iris_doc stops handing its result to the sequencer" and
"the sequencer stops calling iris_test" — each red on its named tests, restored green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(iris_doc): four TestGate messages carried leaked indentation

Ran this PR's test-after-put against a real instance for the first time. The behaviour is right in all
five cases — a green suite gives test_ok: true; a RED suite gives test_ok: false while the write's own
success stays true, which is the #310 separation working live; no compile gives test_skipped; a compile
failure gives COMPILE_ERROR with the write envelope untouched and no test block; and no `test`
parameter adds nothing at all.

What was wrong is the text. The skip message came back as:

  not run: `test` needs a COMPILED class and this call did not compile. Pass                  compile=true

Four messages, all in this PR: the three TestGate::skipped_reason arms and the unparseable-result
note. Same collapsed line-continuation as the seven fixed in #376/#377 — I fixed those on the branches
I happened to be working in and did not check the others.

## The scan that found the rest, and the one that did not

Running the earlier detector over every branch was useless: master-descended branches report 15-40 hits
each, because most matches are whitespace test fixtures, embedded ObjectScript whose indentation is
deliberate, and aligned doc-comment tables. No conclusion is available from those numbers.

Scoping it to the lines a branch ADDS (`git diff origin/master...origin/<br>`, `^+` only) makes the
population precise. Across all fifteen open PR branches that gives four real leaks, all four here.
Everything else the scan flags on those branches is intentional: measurement tables in doc comments,
the deliberate `"ERROR #5002:   "` truncation fixture, the real IRIS SQLCODE text in #368's fixtures
(which genuinely contains double spaces), and the leaked string the #380 detector test asserts on.

Verified at runtime after the fix by driving the built binary: the skip message contains no double
space.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@PYDuquesnoy
PYDuquesnoy merged commit 5dd8a6a into master Sep 28, 2026
10 checks passed
@PYDuquesnoy
PYDuquesnoy deleted the test/advertised-text-is-clean branch September 28, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant