Skip to content

fix(iris_lookup_manage): length-prefix the listings, so a newline in a name is not two entries - #395

Open
PYDuquesnoy wants to merge 1 commit into
masterfrom
fix/lookup-listing-framing
Open

PYDuquesnoy wants to merge 1 commit into
masterfrom
fix/lookup-listing-framing

Conversation

@PYDuquesnoy

Copy link
Copy Markdown
Contributor

Closes #394

list_keys and list_tables framed each entry with $CHAR(10) and the Rust side split the program output on newlines. A key or table name containing a line feed was therefore reported as two entries, and neither reported name existed.

Measured before, with the global as the authority

Namespace USER. $ORDER straight off ^Ens.LookupTable is the arbiter of what is actually stored:

one key "a\nb"          $ORDER: key len=3, hasLF=1        (exactly ONE key)
  list_keys   -> {"count":2,"keys":["a","b"]}
  get "a\nb"  -> success, value "v"                       <- the real key
  get "a"     -> KEY_NOT_FOUND                            <- neither reported name exists
  get "b"     -> KEY_NOT_FOUND

one table "ZzA\nZzB"    $ORDER: table len=7, hasLF=1      (exactly ONE table)
  list_tables -> {"count":3,"total_count":3,
                  "tables":["%IRIS_X12ReplyType","ZzA","ZzB"]}

set, get, delete and the value path were all correct — a value containing a newline round-tripped intact. The defect was only the framing of the two listings.

It matters more than the unusual input suggests: these listings exist so a caller can feed names back into get/delete, and the KEY_NOT_FOUND refusal says "Use action=list_keys to see which keys this table holds." A caller following that advice got two names that both fail.

The change

Both programs now write <length>:<entry> with no separator, and a shared parser walks the stream:

Set tKey="" For { Set tKey=$ORDER(^Ens.LookupTable(t,tKey)) Quit:tKey=""  Write $LENGTH(tKey)_":"_tKey }

Three deliberate choices:

  • A length prefix, not base64. base64_encode/base64_decode are not on master — they arrive with feat(stream_inspect): adopt the capability #352 asked about, not upstream's implementation #376, still open — so base64 would either duplicate them or have to wait. And no delimiter byte is safe: the entry can contain anything the global can.
  • parse_length_prefixed_entries returns Err, routed to PARSE_ERROR (which already carries a remedy). This is the crux. The old framing degraded silently into a plausible list, and a parser that skipped what it could not read would do the same — reporting fewer keys than the table holds as though that were the answer. A malformed reply is a fault in this server's own output and says so.
  • The per-entry .trim() is gone. The length prefix delimits exactly, so a name with leading or trailing spaces now survives instead of being silently rewritten.

list_tables also stops accumulating the whole listing into a local tOut before writing it, which removes a <MAXSTRING> ceiling as a side effect.

Verified live after the change

one key "a\nb"          -> {"count":1,"keys":["a\nb"]}          (was count 2)
table "ZzA\nZzB"        -> ["%IRIS_X12ReplyType","ZzA\nZzB","ZzNl"]   one entry, not two
ordinary table, 2 keys  -> {"count":2,"keys":["k1","k2"]}       control
get k1                  -> "v1"                                  control
list_keys absent table  -> TABLE_NOT_FOUND                       control: precheck survived

One precision on the table count: it reads 3 both before and after, but that is not a like-for-like comparison — ZzNl was still present during the second run. The evidence is the newline name arriving as one entry, not the count.

Tests

8 new assertions, every one mutation-checked. Two are worth naming:

the_length_precedes_the_entry_in_the_written_expression compares positions rather than checking contains. Writing the entry and then its length would keep every substring intact while making the stream undecodable — three assertions in tonight's other PRs survived exactly that shape before being tightened.

both_programs_write_a_length_prefix_and_no_newline_delimiter loops over both programs, so a half-applied fix fails. The mutant that reverts only list_tables while leaving list_keys correct is included for that reason: sibling asymmetry accounted for four separate findings tonight (#362 step 3 vs the query() path, #384 create vs delete, #386 get vs delete, #392 a check one branch too late).

A note on the test fixture: my first draft mislabelled a 2-character entry as length 1, and the parser refused it rather than truncating. The failure was mine, and it happened to exercise the property the parser exists for.

…a name is not two entries

list_keys and list_tables framed each entry with $CHAR(10) and the Rust side split the
program output on newlines, so a key or table name containing a line feed was reported as
TWO entries and neither reported name existed.

Measured with the global as the authority — $ORDER straight off ^Ens.LookupTable:

  one key "a\nb"        $ORDER: key len=3, hasLF=1        (exactly ONE key)
    list_keys   -> {"count":2,"keys":["a","b"]}
    get "a\nb"  -> success, value "v"                     the real key
    get "a"     -> KEY_NOT_FOUND                          neither reported name exists
    get "b"     -> KEY_NOT_FOUND

  one table "ZzA\nZzB"  $ORDER: table len=7, hasLF=1      (exactly ONE table)
    list_tables -> ["%IRIS_X12ReplyType","ZzA","ZzB"]

set, get, delete and the value path were all correct; a value containing a newline
round-tripped intact. Only the two listings were wrong. It matters because the listings
exist so a caller can feed names back into get/delete, and the KEY_NOT_FOUND refusal says
"Use action=list_keys to see which keys this table holds" — a caller following that advice
got two names that both fail.

Both programs now write <length>:<entry> with no separator, and one parser walks the
stream. Three deliberate choices:

  - A length prefix rather than base64: base64_encode/base64_decode are not on master,
    they arrive with #376, and no delimiter byte is safe when the entry can contain
    anything the global can.
  - parse_length_prefixed_entries returns Err, routed to PARSE_ERROR, which already
    carries a remedy. This is the crux: the old framing degraded SILENTLY into a plausible
    list, and a parser that skipped what it could not read would do the same, reporting
    FEWER keys than the table holds as though that were the answer.
  - The per-entry .trim() is gone. The length delimits exactly, so a name with leading or
    trailing spaces survives instead of being silently rewritten.

list_tables also stops accumulating the whole listing into a local before writing it,
which removes a <MAXSTRING> ceiling as a side effect.

Verified live: the newline key is one key again, the newline table name is one table, an
ordinary two-key table still lists both, get still returns its value, and list_keys on an
absent table still refuses with TABLE_NOT_FOUND.

Eight new assertions, all eight mutations killed by the predicted test. Two are shaped
against specific failure modes seen tonight: the length-before-entry check compares
POSITIONS, because writing the entry then its length keeps every substring while making
the stream undecodable; and the program assertion loops over BOTH programs, so the mutant
that reverts only list_tables fails — sibling asymmetry accounted for four separate
findings tonight (#362, #384, #386, #392).

Closes #394

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@PYDuquesnoy

Copy link
Copy Markdown
Contributor Author

CI verified on the branch via workflow_dispatch (run 36090422789) — a pull_request run skips e2e-tests. All five jobs green: test, e2e-tests, benchmark-lint, cross-compile-check, windows-handshake. 130 test result: lines.

Confirmed by name, each twice (once in test, once in e2e-tests):

an_entry_containing_a_newline_stays_one_entry                2
several_entries_decode_in_order_including_awkward_ones       2
no_entries_is_an_empty_list_not_an_error                     2
a_malformed_stream_is_an_error_not_a_shorter_list            2
a_length_that_lies_long_is_refused_rather_than_clamped       2
both_programs_write_a_length_prefix_and_no_newline_delimiter 2
the_length_precedes_the_entry_in_the_written_expression      2
list_keys_still_refuses_a_missing_table_first                2
a_name_never_in_this_log_control                            0   <- control on the grep itself

Matched on escape-free anchors, since this log encodes ANSI escapes as literal ^[.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iris_lookup_manage: list_keys and list_tables split on $CHAR(10), so a name containing a newline is reported as two entries and inflates the count

1 participant