fix(redaction): preserve raw JSON layout atomically

.github/workflows/sandbox-log-redaction-quality-ci.yml

@@ -10,6 +10,7 @@ on:
       - "scripts/ci/redact_sensitive_log.py"
       - "scripts/ci/sandboxed_verify.py"
       - "scripts/ci/sandboxed_web_e2e.py"
+      - "tests/test_atomic_json_redaction.py"
       - "tests/test_command_wrapper_redaction.py"
       - "tests/test_opencode_security_boundaries.py"
       - "tests/test_sandboxed_verify.py"
@@ -67,6 +68,7 @@ jobs:
         run: |
           test "$(git rev-parse HEAD)" = "${{ github.event.pull_request.head.sha || github.sha }}"
           python -m coverage run --branch -m pytest \
+            tests/test_atomic_json_redaction.py \
             tests/test_command_wrapper_redaction.py \
             tests/test_opencode_security_boundaries.py \
             tests/test_sandboxed_verify.py \
@@ -100,6 +102,7 @@ jobs:
             scripts/ci/redact_sensitive_log.py \
             scripts/ci/sandboxed_verify.py \
             scripts/ci/sandboxed_web_e2e.py \
+            tests/test_atomic_json_redaction.py \
             tests/test_command_wrapper_redaction.py \
             tests/test_opencode_security_boundaries.py \
             tests/test_sandboxed_verify.py \

CHANGELOG.md

@@ -13,6 +13,7 @@ Semantic Versioning where the repository publishes a release.
 
 ### Fixed
 
+- Redacted complete multiline JSON and bounded JSON spans before line splitting with an iterative, resource-limited token/span rewriter that preserves untouched layout, escape spelling, duplicate-member order/count, scalar categories, container shape, stable markers, and mixed diagnostic records; malformed sensitive candidates and exhausted byte/depth/token/string/replacement/work limits fail closed without parser diagnostics.
 - Redacted opaque Docker/Podman login credentials inside bounded GNU `env` split-string and supported shell `-c` wrapper operands with one shared literal matcher, input/token/work budget, and four-level depth boundary; malformed or compound grammar fails closed while valid `--password-stdin` registries, env unset/chdir operands, Docker publish ports, SSH ports, and unrelated `login` arguments remain visible.
 - Redacted credential-shaped stdout, stderr, timeout evidence, and bounded backend/frontend service-log tails emitted by sandboxed verification and web-E2E subprocesses before those values become CI or review evidence. The boundary now canonicalizes safe ANSI styling and fails closed on single- or multiline rendering controls, scans JSON keys and opaque string values without corrupting scalar types or result schemas, preserves benign metadata and diagnostic domains, removes authorization headers, URL userinfo, and private-key blocks, protects raw and escaped explicitly allowed values before setup and tail selection, handles separated credential options, preserves markers and line boundaries, falls back safely on excessive JSON nesting, and parses high-volume diagnostics within bounded time. Normal executed argv, child exit, timeout, readiness, and network behavior remains unchanged; ambiguous short or fixed-evidence-colliding allowed values and setup/launch exceptions return redacted code `126` evidence before a raw traceback can escape.
 - Bounded the Strix quality self-test's deterministic timeout fixtures to 3-second process and 5-second fake-sleep budgets so exact-head policy evidence completes inside the existing job limit without changing production Strix scanner timeouts, providers, credentials, or review semantics.

docs/doctoring/sandbox-log-redaction.md

@@ -41,7 +41,11 @@ A consumer-path review then found additional integration hazards: post-processin
 
 Issue #907 then exposed a distinct argv-evidence boundary: direct Docker/Podman login options were protected, but opaque credentials inside supported `env` split-string and shell `-c` operands were still treated as ordinary data. The repair recognizes only exact `env -S`, `env --split-string`, `env --split-string=...`, and exact `sh`/`bash`/`dash`/`ksh`/`zsh` basenames with an exact or combined `-c` selector. One root-owned context compiles caller literals once and shares limits of 65,536 UTF-8 input bytes, 4,096 parsed tokens, 262,144 cumulative scan bytes, and four wrapper levels. Unsupported quoting, expansion, comments, escapes, compound-shell syntax, ambiguous trailing argv, or exhausted root budgets fail closed; the depth boundary replaces the remaining nested operand rather than scanning it again. The command is never executed or rewritten at its execution boundary.
 
-The same repair fixes option-context false positives: only an exact Docker/Podman `login` subcommand gives `-p` password meaning; Docker publish ports, SSH ports, unrelated `login` arguments, GNU env unset/chdir operands, and the registry after valid `--password-stdin` remain visible. The invalid `--password-stdin=...` spelling remains conservatively redacted. Atomic multiline/duplicate-key raw JSON preservation remains separate Issue #908 and output/service-file memory quotas remain Issue #766.
+The same repair fixes option-context false positives: only an exact Docker/Podman `login` subcommand gives `-p` password meaning; Docker publish ports, SSH ports, unrelated `login` arguments, GNU env unset/chdir operands, and the registry after valid `--password-stdin` remain visible. The invalid `--password-stdin=...` spelling remains conservatively redacted. Output/service-file memory quotas remain separate Issue #766.
+
+Issue #908 then exposed an ordering and representation defect in the raw-text path. Splitting evidence into lines before structural handling loses the association between a multiline sensitive key, colon, and value. Parsing a complete document into a Python dictionary would avoid that split but collapse duplicate members and normalize whitespace, escapes, number spelling, and punctuation. The repair therefore recognizes complete JSON and JSON spans before line splitting with an iterative token/span state machine. It decodes only bounded key and string tokens for classification, retains every untouched source slice, and applies non-overlapping scalar/key span replacements afterward. Duplicate members retain order and count; sensitive arrays and objects retain shape while scalar leaves keep their string, integer, floating-point, boolean, or null category.
+
+The raw parser shares one root budget of 65,536 input bytes, depth 64, 8,192 tokens, 32,768 bytes per string token, 2,048 replacements, and 262,144 cumulative work units. Limit exhaustion and malformed structural evidence with a sensitive key fail closed to one stable marker without exception text. Command strings and argv arrays reuse the bounded wrapper redactor; when its fail-closed representation changes argv length, the JSON path preserves array shape by replacing every original element. Complete valid spans can coexist with prefixes, suffixes, and multiple records. This does not change `redact_json_value()` for already-materialized trusted objects and does not claim Issue #766's output-memory closure.
 
 ## Test-first evidence
 
@@ -65,12 +69,16 @@ The follow-up fail-first contract independently reproduced eight initial failure
 
 The wrapper follow-up began with **31 focused failures** covering both GNU env split spellings, all five supported shells, combined `-c` selectors, env-to-shell nesting, malformed/compound operands, trailing positional ambiguity, resource limits, depth, and option false positives. The bounded implementation reports **142 passed** in the permanent focused quality selection with exact **100% statement and branch coverage** across the three owned modules (`784` statements and `302` branches), followed by **1,060 passed plus 16 subtests** and exact complete owned-production coverage (`7,395` statements and `2,960` branches) in the complete repository suite and a passing Strix quick gate. Fixtures construct opaque credentials at runtime.
 
+The atomic JSON follow-up began on exact parent `18a6d125fead8cb95972fe3e1a97e4cc4163e9d2` with a test-only head that produced the intended local RED result: **5 failed, 1 passed**. The failures reproduced multiline separation, duplicate-key loss, scalar/container normalization, mixed-record handling, and malformed-candidate leakage before production changed. The GREEN boundary reports **155 focused tests passed** with exact **100% statement and branch coverage** across the three owned modules (`1,039` statements and `416` branches). The complete repository suite reports **1,073 passed plus 16 subtests** and exact complete owned-production coverage (`7,650` statements and `3,074` branches); public callable docstrings, compilation, diff integrity, and the central Strix quick gate also pass. Additional fixtures exercise escaped spelling, empty containers, iterative depth, token, byte, string and replacement exhaustion, malformed parser states, shape-preserving command fallback, benign oversized input, and trusted materialized-object collision handling without committing a fixed credential-shaped literal.
+
 ## Security and privacy interpretation
 
 Redaction is defense in depth, not authorization. It does not make arbitrary sensitive material safe to publish and it does not authorize repositories to pass secrets into sandbox commands. The existing environment minimization remains the primary ingress control; deterministic output redaction limits accidental disclosure if a child process or service emits sensitive-looking evidence anyway.
 
 ANSI styling canonicalization preserves visible diagnostic text and line separators, while cursor movement, backspace, multiline/unterminated control payloads, and invisible Unicode format controls fail closed for every affected evidence line or value. Structured JSON retains benign failure, policy, count, status, expiry, type, and usage metadata, while credential-denoting keys, credential material used as a key, credential-shaped string values, authorization headers, URL userinfo, and multiline private-key blocks are replaced without changing boolean or numeric types. Result markers remain one-line valid JSON with stable trusted keys after redaction, and the collector's literal `api.deepseek.com` classification signal remains visible. Literal protection is limited to non-empty values of names explicitly passed through `--allow-env`; values shorter than eight characters or colliding with fixed evidence text are rejected before execution, while whitespace-bearing values remain supported. The ordinary safe environment allowlist is not treated as secret, avoiding blanket removal of paths, locale data, or other useful diagnostics.
 
+Raw JSON layout preservation applies before line-oriented normalization. It does not interpret arbitrary prefixes as trusted JSON, publish parser errors, reconstruct a dictionary, or silently accept a partially parsed sensitive candidate. Existing escape spelling and line endings are therefore diagnostic evidence rather than data to canonicalize. A bounded benign non-JSON or malformed fragment still uses the general redactor; a fragment that establishes a sensitive structural key but cannot establish a safe value boundary fails closed.
+
 The redactor operates on CI-facing text only. It does not mutate files in the copied repository, service log files on disk, subprocess input, or successful child-process status and lifetime. The wrappers separately apply the documented pre-execution and exception code `126` policy. Operators should therefore interpret `[REDACTED]` as evidence suppression, not as successful removal of sensitive data from the source system that produced it.
 
 ## Rollback