Skip to content

[Benchmark] "none" is no key for --credential-env; the key is redacted as written and as a whole word - #4499

Open
coryjett wants to merge 1 commit into
Niko1221:mainfrom
coryjett:fix-bench-none-key
Open

coryjett wants to merge 1 commit into
Niko1221:mainfrom
coryjett:fix-bench-none-key

Conversation

@coryjett

Copy link
Copy Markdown

Problem and result

Fixes #4483. With a run config whose api_key is "none" and STRATA_KEY=none in the environment,
strata bench run --profile ... --credential-env STRATA_KEY sent Bearer none on every request and, because the
redactor replaced the key as a case-insensitive substring, wrote "the first mpty reasoning" for "nonempty" in its
README (the run in #4468). Any key of four or more characters that is also a common word, or a short hex string that
occurs inside a hash, corrupted the README, outputs.jsonl, config.json and engine.log the same way.

After: bench_facts.credential_value() treats an empty or "none" (any case) credential as no key, the same rule the
run-config path already applied; bench run uses it for --credential-env. The key is redacted exactly as written
(case-sensitive) and only as a whole word, in strata_cli/bench_facts.py and in tools/strata_bench.py. User and
host names, home folders and private addresses are redacted as before.

Related discussion

#4483

Approach

A small helper plus an exact flag on the redactor's add() in both runners. The Bearer <token> pattern is
unchanged, so an authorization header is still hidden whatever the key. Touches strata_cli/cli.py at the
bench run key read (~line 1255), well away from #4476's _redact_argv change; the two apply independently.

Validation

Passed:

  • python3.12 -m unittest tests.terminal.test_bench_redactor tools.test_strata_bench tests.terminal.test_bench_standard tests.terminal.test_bench_facts_amd:
    Ran 106 tests in 54.144s, OK (1 skipped). Strata Control 0.1.52's bundled Python 3.12.15, Ubuntu 26.10, this branch (main 46c152a + this commit).
  • The new tests against main's bench_facts.py, cli.py and tools/strata_bench.py (production files stashed, tests kept): Ran 11 tests in 0.012s FAILED (failures=3, errors=2) (credential_value is missing on main; the substring and case cases fail).

Failed:

  • None observed.

Not run:

  • A new strata bench run with this branch.

Limits

  • A key shorter than four characters is still never redacted (unchanged floor; the Bearer pattern needs eight).
    Warning on short keys would be a separate change.
  • A key directly followed by - or a word character (abcd1234-x) is not a whole word and is not replaced; the
    old substring match would have caught it, at the cost above.

🤖 Generated with Claude Code

…written and as a whole word (Niko1221#4483)

bench run took a credential-env value of "none" (serve's own no-key value) as a real key, sent it as a bearer
token and, because the redactor replaced the key as a case-insensitive substring, wrote "nonempty" as
"<key>mpty" in its README. credential_value() treats none/empty as no key, as the run-config path already did;
the key pattern is now exact-case and word-bounded in both runners.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Benchmark] bench run takes a credential-env value of "none" as a real key and redacts the word everywhere

1 participant