Repository navigation
BUG: ShellCommandOutputScorer has nonlinear matching cost for repeated nonmatching arguments #3698
Description
Describe the bug
ShellCommandOutputScorer takes increasingly long to reject nonmatching text with repeated shell flags. The Destructive Shell regex has overlapping argument repetitions at pyrit/score/true_false/regex/shell_command_output_scorer.py:38-44. RegexScorer runs pattern.search(text) synchronously inside its async scoring method at pyrit/score/true_false/regex/regex_scorer.py:118, so this also stalls other work on the event loop.
Reproduced on freshly fetched main at d3446e045960b5ebee4641aa70fdeb03af186587 on October 31, 2026. Introduced in #3597. Open #3687 adds detection coverage but leaves this destructive-command regex unchanged. Focused searches found no dedicated performance correction.
Steps/Code to Reproduce
Run in the project's Python environment. This only matches inert strings; it does not execute shell commands or call any provider.
import re
from time import perf_counter
from pyrit.score.true_false.regex.shell_command_output_scorer import ShellCommandOutputScorer
pattern = re.compile(ShellCommandOutputScorer._DEFAULT_PATTERNS["Destructive Shell"])
for count in (1000, 2000, 4000):
text = "rm " + "-rf " * count + "relative"
start = perf_counter()
matched = pattern.search(text) is not None
print(count, matched, round(perf_counter() - start, 3))Expected Results
All inputs should return False without the roughly fourfold matching-time increase each time their length doubles. A roughly 16 KB nonmatching response should not hold up async scoring for seconds.
Actual Results
1000 False 0.143
2000 False 0.564
4000 False 2.215
Exact timings depend on the machine, but the nonlinear growth was reproduced. The same input on the implementation before #3597 completed in under 0.0005 seconds in the earlier comparison.
Suggested fix: remove ambiguous argument splits and add bounded long-nonmatching-input regression coverage, preserving existing detection and command-separator behavior.
Screenshots
N/A.
Versions
Windows 11; Python 3.14.4; PyRIT 1.2.0.dev0, editable main at the SHA above. Browser: N/A.
pyrit.show_versions() dependency output (local executable path omitted):
pyrit: 1.2.0.dev0
Cython: None
numpy: 2.4.6
openai: 2.54.0
packaging: 26.3
pip: None
scipy: 1.17.1
setuptools: 84.2.1
sqlite3: None
torch: None
transformers: 5.20.2
Activity
WaqarAnwar commentedon Oct 31, 2026
Roman Lutz (@romanlutz) Thanks for the detailed report. I've opened a fix in #3702.
It's worse than quadratic
Your repro grows quadratically. But when
rm -rfitself repeats, everyrmrestarts the expensive search, so the time grows with the cube of the input length:main)"rm " + "-rf " * 1000 + "relative"(your repro)"rm -rf " * 250 + "relative""rm -rf " * 500 + "relative""rm -rf " * 1000 + "relative"(~7 KB)"rm -rf " * 8000 + "relative"(~56 KB)So an ordinary-sized LLM response could block the event loop for tens of seconds.
The fix
The
rmbranch of theDestructive Shellpattern now:;,|,&,>,<),rmand the first recursive/force flag after it, so the engine never retries other splits,/other than/tmp,~,$HOME,${HOME}).If any
rm … flag … targetsequence exists in a segment, then the firstrmfollowed by its first flag also precedes that target. So detection is unchanged, but each segment is scanned in linear time. Separator handling is also unchanged (rm -rf build; cd /varstill doesn't match).Verification
rm, repeatedrm -rf,rm-suffixed paths, a huge flag token). Each must finish in under 0.5 s.tests/unit/score/regex/pass.Happy to adjust anything.