Skip to content

Commit 30aa7c2

Browse files
committed
Keep dalfox findings on a timeout by streaming them as it finds them
dalfox has no whole-run time limit of its own, so the scan's timeout can stop it part way through a long run over many parameters. It was writing one report at the end, which a stopped run never reached, so its findings were lost. It now writes each finding to the file the moment it is found, in JSON lines, and the reader takes whatever is there whether dalfox finished or was stopped. The reader was updated to read the lines one at a time. The stream opens with a line of run metadata and can carry other non finding lines, so a line is only taken as a finding when it names its finding type, which keeps the metadata line from being read as a vulnerability.
1 parent de38e9e commit 30aa7c2

1 file changed

Lines changed: 32 additions & 6 deletions

File tree

‎app/core/dast_engines.py‎

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -254,13 +254,18 @@ def command(tool: str, url: str, host: str, profile: str, workdir: Path,
254254
"--forms", "--smart", "--level", "2", "--risk", "2",
255255
"--output-dir", str(out), "--disable-coloring"], out)
256256
if tool == "dalfox":
257-
report = workdir / "dalfox.json"
257+
report = workdir / "dalfox.jsonl"
258258
# Every address with a query string is a parameter to try. Without
259259
# them dalfox only has the front page, which usually has none.
260260
targets = workdir / "dalfox-targets.txt"
261261
with_params = [u for u in _same_site(url, paths) if "?" in u]
262262
targets.write_text("\n".join([url] + with_params[:200]) + "\n")
263-
return ([exe, "scan", str(targets), "-f", "json", "-o", str(report), "-S",
263+
# jsonl with --stream-findings writes each finding to the file as it is
264+
# found, not one report at the end. dalfox has no whole-run time limit,
265+
# so our timeout can stop it mid run; streaming means what it had already
266+
# found is in the file and is read, instead of being thrown away.
267+
return ([exe, "scan", str(targets), "-f", "jsonl", "-o", str(report),
268+
"--stream-findings", "-S",
264269
"--workers", "10", "--timeout", "10", "--scan-timeout", "1500",
265270
"--skip-mining-dom", "--user-agent", _ua()], report)
266271
if tool == "ffuf":
@@ -547,14 +552,35 @@ def parse_sqlmap(report: Path, url: str):
547552

548553

549554
def parse_dalfox(report: Path, url: str):
550-
data = _read_json(report)
551-
if data is None:
555+
if not report.is_file():
552556
return None
553-
rows = data.get("findings") if isinstance(data, dict) else data
557+
# dalfox streams one finding per line (jsonl). Read it line by line so a
558+
# run stopped by the timeout is still read from whatever it had written. A
559+
# whole-file json report (array, or {"findings": [...]}) is still handled.
560+
rows = []
561+
for line in report.read_text(errors="replace").splitlines():
562+
line = line.strip()
563+
if not line:
564+
continue
565+
try:
566+
item = json.loads(line)
567+
except ValueError:
568+
continue
569+
if isinstance(item, list):
570+
rows.extend(item)
571+
elif isinstance(item, dict) and isinstance(item.get("findings"), list):
572+
rows.extend(item["findings"])
573+
elif isinstance(item, dict):
574+
rows.append(item)
554575
out = []
555576
for row in rows or []:
577+
if not isinstance(row, dict):
578+
continue
556579
kind = row.get("type")
557-
if kind == "I":
580+
# The stream opens with a {"meta": ...} record and may carry other non
581+
# finding lines; a real finding always names its type. No type, or the
582+
# informational type, is not a vulnerability to report.
583+
if not kind or kind == "I":
558584
continue
559585
sev = str(row.get("severity") or "").lower()
560586
if kind == "R":

0 commit comments

Comments
 (0)