Skip to content

Commit 3b63b06

Browse files
committed
nuclei scans the site and its install folders, not fifty paths
nuclei ran out of its hour on 88 of the last 143 scans, many after more than 250,000 requests, with the template set half done. It was handed the site and up to fifty paths, and runs every template against every one. Its templates build their own paths from each target, so a target of /blog/ checks a WordPress installed there, while a target of /api/orders/12 only asks for /api/orders/12/wp-login.php. It now gets the site and at most five folders an application may be installed in: a first folder with paths under it, never an API, version or static folder, and never another host. That cuts the requests for the same template set by about eight times. Injection on deep paths is left to wapiti, sqlmap and dalfox, which take every path, as before. Checked against the last 400 scans: every nuclei finding above low on a deep path came from a template that builds that path from the root, so it is still found. The one exception was a template run against robots.txt as if it were a folder.
1 parent bbed258 commit 3b63b06

1 file changed

Lines changed: 33 additions & 1 deletion

File tree

‎app/core/dast_engines.py‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ def command(tool: str, url: str, host: str, profile: str, workdir: Path,
219219
}.get(profile, "exposure,misconfig,cve,default-login,token-spray,ssrf,tech")
220220
report = workdir / "nuclei.jsonl"
221221
targets = workdir / "nuclei-targets.txt"
222-
targets.write_text("\n".join([url] + _same_site(url, paths)[:50]) + "\n")
222+
targets.write_text("\n".join([url] + mount_roots(url, paths)) + "\n")
223223
# -o with -jsonl writes each finding to the file the moment it is found,
224224
# not -jsonl-export, which only writes the whole set when nuclei finishes.
225225
# nuclei has no run-time limit of its own, so our timeout kills it mid
@@ -311,6 +311,38 @@ def command(tool: str, url: str, host: str, profile: str, workdir: Path,
311311
raise ValueError(f"unknown engine {tool}")
312312

313313

314+
MAX_MOUNT_ROOTS = 5
315+
# A first folder that is a page or a resource, not a place an app is installed.
316+
_NOT_A_MOUNT = re.compile(r"\.[A-Za-z0-9]{1,5}$|^(api|v\d+|static|assets|css|js|img|images|"
317+
r"fonts|media|cdn-cgi|_next|\.well-known)$", re.I)
318+
319+
320+
def mount_roots(url: str, paths, limit: int = MAX_MOUNT_ROOTS) -> list:
321+
"""The folders below the root an application may be installed in, for nuclei.
322+
323+
nuclei's templates build their own paths from each target, so a target
324+
of /blog/ checks a WordPress installed there, and a target of
325+
/api/orders/12 only asks for /api/orders/12/wp-login.php. It used to get
326+
fifty paths, which multiplied every template by fifty, and ran out of time
327+
on most sites with the set half done. A first folder counts when two or
328+
more paths sit under it, or a path ends in it; injection on deep paths is
329+
left to wapiti, sqlmap and dalfox, which take every path.
330+
"""
331+
from collections import Counter
332+
from urllib.parse import urlparse
333+
334+
base = urlparse(url)
335+
root = f"{base.scheme}://{base.netloc}"
336+
seen = Counter()
337+
for item in _same_site(url, paths):
338+
segs = [s for s in urlparse(item).path.split("/") if s]
339+
if not segs or _NOT_A_MOUNT.search(segs[0]):
340+
continue
341+
seen[segs[0]] += 1 if len(segs) > 1 or item.endswith("/") else 0
342+
picked = [name for name, n in seen.most_common() if n >= 1]
343+
return [f"{root}/{name}/" for name in picked[:limit]]
344+
345+
314346
def _same_site(url: str, paths) -> list:
315347
"""Only addresses on the site being scanned, whatever was handed in."""
316348
from urllib.parse import urlparse

0 commit comments

Comments
 (0)