feat(detection): yield the event loop, cap per-request scan time, add semantic-only header opt-out

docs/adapters/decorators.md

@@ -50,6 +50,7 @@ class RouteConfig:
         self.api_key_required: bool = False
         self.geo_rate_limits: dict[str, tuple[int, int]] | None = None
         self.excluded_detection_headers: set[str] | None = None
+        self.excluded_semantic_detection_headers: set[str] | None = None
         self.excluded_detection_params: set[str] | None = None
         self.excluded_detection_body_fields: set[str] | None = None
         self.enabled_detection_categories: set[str] | None = None

docs/api/decorators.md

@@ -156,15 +156,17 @@ def detection_exclusion(
     body_fields: set[str] | None = None,
     categories: set[str] | None = None,
     scan_body: bool | None = None,
+    semantic_headers: set[str] | None = None,
 ) -> Callable[[Callable[..., Any]], Callable[..., Any]]: ...
 ```
 
-All five kwargs are optional (`set[str] | None` for `headers`/`params`/`body_fields`/`categories`, `bool | None` for `scan_body`). Passing `None` (or omitting) leaves the corresponding `RouteConfig` field unset, the route inherits the global `SecurityConfig` value at request time. Passing a value replaces the inherited value at this route only.
+All six kwargs are optional (`set[str] | None` for `headers`/`params`/`body_fields`/`categories`/`semantic_headers`, `bool | None` for `scan_body`). Passing `None` (or omitting) leaves the corresponding `RouteConfig` field unset, the route inherits the global `SecurityConfig` value at request time. Passing a value replaces the inherited value at this route only.
 
 - `headers`, header names skipped by detection. Merged with `SecurityConfig.excluded_detection_headers` and the hardcoded default exclusion list.
 - `params`, query parameter names skipped by detection. Replaces (does not merge with) the global set when set.
 - `body_fields`, top-level JSON body keys skipped by detection. Replaces the global set when set.
 - `categories`, categories the regex scanner runs at this route. Replaces the global `enabled_detection_categories`. Custom user patterns always run regardless.
+- `semantic_headers`, header names whose values skip the semantic analyzer only (regex scanning is unchanged). Merged with `SecurityConfig.excluded_semantic_detection_headers`.
 - `scan_body`, whether to scan the request body at this route. Replaces the global `detection_scan_body` when set.
 
 ```python

docs/api/models.md

@@ -48,6 +48,7 @@ class SecurityConfig(BaseModel):
     global_behavior_rules: tuple[BehaviorRuleConfig, ...] = Field(default_factory=tuple)
 
     excluded_detection_headers: set[str] = Field(default_factory=set)
+    excluded_semantic_detection_headers: set[str] = Field(default_factory=set)
     excluded_detection_params: set[str] = Field(default_factory=set)
     excluded_detection_body_fields: set[str] = Field(default_factory=set)
     enabled_detection_categories: frozenset[str] = Field(
@@ -135,6 +136,7 @@ class SecurityConfig(BaseModel):
     endpoint_rate_limits: dict[str, tuple[int, int]] = Field(default_factory=dict)
 
     detection_compiler_timeout: float = Field(default=2.0, ge=0.1, le=10.0)
+    detection_max_scan_seconds: float = Field(default=10.0, ge=0.1, le=60.0)
     detection_max_content_length: int = Field(default=10000, ge=1000, le=100000)
     detection_max_body_inspect_bytes: int = Field(default=262144, ge=1024, le=10485760)
     detection_preserve_attack_patterns: bool = Field(default=True)
@@ -184,6 +186,7 @@ These fields opt specific request components out of penetration detection. The h
 | Field                              | Type        | Default                          | Description                                                                |
 |------------------------------------|-------------|----------------------------------|----------------------------------------------------------------------------|
 | `excluded_detection_headers`       | `set[str]`  | `set()`                          | Headers exempted only from the categories known to false-positive on their typical values: identity and proxy headers such as `X-Forwarded-For`, `X-Real-IP`, `Host`, `Origin` and `Via` skip `ssrf` only, and every other category (sqli, xss, cmd_injection, template, Log4Shell) still scans them. A name outside the built-in identity set inherits the same rule when its value looks like an address chain, otherwise it gets no exclusion at all. Merged with the hardcoded default list. |
+| `excluded_semantic_detection_headers` | `set[str]`  | `set()`                          | Opt-in. Header names (case-insensitive) whose values skip the semantic analyzer only; regex category scanning of these headers is unchanged, and `excluded_detection_headers` semantics are untouched. Use it when a header legitimately carries high-entropy encoded data, for example `cookie` or `authorization` for apps with JWT sessions, whose base64 segments score close to `detection_semantic_threshold`. Merged with the route-level set from `@security.detection_exclusion(semantic_headers=...)`. |
 | `excluded_detection_params`        | `set[str]`  | `set()`                          | Query parameter names skipped by detection.                                |
 | `excluded_detection_body_fields`   | `set[str]`  | `set()`                          | Top-level JSON body keys skipped by detection.                             |
 | `enabled_detection_categories`     | `frozenset[str]`  | `frozenset(ALL_DETECTION_CATEGORIES)`  | Categories scanned for. Validator rejects unknown labels.                  |

docs/configuration/detection-tuning.md

@@ -25,6 +25,14 @@ Maximum time in seconds for a single regex pattern match. Patterns that exceed t
 | `2.0`    | Balanced. Catches most attacks while limiting resource usage.|
 | `5.0`    | Permissive. Better detection but higher latency risk.        |
 
+### `detection_max_scan_seconds`
+
+**Type**: `float` | **Default**: `10.0` | **Range**: 0.1 - 60.0
+
+CPU time allowed for regex scanning one request (thread CPU time, not wall time, so a busy or CPU-starved host does not exhaust it), summed across every field (query parameters, headers, body values) and every pattern, not per `detect()` call. Once the budget is spent, the remaining patterns and request values are not scanned and the request gets a weighted `pattern_timeout` threat (pattern `scan_time_budget_exhausted`) that counts toward the anomaly score and the reported `timeouts`, the same way a per-pattern `detection_compiler_timeout` does, so padding a request with filler cannot silently skip detection. The default is far above normal traffic (a few milliseconds per request, roughly 16 ms for 8,863 characters on one measured CPU) and about 6 times the worst legitimate case the value cap allows (256 benign query parameters, 512 scan units, measured at about 1.5 s on that CPU), which leaves headroom for slower hosts. Only regex pattern CPU time counts, not decoding or semantic analysis. A pattern that hits `detection_compiler_timeout` is charged that timeout.
+
+In the async tree the scan awaits the regex worker instead of blocking the event loop, so other requests run between patterns. A single regex that is already executing still holds the GIL until it returns. The sync tree has no event loop and keeps blocking.
+
 ### `detection_pattern_validation_cache_path`
 
 **Type**: `str | None` | **Default**: `None`
@@ -82,6 +90,12 @@ Minimum score from the `SemanticAnalyzer` to classify content as a threat. The s
 !!! info "Semantic vs Regex"
     Regex patterns provide definitive threat detection. Semantic analysis is a secondary layer that catches obfuscated or novel attacks. Lowering the semantic threshold increases the chance of catching evasion attempts but also increases false positives on legitimate content containing technical terms.
 
+### `excluded_semantic_detection_headers`
+
+**Type**: `set[str]` | **Default**: `set()`
+
+Opt-in list of header names whose values skip the semantic analyzer only, matched case-insensitively. Regex category scanning of those headers is unchanged, and `excluded_detection_headers` keeps its meaning (it never silences a header). The semantic analyzer counts base64 as obfuscation and encoding layers, so JWT session cookies and bearer tokens can score close to `detection_semantic_threshold` (0.30 to 0.59 measured on sample cookies, 0.71 reported). For apps with JWT sessions, set `excluded_semantic_detection_headers={"cookie", "authorization"}` instead of raising the global threshold. A route can add names with `@security.detection_exclusion(semantic_headers={...})`.
+
 ### `detection_anomaly_threshold`
 
 **Type**: `float` | **Default**: `3.0` | **Range**: 1.0 - 10.0

docs/configuration/security-config.md

@@ -180,6 +180,7 @@ These fields opt request components out of penetration detection. The header set
 | Field                              | Type        | Default                          | Description                                                                |
 |------------------------------------|-------------|----------------------------------|----------------------------------------------------------------------------|
 | `excluded_detection_headers`       | `set[str]`  | `set()`                          | Headers exempted only from the categories known to false-positive on their typical values: identity and proxy headers such as `X-Forwarded-For`, `X-Real-IP`, `Host`, `Origin` and `Via` skip `ssrf` only, and every other category (sqli, xss, cmd_injection, template, Log4Shell) still scans them. A name outside the built-in identity set inherits the same rule when its value looks like an address chain, otherwise it gets no exclusion at all. Merged with the hardcoded default list. |
+| `excluded_semantic_detection_headers` | `set[str]`  | `set()`                          | Opt-in. Header names (case-insensitive) whose values skip the semantic analyzer only; regex category scanning of these headers is unchanged, and `excluded_detection_headers` semantics are untouched. Use it when a header legitimately carries high-entropy encoded data, for example `cookie` or `authorization` for apps with JWT sessions, whose base64 segments score close to `detection_semantic_threshold`. Merged with the route-level set from `@security.detection_exclusion(semantic_headers=...)`. |
 | `excluded_detection_params`        | `set[str]`  | `set()`                          | Query parameter names skipped by detection.                                |
 | `excluded_detection_body_fields`   | `set[str]`  | `set()`                          | Top-level JSON body keys skipped by detection.                             |
 | `enabled_detection_categories`     | `frozenset[str]`  | full `ALL_DETECTION_CATEGORIES`  | Categories scanned for. Validator rejects unknown labels.                  |
@@ -393,6 +394,7 @@ Detection Engine
 | `detection_max_tracked_patterns`    | `int`   | `1000`  | 100 - 5000    | Maximum patterns to track for performance.   |
 | `detection_max_body_inspect_bytes`  | `int`   | `262144`| 1024 - 10485760 | Body size cap read/scanned for detection; distinct from `detection_max_content_length` and `max_request_size`. A body whose declared `Content-Length` exceeds this cap is read through the adapter's bounded reader (`read_body_prefix`) and only the first this-many bytes are scanned, if the adapter implements one; without a bounded reader the body is not read at all, the same skip as before this cap existed. Either way a one-time warning names the cap, the client, and which of the two happened. A signature that straddles the cap boundary is still caught: a small overlap (the longest compiled pattern length, up to 256 bytes) is read past the cap on every path, including a declared oversized `Content-Length`. |
 | `detection_max_scan_values`         | `int`   | `512`   | 2 - 100000    | Maximum values (query params, headers, JSON keys/values, form/multipart fields) scanned per request; remaining values are skipped and a one-time warning logs the client IP once reached. Each named value costs two scan units (name, then value), so the minimum is 2. |
+| `detection_max_scan_seconds`        | `float` | `10.0`   | 0.1 - 60.0    | Cap on the CPU time spent regex scanning one request (not wall time, so a busy or CPU-starved host does not exhaust it), summed across every field and pattern (not per `detect()` call). Once reached, remaining patterns and values are not scanned and the request receives a weighted `pattern_timeout` threat (pattern `scan_time_budget_exhausted`) that counts toward the anomaly score and the event `timeouts`, so padding a request cannot skip detection; a one-time warning logs the client IP. Only regex pattern CPU time is counted. |
 | `detection_max_scan_chars`          | `int`   | `65536` | 1024 - 262144 | Maximum total characters, across every value handed to the pattern engine per request (counted at the same point as `detection_max_scan_values`), before remaining values are skipped and a one-time warning logs the client IP. A value already in progress when the budget is checked is always scanned in full; only values that would start after the budget is spent are skipped, so a single large value is never silently dropped (GHSA-3hfx-8m47-5f9h residual). |
 | `detection_max_json_depth`          | `int`   | `32`    | 1 - 1000      | Maximum nesting depth of a JSON request body walked structurally. A dict or list reached at this depth is not descended into: it is serialized back to text and scanned as one value instead, bounded by `detection_max_content_length`, so content hidden below the cap is still scanned as text rather than structurally. A one-time warning logs the client IP once reached (GHSA-f6cf-jjhc-qp85). |
 | `detection_binary_min_run_length`   | `int`   | `16`    | 4 - 1024      | Minimum length a printable run inside a binary-dense multipart file-part payload must reach to be pattern-scanned. A part payload whose binary artifact characters make up at least a fifth of it is reduced to its printable runs before scanning; compressed bytes produce attack-shaped printable fragments at a rate that grows with file size, so shorter runs are skipped instead of matched. Text uploads and short or mostly-text payloads stay below the ratio and keep their full scan, whole-body fallback scans are never reduced (raw-body signature coverage stays intact), and text embedded in uploads (scripts inside PDFs, stored paths in archives) forms runs past the threshold and is scanned in full. File names, multipart field names, and part headers are scanned regardless (see `docs/configuration/detection-tuning.md`). |

docs/internals/detection-engine.md

@@ -48,6 +48,8 @@ flowchart TD
 
 Built-in (compile-time-vetted) patterns match directly via `pattern.search()`, with no per-match thread or timeout. Only custom patterns (added via `add_pattern(..., custom=True)`) are routed through the safe-matcher timeout wrapper, a custom-pattern match is flagged as timed out once elapsed time reaches `0.9 * compiler.default_timeout` (i.e. `0.9 * detection_compiler_timeout` when a `SecurityConfig` is wired; legacy/no-compiler mode falls back to a hardcoded 2.0s field default).
 
+The async tree awaits each shared-executor future (`_await_regex_future` in `guard_core/detection_engine/compiler.py`) rather than calling `future.result`, so the event loop runs other tasks between patterns; the sync mirror keeps the blocking `future.result` (the unasync script swaps the helper). A per-request regex CPU time budget (`detection_max_scan_seconds`, tracked in request-scoped context variables by `_scan_value_budget`, fed by thread CPU time measured around inline scans and inside the executor worker) stops the pattern loop once spent and emits a weighted `pattern_timeout` threat named `scan_time_budget_exhausted`; `excluded_semantic_detection_headers` skips only the semantic analyzer for the named headers.
+
 When no `SecurityConfig` is wired at construction (legacy mode), `SusPatternsManager` builds neither a `PatternCompiler` nor a `ContentPreprocessor`; content is instead truncated to `detection_max_content_length` (default 10000) via direct slicing before matching, and every pattern, including what would otherwise be a "custom" pattern, runs through the same shared-executor timeout path described above (`_check_pattern_with_timeout`), reporting to the same success/timeout counters. Running detection unconfigured this way is deprecated and will be removed in a future major release: the first time `detect()` actually executes in this state, it raises a `DeprecationWarning` and logs an identical `logger.warning` naming the reason (`_LEGACY_DETECTION_WARNING`, checked by the `_warn_if_legacy_detection` helper in `guard_core/handlers/suspatterns_handler.py`), once per process, a module-level flag shared by every `SusPatternsManager` instance, not a per-instance one. The check runs from inside `detect()` itself, not at construction or at `configure()`, since the module-level singleton is always built unconfigured at import time and only gets a `SecurityConfig` later from the pipeline; a normally initialised pipeline configures the singleton (directly via `sus_patterns_handler.configure(config)`, or implicitly through `detect_penetration_attempt(request, config)`) before it ever runs detection, so it never warns.
 
 ___