Development merge for 0.5.3

AGENTS.md

@@ -168,19 +168,20 @@ def __init__(self, name="MyDetector", config=MyDetectorConfig()):
     self._register_persistency(self.persistency)  # must be last
 ```
 
-**2. Preserve `config.persist` across `set_configuration()` rebuilds:**
+**2. Write only your outputs in `set_configuration()` — never rebuild `self.config`:**
 
-`set_configuration()` replaces `self.config` via `from_dict()`, which produces a config with no `persist` key — silently dropping the user's persist settings. Save and restore it:
+`set_configuration()` must not reassign `self.config` (e.g. via `from_dict()`). Write only what the configure phase produced, then flip `auto_config` off:
 
 ```python
 def set_configuration(self) -> None:
-    old_persist = self.config.persist
-    # ... build config_dict, call from_dict() ...
-    self.config = MyDetectorConfig.from_dict(config_dict, self.name)
-    self.config.persist = old_persist
+    variables = {...}  # whatever the configure phase decided
+    self.config.events = generate_events_config(variables, self.name)
+    self.config.auto_config = False
 ```
 
-Omitting either step means a `persist:` block in the YAML is silently ignored with no error.
+(`EventSequenceDetector` additionally writes `self.config.fixed_window_size`, since its configure phase picks a window length rather than a variable selection.)
+
+Every other field — `persist`, `auto_config_params`, any detector-specific param — is operator input and is left untouched by construction, since nothing here reassigns `self.config`. Rebuilding the config wholesale is what used to drop `persist` (and everything else) silently.
 
 ## Code Quality
 

docs/detectors.md

@@ -200,7 +200,10 @@ def configure(self, input_):
     )
 ```
 
-The `set_configuration()` method queries the tracker results and generates the final config:
+The `set_configuration()` method queries the tracker results and writes the
+final `events` block. It touches nothing else on the config — everything the
+operator set under `params` or `auto_config_params` must survive untouched, so
+`set_configuration` never rebuilds the config from scratch:
 
 ```python
 def set_configuration(self):
@@ -209,12 +212,8 @@ def set_configuration(self):
         stable_vars = tracker.get_features_by_classification("STABLE")
         variables[event_id] = stable_vars
 
-    config_dict = generate_detector_config(
-        variable_selection=variables,
-        detector_name=self.name,
-        method_type=self.config.method_type,
-    )
-    self.config = MyDetectorConfig.from_dict(config_dict, self.name)
+    self.config.events = generate_events_config(variables, self.name)
+    self.config.auto_config = False
 ```
 
 ### Full lifecycle with auto-configuration
@@ -228,78 +227,171 @@ def set_configuration(self):
 
 When `auto_config` is `False`, steps 1 and 2 are skipped entirely.
 
-
-### Stability segmentation (optional)
-
-Stability classification splits a variable's change history into four segments and
-compares each segment's rate of change against a threshold. By default the segments
-are **equal-count**: each holds the same number of observations, regardless of how much
-time they cover. For bursty log sources that is misleading — a variable that changed
-constantly during a quiet night and then went silent under a flood of daytime traffic
-looks stable, because the flood supplies enough samples to dominate the later segments.
-
-Setting `stability_segmentation: time` switches the segmentation to **equal-duration** cuts
-of the observed time span, so each segment covers the same amount of wall-clock time. The
-detector then needs an event time per record, which it reads from the log's named
-variables (`logFormatVariables`, i.e. the fields declared in the parser's `log_format`)
-under the name given by `timestamp_variable`.
-
-These three parameters live on every `VariableDetector` subclass (`NewValueDetector`,
+That distinction is visible in the config. A detector's settings live in two
+blocks:
+
+* **`auto_config_params`** — inputs *to* the configure phase. They pick which
+  variables the phase selects and are read only while `auto_config` is `True`.
+* **`params`** — operational settings, read during training and detection on
+  every run.
+
+The configure phase writes its results into the top-level `events` block (and,
+for `EventSequenceDetector`, into `fixed_window_size`) and then sets
+`auto_config` to `False`. It never modifies either input block, so a config can
+be rerun with `auto_config: False` and reproduce the same detector.
+
+Both `auto_config` and `Component.configure()` are declared on the shared base,
+so `auto_config_params` is declared there too — on `BasicConfig`, beside
+`auto_config` — rather than on the detector config alone. Detectors are the only
+component type with a real configure phase today, so they are the only ones that
+narrow the block with fields; parsers and alert aggregators inherit it empty, and
+an empty block is omitted from the serialized config, so their YAML is unaffected.
+A component type that grows a configure phase later subclasses `AutoConfigParams`
+and overrides the field, exactly as the variable, combo and sequence detector
+families do.
+
+
+### Stability classification (optional)
+
+Stability classification decides whether a variable's change history counts as
+`STABLE` by running one or more classification methods against it and combining
+their verdicts. There are four independent methods, over two primitives and two
+axes:
+
+| method | what it thresholds | axis |
+|---|---|---|
+| `index` | segment-mean thresholds | equal-count boundaries |
+| `time` | segment-mean thresholds | equal-duration boundaries |
+| `slope_index` | change centroid vs. `slope_threshold` | index positions |
+| `slope_time` | change centroid vs. `slope_threshold` | normalized timestamps |
+
+Any subset of the four may be enabled, and any single one may stand alone. The
+default — `index` alone — is the historical behaviour: each segment's mean rate
+of change is compared against a threshold, and the segments are **equal-count**:
+each holds the same number of observations, regardless of how much time they
+cover. For bursty log sources that is misleading — a variable that changed
+constantly during a quiet night and then went silent under a flood of daytime
+traffic looks stable, because the flood supplies enough samples to dominate the
+later segments. Enabling `time` cuts the same four segments at **equal
+durations** instead, so each segment covers the same amount of wall-clock time;
+the detector then needs an event time per record, which it reads from the log's
+named variables (`logFormatVariables`, i.e. the fields declared in the parser's
+`log_format`) under the name given by `timestamp_variable`. `slope_index` and
+`slope_time` ask a different question — whether the change centroid sits early
+or late in the series — on the index axis and the time axis respectively.
+
+These parameters live on every `VariableDetector` subclass (`NewValueDetector`,
 `NewValueComboDetector`, `ValueRangeDetector`, `CharsetDetector`, `BigramDetector`, …)
-and go in the detector's top-level `params` block:
+and go in the detector's `auto_config_params` block — they are inputs to the
+auto-configuration phase, read only while `auto_config` is `True`, and never
+consulted at detection time.
 
 ```yaml
 detectors:
   NewValueDetector:
     method_type: new_value_detector
     auto_config: True
-    params:
-      stability_segmentation: time
-      timestamp_variable: Time      # a field name from the parser's log_format
-      timestamp_format: "%y%m%d %H%M%S"   # optional; omit to auto-detect
+    auto_config_params:
+      use_stable_vars: True
+      use_static_vars: True
+      classification:
+        index: True            # segment-mean thresholds, equal-count cuts
+        time: False            # segment-mean thresholds, equal-duration cuts
+        slope_index: False     # change centroid over index positions
+        slope_time: False      # change centroid over normalized time
+        slope_threshold: -0.05 # shared by both slope methods
+        decision: consensus    # consensus | majority
+      timestamp_variable: Time
+      timestamp_format: "%y%m%d %H%M%S"
 ```
 
-Setting `stability_segmentation: both` runs *both* segmentations and calls the variable
-stable only when each one does. Neither segmentation subsumes the other — a variable that
-churns in a burst and then settles is unstable by count but stable by time, and one whose
-late churn is buried under a dense tail of repeats is the reverse — so `both` is strictly
-stricter than either. Use it when a false "stable" is more costly than a missed one; use
-`time` when the point is specifically to forgive early churn on a bursty source.
+Defaults reproduce the historical behaviour exactly: `index: True`, the other
+three `False`, `decision: consensus`, `slope_threshold: -0.05`. A config that
+sets nothing under `classification` classifies identically to before this change.
+
+#### The decision rule
+
+When more than one method is enabled, `decision` picks how their verdicts
+combine. `consensus` requires every enabled method to return stable; `majority`
+requires strictly more than half of them to.
+
+| enabled | `consensus` needs | `majority` needs | differ? |
+|---|---|---|---|
+| 1 | 1/1 | 1/1 | no |
+| 2 | 2/2 | 2/2 (a 1–1 tie is UNSTABLE) | no |
+| 3 | 3/3 | 2/3 | yes |
+| 4 | 4/4 | 3/4 (a 2–2 tie is UNSTABLE) | yes |
+
+Ties resolve to UNSTABLE. That keeps `majority` from ever being more lenient
+than a coin-flip, and makes it collapse onto `consensus` at one and two enabled
+methods — turning a third method on is the only place the rule starts to matter.
+
+**All four methods false is a config error**, rejected by a pydantic validator.
+It is not a harmless no-op: classification decides `INSUFFICIENT_DATA`,
+`STATIC` and `RANDOM` before any method is consulted, so a method-less config
+would silently classify every remaining variable `STABLE`.
 
 #### Fields
 
+All of these live in the detector's `auto_config_params` block.
+
 | Field | Type | Default | Description |
 |---|---|---|---|
-| `stability_segmentation` | `"count" \| "time" \| "both"` | `"count"` | How to cut the change history into segments. `count` uses equal sample counts; `time` uses equal time spans; `both` requires the variable to be stable under each. With `count` the other two fields are ignored and no timestamps are recorded. |
-| `timestamp_variable` | `str \| null` | `null` | Name of the field in `logFormatVariables` holding the record's event time. Required for `time` and `both` to have any effect. Only named log-format fields are consulted — never the positional `variables` list. |
+| `use_stable_vars` | `bool` | `true` | Include variables classified `STABLE` in the generated configuration. |
+| `use_static_vars` | `bool` | `true` | Include variables classified `STATIC`. Defaults to `false` on `NewValueComboDetector`. |
+| `classification` | `ClassificationMethods` | see below | Which classification methods run and how their verdicts combine. |
+| `timestamp_variable` | `str \| null` | `null` | Name of the field in `logFormatVariables` holding the record's event time. Required for `time` and `slope_time` to have any effect. Only named log-format fields are consulted — never the positional `variables` list. |
 | `timestamp_format` | `str \| null` | `null` | Explicit [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) pattern for parsing that field. When unset, `TimeFormatHandler` auto-detects the format (ISO 8601, Apache, syslog, numeric epoch seconds/milliseconds, and other common layouts). |
 
 Set `timestamp_format` when the source uses a layout the auto-detection does not
 know. The HDFS loghub corpus, for example, stamps records as `081109 203615`, which
 only parses with an explicit `"%y%m%d %H%M%S"`.
 
+`classification`'s six fields:
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `index` | `bool` | `true` | Segment-mean thresholds, equal-count boundaries. |
+| `time` | `bool` | `false` | Segment-mean thresholds, equal-duration boundaries. Needs `timestamp_variable`. |
+| `slope_index` | `bool` | `false` | Change centroid vs. `slope_threshold`, measured on index positions. |
+| `slope_time` | `bool` | `false` | Change centroid vs. `slope_threshold`, measured on normalized timestamps. Needs `timestamp_variable`. |
+| `slope_threshold` | `float` | `-0.05` | The change-centroid cut-off both slope methods compare against, on a shared `[-0.5, +0.5]` scale. A variable passes when its centroid is at or below this value. |
+| `decision` | `"consensus" \| "majority"` | `"consensus"` | How verdicts from more than one enabled method combine; see above. |
+
 #### Fallback behaviour
 
-Time-aware segmentation is best-effort and never fails a run:
+Time-aware classification is best-effort and never fails a run:
 
-* If `stability_segmentation` is not `count` but `timestamp_variable` is unset, or the named field
-  is absent from a record, or its value cannot be parsed, the detector logs a
+* If `time` or `slope_time` is enabled but `timestamp_variable` is unset, or the named
+  field is absent from a record, or its value cannot be parsed, the detector logs a
   **single** warning (once per detector, so a bad config cannot flood the log) and
-  falls back to count-based segmentation.
+  falls back to the index axis.
 * If timestamps stop lining up with the recorded observations, or the observed time
-  span is zero, or they arrive out of order, the classifier silently falls back to
-  count-based segmentation for that variable.
-* Under `both`, any of the fallbacks above make the time pass reuse the count boundaries,
-  so the mode degrades to plain `count` rather than to an unconditional pass.
+  span is zero, or they arrive out of order, `time` silently reuses the equal-index
+  cuts, and `slope_time` computes its centroid on the index axis instead — it
+  degrades to `slope_index`.
+* Under `majority`, a fallen-back method still casts its own vote: if `slope_index`
+  and `slope_time` are both enabled and timestamps are unusable, both entries compute
+  the same index-axis centroid, and that verdict carries two of the votes rather than
+  one. This is deliberate — dropping a fallen-back method from the vote would change
+  the enabled count from variable to variable and make `majority` mean something
+  different for each one. The reason string names the axis each slope actually used,
+  so a doubled vote is visible in the note.
+* The same doubling applies to the segment-threshold pair: if `index` and `time` are
+  both enabled and timestamps are unusable, `time` silently reuses the same equal-count
+  cuts as `index`, so an identical verdict again carries two votes under `majority`
+  rather than one. Unlike the slope pair, the reason string does not surface this —
+  each entry is still labelled by its configured method name (`index` or `time`), not
+  by the axis it actually used, so a doubled segment-pair vote is invisible in the note.
 
 In every fallback case classification still runs and produces a result — only the
-segmentation rule changes back to the default.
+axis behind it changes back to index.
 
 A segment with no observations in it is *not* a fallback: it scores a mean of 0.0,
 because nothing observed means nothing changed. Equal-duration cuts of a bursty
 variable leave such segments routinely, so `time` on its own is lenient towards a
-burst of churn followed by silence. Use `both` when that leniency matters — the
-count pass keeps every segment populated.
+burst of churn followed by silence. Enable `index` and `time` together when that
+leniency matters — the index pass keeps every segment populated.
 
 
 ### Saving state (persist)

docs/detectors/charset.md

@@ -1,4 +1,4 @@
-# New Value Detector
+# Charset Detector
 
 The Charset Detector raises alerts when previously unseen characters appear in configured fields. It is useful to detect novelty, configuration drift, or the appearance of new actors in the environment.
 

docs/detectors/combo.md

@@ -17,7 +17,7 @@ detectors:
     NewValueComboDetector:
         method_type: new_value_combo_detector
         auto_config: False
-        params:
+        auto_config_params:
             max_combo_size: 3
         events:
             1:

docs/detectors/ecvc_detector.md

@@ -11,6 +11,8 @@ The Event Count Vector Clustering Detector (ECVC) detects anomalies by calculati
 
 A count vector is form by counting the number of appearance of each event ID in a sequence of a specific window size.
 
+Count vectors learned during training are stored via [persistency](../auxiliar/persistency.md), so a trained model can be saved and restored with a `persist:` block. A count vector is only comparable within the window it was counted over, so restoring state at a different `window_size` logs a warning — the restored vectors cannot match and every window would alert.
+
 
 ## Configuration example
 

docs/detectors/event_sequence.md

@@ -44,14 +44,19 @@ detectors:
         method_type: event_sequence_detector
         auto_config: True
         data_use_configure: 500
-        params:
+        auto_config_params:
             min_window_size: 2
             max_window_size: 10
 ```
 
 | Parameter | Default | Description |
 |---|---|---|
-| `fixed_window_size` | `None` | Length of the sliding event-ID window. Overrides `min_window_size`/`max_window_size` and skips auto configuration. Auto configuration writes its own choice here. While it is `None` the detector neither trains nor alerts. Must be `>= 1`. |
+| `fixed_window_size` | `None` | Length of the sliding event-ID window. Overrides the `auto_config_params` window range and skips auto configuration. Auto configuration writes its own choice here. While it is `None` the detector neither trains nor alerts. Must be `>= 1`. |
+
+#### `auto_config_params`
+
+| Field | Default | Description |
+|---|---|---|
 | `min_window_size` | `2` | Shortest window length tried during auto configuration. Must be `>= 1`. |
 | `max_window_size` | `10` | Longest window length tried during auto configuration. Must be `>= min_window_size`. |
 

docs/detectors/scvs_detector.md

@@ -11,6 +11,8 @@ The Sequence Count Vector Set Detector (SCVS) detects anomalies by finding count
 
 A count vector is formed by counting the number of appearance of each event ID in a sequence of a specific window size.
 
+Count vectors learned during training are stored via [persistency](../auxiliar/persistency.md), so a trained model can be saved and restored with a `persist:` block. A count vector is only comparable within the window it was counted over, so restoring state at a different `window_size` logs a warning — the restored vectors cannot match and every window would alert.
+
 
 ## Configuration example