Add a smart storage logic to enable configuration and training to use the same data

[viktorbeck98]

Configuration and training are not able to use the same data currently in case of a data stream, which would be preferred. Data that is used in the configuration phase will be lost otherwise.

[ernstleierzopf]

I think this issue from February is deprecated. Looking at the following test, we use the same data for the configuration phase as for the training phase:

class TestNewValueDetectorEndToEnd:
    """Regression test: full configure/train/detect pipeline on audit.log."""

    def test_audit_log_anomalies(self):
        parser = MatcherParser(config=_PARSER_CONFIG)
        detector = NewValueDetector()

        logs = list(From.log(parser, in_path=AUDIT_LOG, do_process=True))

        for log in logs[:TRAIN_UNTIL]:
            detector.configure(log)
        detector.set_configuration()

        for log in logs[:TRAIN_UNTIL]:
            detector.train(log)

        detected_ids: set[str] = set()
        for log in logs[TRAIN_UNTIL:]:
            output = schemas.DetectorSchema()
            if detector.detect(log, output_=output):
                detected_ids.add(log["logID"])

        assert detected_ids == {'1859', '1860', '1861', '1862', '1864', '1865', '1866', '1867'}

Please reopen if this issue is still relevant.

[viktorbeck98]

@ernstleierzopf you misunderstand this issue. In a stream it is not possible to use the same training data because it is not saved anywhere

[ernstleierzopf]

@viktorbeck98 okay, if I understand it correctly, we are not able to train until the configuration phase is finished (we need the config to be set in train). Then our only option is to store the streamed data and reuse it later? I see no easy way to get the data out of auto_conf_persistency.

I have a few questions and points to clarify the issue, as I looked into the codebase.

• The current implementation does not offer functionality for streams yet. At least I do not see it and there are no unittests for this. If there is a way please reference the important classes and we should create unittests for this.

• The data needs to be preserved only when using auto_config: True, otherwise the user can easily create his own buffer, because he sees that the data is lost.

• Even though you are allowed to keep data_use_configure and data_use_training None, this brings the FitLogic into a NOTHING state. So it is expected to set these values. With data_use_configure there is no configuration stage -> no data needs to be kept. If both variables are set, you know exactly how much data needs to be kept (data_use_configure).

  As I am writing this, I see that in the _fit_logic.py file there is a finish_config method after which the previously stored config data could be immediately used for training. This method is only used in the core.py file. There the self.buffer_train variable (TrainBuffer()) is used to keep the data used for configuration.

I think this problem does not exist, it is already solved in code.
Only when you do not use auto_config: True, then you need to keep you data buffered on your own. But I think that is the point of not using auto configuration. In that case you need to write code anyways and can easily keep track of the data on your own.

@ipmach I think you wrote the code, do you agree on my analysis of this issue? I figured it literally out while thinking about solutions for this issue.

[viktorbeck98]

The current implementation does not offer functionality for streams yet. At least I do not see it and there are no unittests for this. If there is a way please reference the important classes and we should create unittests for this.

• Well, kind of it does but we have not tested the whole pipeline with streaming, that is correct. One thing that is open e.g. is the dependency of parsing and anomaly detection -> what happens when a pared template changed, how will this affect the detector during operation/training?

The data needs to be preserved only when using auto_config: True, otherwise the user can easily create his own buffer, because he sees that the data is lost.

• Correct. We have planned a buffer component, that @whotwagner is working on

Even though you are allowed to keep data_use_configure and data_use_training None, this brings the FitLogic into a NOTHING state. So it is expected to set these values. With data_use_configure there is no configuration stage -> no data needs to be kept. If both variables are set, you know exactly how much data needs to be kept (data_use_configure).

• I will leave it to @ipmach to answer this