Create EntropyDetector (new)

[ernstleierzopf]

https://stackoverflow.com/questions/2979174/how-do-i-compute-the-approximate-entropy-of-a-bit-string

Additional, potentially useful ideas from ChatGPT (interpreted by me):

• Entropy on event occurrence: for example a window-based approach checking frequencies of events. (Edit: this is the EventFrequencyDetector)
• Inside of one event with multiple variables (entropy between words): is there a low entropy (always similar/same log) or high entropy (values are always different) (Edit: this is also the EventFrequencyDetector)
• Per variable: learn unigrams / n-grams of the values inside the values. But here you cannot simply learn the average or median over 1000 logs, because this loses rare events. You could however store all frequencies in the persistency (one per event) and then compare if the log matches any of the learned frequencies (within the threshold). (Edit: Variables usually hold only one word and therefore frequencies of characters are not representative. You need a longer text for that.)

These are some ideas for future detectors that might be implemented.

[ernstleierzopf]

also add files to
detectors/init.py
docs/detectors/detectorname.md
docs/detectors.md
config/pipeline_config_default.yaml

[ernstleierzopf]

Idea for ShannonEntropyDetector:
The ShannonEntropyDetector must be a window based detector.

1. It calculates the entropy for each variable over a specified window.
2. Calculate entropy for combinations of variables over a specified window.
3. Normalize all entropy values, because a large dataset naturally can have a higher entropy than a small dataset.
4. Learn a baseline of mean entropy and standard deviation; Calculate a z-score to find anomalies. Make sure to find increasing entropies, but also decreasing entropies which could also be an anomaly.
5. Add a combinations parameter which limits the surveilled combinations; otherwise the detector might become too expensive and noisy.
6. Use sliding windows to get a time series of entropy. This allows us to calculate new entropies and find anomalies for every new event or every N events.
7. Windows can either be based on time or number of events (logs).
8. Possibly allow multiple window sizes. Larger windows can have lower standard deviations and are more precise, however they lag behind. Smaller window sizes might indicate these anomalies.
9. Define a parameter for the z-score threshold.