AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection

06/30/2022
by   Marius Drăgoi, et al.
93

Analyzing the distribution shift of data is a growing research direction in nowadays Machine Learning, leading to emerging new benchmarks that focus on providing a suitable scenario for studying the generalization properties of ML models. The existing benchmarks are focused on supervised learning, and to the best of our knowledge, there is none for unsupervised learning. Therefore, we introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. This kind of data meets the premise of shifting the input distribution: it covers a large time span (10 years), with naturally occurring changes over time (users modifying their behavior patterns, and software updates). We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. Next, we propose AnoShift, a protocol splitting the data in IID, NEAR, and FAR testing splits. We validate the performance degradation over time with diverse models (MLM to classical Isolation Forest). Finally, we show that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical IID training (by up to 3%, on average). Dataset and code are available at https://github.com/bit-ml/AnoShift/.

READ FULL TEXT

page 6

page 7

08/30/2021

SHIFT15M: Multiobjective Large-Scale Fashion Dataset with Distributional Shifts

Many machine learning algorithms assume that the training data and the t...
10/13/2021

Challenges for Unsupervised Anomaly Detection in Particle Physics

Anomaly detection relies on designing a score to determine whether a par...
04/27/2021

Extending Isolation Forest for Anomaly Detection in Big Data via K-Means

Industrial Information Technology (IT) infrastructures are often vulnera...
08/11/2020

BREEDS: Benchmarks for Subpopulation Shift

We develop a methodology for assessing the robustness of models to subpo...
07/21/2020

Interpretable Anomaly Detection with DIFFI: Depth-based Feature Importance for the Isolation Forest

Anomaly Detection is one of the most important tasks in unsupervised lea...
05/31/2022

Rethinking Graph Neural Networks for Anomaly Detection

Graph Neural Networks (GNNs) are widely applied for graph anomaly detect...
07/31/2021

Online unsupervised Learning for domain shift in COVID-19 CT scan datasets

Neural networks often require large amounts of expert annotated data to ...