Log In Sign Up

Expected Similarity Estimation for Large-Scale Batch and Streaming Anomaly Detection

by   Markus Schneider, et al.

We present a novel algorithm for anomaly detection on very large datasets and data streams. The method, named EXPected Similarity Estimation (EXPoSE), is kernel-based and able to efficiently compute the similarity between new data points and the distribution of regular data. The estimator is formulated as an inner product with a reproducing kernel Hilbert space embedding and makes no assumption about the type or shape of the underlying data distribution. We show that offline (batch) learning with EXPoSE can be done in linear time and online (incremental) learning takes constant time per instance and model update. Furthermore, EXPoSE can make predictions in constant time, while it requires only constant memory. In addition, we propose different methodologies for concept drift adaptation on evolving data streams. On several real datasets we demonstrate that our approach can compete with state of the art algorithms for anomaly detection while being an order of magnitude faster than most other approaches.


page 1

page 2

page 3

page 4


Adaptive Model Pooling for Online Deep Anomaly Detection from a Complex Evolving Data Stream

Online anomaly detection from a data stream is critical for the safety a...

InQMAD: Incremental Quantum Measurement Anomaly Detection

Streaming anomaly detection refers to the problem of detecting anomalous...

Real-time Anomaly Detection for Multivariate Data Streams

We present a real-time multivariate anomaly detection algorithm for data...

Supervised Anomaly Detection in Uncertain Pseudoperiodic Data Streams

Uncertain data streams have been widely generated in many Web applicatio...

Isolation Distributional Kernel: A New Tool for Point Group Anomaly Detection

We introduce Isolation Distributional Kernel as a new way to measure the...

Exact Optimization of Conformal Predictors via Incremental and Decremental Learning

Conformal Predictors (CP) are wrappers around ML methods, providing erro...

Concept Drift Challenge in Multimedia Anomaly Detection: A Case Study with Facial Datasets

Anomaly detection in multimedia datasets is a widely studied area. Yet, ...