An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination

by   Minkyung Kim, et al.

Most deep anomaly detection models are based on learning normality from datasets due to the difficulty of defining abnormality by its diverse and inconsistent nature. Therefore, it has been a common practice to learn normality under the assumption that anomalous data are absent in a training dataset, which we call normality assumption. However, in practice, the normality assumption is often violated due to the nature of real data distributions that includes anomalous tails, i.e., a contaminated dataset. Thereby, the gap between the assumption and actual training data affects detrimentally in learning of an anomaly detection model. In this work, we propose a learning framework to reduce this gap and achieve better normality representation. Our key idea is to identify sample-wise normality and utilize it as an importance weight, which is updated iteratively during the training. Our framework is designed to be model-agnostic and hyperparameter insensitive so that it applies to a wide range of existing methods without careful parameter tuning. We apply our framework to three different representative approaches of deep anomaly detection that are classified into one-class classification-, probabilistic model-, and reconstruction-based approaches. In addition, we address the importance of a termination condition for iterative methods and propose a termination criterion inspired by the anomaly detection objective. We validate that our framework improves the robustness of the anomaly detection models under different levels of contamination ratios on five anomaly detection benchmark datasets and two image datasets. On various contaminated datasets, our framework improves the performance of three representative anomaly detection methods, measured by area under the ROC curve.


page 1

page 13


Image anomaly detection with capsule networks and imbalanced datasets

Image anomaly detection consists in finding images with anomalous, unusu...

Unsupervised Deep One-Class Classification with Adaptive Threshold based on Training Dynamics

One-class classification has been a prevailing method in building deep a...

Early Anomaly Detection by Learning and Forecasting Behavior

Graph anomaly detection systems aim at identifying suspicious accounts o...

Towards Anomaly Detection in Dashcam Videos

Inexpensive sensing and computation, as well as insurance innovations, h...

Image Anomalies: a Review and Synthesis of Detection Methods

We review the broad variety of methods that have been proposed for anoma...

FlexFringe: Modeling Software Behavior by Learning Probabilistic Automata

We present the efficient implementations of probabilistic deterministic ...

Testing for Typicality with Respect to an Ensemble of Learned Distributions

Methods of performing anomaly detection on high-dimensional data sets ar...

Please sign up or login with your details

Forgot password? Click here to reset