Sub-clusters of Normal Data for Anomaly Detection

11/17/2020
by   Gahye Lee, et al.
26

Anomaly detection in data analysis is an interesting but still challenging research topic in real world applications. As the complexity of data dimension increases, it requires to understand the semantic contexts in its description for effective anomaly characterization. However, existing anomaly detection methods show limited performances with high dimensional data such as ImageNet. Existing studies have evaluated their performance on low dimensional, clean and well separated data set such as MNIST and CIFAR-10. In this paper, we study anomaly detection with high dimensional and complex normal data. Our observation is that, in general, anomaly data is defined by semantically explainable features which are able to be used in defining semantic sub-clusters of normal data as well. We hypothesize that if there exists reasonably good feature space semantically separating sub-clusters of given normal data, unseen anomaly also can be well distinguished in the space from the normal data. We propose to perform semantic clustering on given normal data and train a classifier to learn the discriminative feature space where anomaly detection is finally performed. Based on our careful and extensive experimental evaluations with MNIST, CIFAR-10, and ImageNet with various combinations of normal and anomaly data, we show that our anomaly detection scheme outperforms state of the art methods especially with high dimensional real world images.

READ FULL TEXT

page 1

page 3

page 5

page 10

research
09/28/2021

Anomaly Detection for High-Dimensional Data Using Large Deviations Principle

Most current anomaly detection methods suffer from the curse of dimensio...
research
12/05/2021

Simple Adaptive Projection with Pretrained Features for Anomaly Detection

Deep anomaly detection aims to separate anomaly from normal samples with...
research
05/12/2020

Unsupervised Anomaly Detection via Deep Metric Learning with End-to-End Optimization

We investigate unsupervised anomaly detection for high-dimensional data ...
research
03/17/2019

Learning Competitive and Discriminative Reconstructions for Anomaly Detection

Most of the existing methods for anomaly detection use only positive dat...
research
12/21/2021

Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

We introduce anomaly clustering, whose goal is to group data into semant...
research
10/29/2018

Feature Bagging for Steganographer Identification

Traditional steganalysis algorithms focus on detecting the existence of ...
research
12/15/2020

Modeling Heterogeneous Statistical Patterns in High-dimensional Data by Adversarial Distributions: An Unsupervised Generative Framework

Since the label collecting is prohibitive and time-consuming, unsupervis...

Please sign up or login with your details

Forgot password? Click here to reset