Anomaly detection using principles of human perception
share
In the fields of statistics and unsupervised machine learning a fundamental and well-studied problem is anomaly detection. Although anomalies are difficult to define, many algorithms have been proposed. Underlying the approaches is the nebulous understanding that anomalies are rare, unusual or inconsistent with the majority of data. The present work gives a philosophical approach to clearly define anomalies and to develop an algorithm for their efficient detection with minimal user intervention. Inspired by the Gestalt School of Psychology and the Helmholtz principle of human perception, the idea is to assume anomalies are observations that are unexpected to occur with respect to certain groupings made by the majority of the data. Thus, under appropriate random variable modelling anomalies are directly found in a set of data under a uniform and independent random assumption of the distribution of constituent elements of the observations; anomalies correspond to those observations where the expectation of occurrence of the elements in a given view is <1. Starting from fundamental principles of human perception an unsupervised anomaly detection algorithm is developed that is simple, real-time and parameter-free. Experiments suggest it as the prime choice for univariate data and it shows promising performance on the detection of global anomalies in multivariate data.
READ FULL TEXT