CFOF: A Concentration Free Measure for Anomaly Detection

01/14/2019
by   Fabrizio Angiulli, et al.
0

We present a novel notion of outlier, called the Concentration Free Outlier Factor, or CFOF. As a main contribution, we formalize the notion of concentration of outlier scores and theoretically prove that CFOF does not concentrate in the Euclidean space for any arbitrary large dimensionality. To the best of our knowledge, there are no other proposals of data analysis measures related to the Euclidean distance for which it has been provided theoretical evidence that they are immune to the concentration effect. We determine the closed form of the distribution of CFOF scores in arbitrarily large dimensionalities and show that the CFOF score of a point depends on its squared norm standard score and on the kurtosis of the data distribution, thus providing a clear and statistically founded characterization of this notion. Moreover, we leverage this closed form to provide evidence that the definition does not suffer of the hubness problem affecting other measures. We prove that the number of CFOF outliers coming from each cluster is proportional to cluster size and kurtosis, a property that we call semi-locality. We determine that semi-locality characterizes existing reverse nearest neighbor-based outlier definitions, thus clarifying the exact nature of their observed local behavior. We also formally prove that classical distance-based and density-based outliers concentrate both for bounded and unbounded sample sizes and for fixed and variable values of the neighborhood parameter. We introduce the fast-CFOF algorithm for detecting outliers in large high-dimensional dataset. The algorithm has linear cost, supports multi-resolution analysis, and is embarrassingly parallel. Experiments highlight that the technique is able to efficiently process huge datasets and to deal even with large values of the neighborhood parameter, to avoid concentration, and to obtain excellent accuracy.

READ FULL TEXT

page 1

page 18

research
08/10/2022

SSDBCODI: Semi-Supervised Density-Based Clustering with Outliers Detection Integrated

Clustering analysis is one of the critical tasks in machine learning. Tr...
research
12/05/2019

Causal structure based root cause analysis of outliers

We describe a formal approach to identify 'root causes' of outliers obse...
research
06/23/2021

Closed-Form, Provable, and Robust PCA via Leverage Statistics and Innovation Search

The idea of Innovation Search, which was initially proposed for data clu...
research
02/21/2017

Interpreting Outliers: Localized Logistic Regression for Density Ratio Estimation

We propose an inlier-based outlier detection method capable of both iden...
research
12/05/2022

AIDA: Analytic Isolation and Distance-based Anomaly Detection Algorithm

We combine the metrics of distance and isolation to develop the Analytic...
research
06/14/2019

Detecting Network Soft-failures with the Network Link Outlier Factor (NLOF)

In this paper, we describe and experimentally evaluate the performance o...
research
02/21/2019

Continuous Outlier Mining of Streaming Data in Flink

In this work, we focus on distance-based outliers in a metric space, whe...

Please sign up or login with your details

Forgot password? Click here to reset