Real-Time Nonparametric Anomaly Detection in High-Dimensional Settings

09/14/2018
by   Mehmet Necip Kurt, et al.
0

Timely and reliable detection of abrupt anomalies, e.g., faults, intrusions/attacks, is crucial for real-time monitoring and security of many modern systems such as the smart grid and the Internet of Things (IoT) networks that produce high-dimensional data. With this goal, we propose effective and scalable algorithms for real-time anomaly detection in high-dimensional settings. Our proposed algorithms are nonparametric (model-free) as both the nominal and anomalous multivariate data distributions are assumed to be unknown. We extract useful univariate summary statistics and perform the anomaly detection task in a single-dimensional space. We model anomalies as persistent outliers and propose to detect them via a cumulative sum (CUSUM)-like algorithm. In case the observed data stream has a low intrinsic dimensionality, we find a low-dimensional submanifold in which the nominal data are embedded and then evaluate whether the sequentially acquired data persistently deviate from the nominal submanifold. Further, in the general case, we determine an acceptance region for nominal data via the Geometric Entropy Minimization (GEM) method and then evaluate whether the sequentially observed data persistently fall outside the acceptance region. We provide an asymptotic lower bound on the average false alarm period of the proposed CUSUM-like algorithm. Moreover, we provide a sufficient condition to asymptotically guarantee that the decision statistic of the proposed algorithm does not diverge in the absence of anomalies. Numerical studies illustrate the effectiveness of the proposed schemes in quick and accurate detection of changes/anomalies in a variety of high-dimensional settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/17/2019

Online Multivariate Anomaly Detection and Localization for High-dimensional Settings

This paper considers the real-time detection of anomalies in high-dimens...
research
06/07/2020

A Novel Algorithm for Optimized Real Time Anomaly Detection in Timeseries

Observations in data which are significantly different from its neighbou...
research
10/29/2021

Autoencoder-based Anomaly Detection in Smart Farming Ecosystem

The inclusion of Internet of Things (IoT) devices is growing rapidly in ...
research
06/05/2019

Robust real-time monitoring of high-dimensional data streams

Robust real-time monitoring of high-dimensional data streams has many im...
research
11/23/2022

FRE: A Fast Method For Anomaly Detection And Segmentation

This paper presents a fast and principled approach for solving the visua...
research
12/24/2020

Am I Rare? An Intelligent Summarization Approach for Identifying Hidden Anomalies

Monitoring network traffic data to detect any hidden patterns of anomali...
research
02/19/2019

Secure Distributed Dynamic State Estimation in Wide-Area Smart Grids

Smart grid is a large complex network with a myriad of vulnerabilities, ...

Please sign up or login with your details

Forgot password? Click here to reset