Robust real-time monitoring of high-dimensional data streams

06/05/2019
by   Ruizhi Zhang, et al.
0

Robust real-time monitoring of high-dimensional data streams has many important real-world applications such as industrial quality control, signal detection, biosurveillance, but unfortunately it is highly non-trivial to develop efficient schemes due to two challenges: (1) the unknown sparse number or subset of affected data streams and (2) the uncertainty of model specification for high-dimensional data. In this article, motivated by the detection of smaller persistent changes in the presence of larger transient outliers, we develop a family of efficient real-time robust detection schemes for high-dimensional data streams through monitoring feature spaces such as PCA or wavelet coefficients when the feature coefficients are from Tukey-Huber's gross error models with outliers. We propose to construct a new local detection statistic for each feature called L_α-CUSUM statistic that can reduce the effect of outliers by using the Box-Cox transformation of the likelihood function, and then raise a global alarm based upon the sum of the soft-thresholding transformation of these local L_α-CUSUM statistics so that to filter out unaffected features. In addition, we propose a new concept called false alarm breakdown point to measure the robustness of online monitoring schemes, and also characterize the breakdown point of our proposed schemes. Asymptotic analysis, extensive numerical simulations and case study of nonlinear profile monitoring are conducted to illustrate the robustness and usefulness of our proposed schemes.

READ FULL TEXT
research
12/15/2017

Efficient Global Monitoring Statistics for High-Dimensional Data

Global monitoring statistics play an important role for developing effic...
research
09/22/2020

Partially Observable Online Change Detection via Smooth-Sparse Decomposition

We consider online change detection of high dimensional data streams wit...
research
12/14/2017

A Two-stage Online Monitoring Procedure for High-Dimensional Data Streams

Advanced computing and data acquisition technologies have made possible ...
research
06/23/2022

Sequential Detection of Transient Signals in High Dimensional Data Stream

Motivated by sequential detection of transient signals in high dimension...
research
09/29/2021

Dynamic probabilistic predictable feature analysis for high dimensional temporal monitoring

Dynamic statistical process monitoring methods have been widely studied ...
research
09/14/2018

Real-Time Nonparametric Anomaly Detection in High-Dimensional Settings

Timely and reliable detection of abrupt anomalies, e.g., faults, intrusi...
research
07/18/2022

Lightweight Automated Feature Monitoring for Data Streams

Monitoring the behavior of automated real-time stream processing systems...

Please sign up or login with your details

Forgot password? Click here to reset