On the Runtime-Efficacy Trade-off of Anomaly Detection Techniques for Real-Time Streaming Data

10/12/2017
by   Dhruv Choudhary, et al.
0

Ever growing volume and velocity of data coupled with decreasing attention span of end users underscore the critical need for real-time analytics. In this regard, anomaly detection plays a key role as an application as well as a means to verify data fidelity. Although the subject of anomaly detection has been researched for over 100 years in a multitude of disciplines such as, but not limited to, astronomy, statistics, manufacturing, econometrics, marketing, most of the existing techniques cannot be used as is on real-time data streams. Further, the lack of characterization of performance -- both with respect to real-timeliness and accuracy -- on production data sets makes model selection very challenging. To this end, we present an in-depth analysis, geared towards real-time streaming data, of anomaly detection techniques. Given the requirements with respect to real-timeliness and accuracy, the analysis presented in this paper should serve as a guide for selection of the "best" anomaly detection technique. To the best of our knowledge, this is the first characterization of anomaly detection techniques proposed in very diverse set of fields, using production data sets corresponding to a wide set of application domains.

READ FULL TEXT

page 5

page 10

page 11

research
07/08/2016

Real-Time Anomaly Detection for Streaming Analytics

Much of the worlds data is streaming, time-series data, where anomalies ...
research
03/03/2022

Anomaly Detection in Big Data

Anomaly is defined as a state of the system that do not conform to the n...
research
02/25/2019

Anomaly Detection for an E-commerce Pricing System

Online retailers execute a very large number of price updates when compa...
research
12/20/2018

An Evaluation of Methods for Real-Time Anomaly Detection using Force Measurements from the Turning Process

We examined the use of three conventional anomaly detection methods and ...
research
03/08/2020

Hardware Architecture Proposal for TEDA algorithm to Data Streaming Anomaly Detection

The amount of data in real-time, such as time series and streaming data,...
research
09/04/2023

Drifter: Efficient Online Feature Monitoring for Improved Data Integrity in Large-Scale Recommendation Systems

Real-world production systems often grapple with maintaining data qualit...
research
12/25/2022

Anomaly Detection of Underwater Gliders Verified by Deployment Data

This paper utilizes an anomaly detection algorithm to check if underwate...

Please sign up or login with your details

Forgot password? Click here to reset