Towards a Rigorous Evaluation of Time-series Anomaly Detection

09/11/2021
by   Siwon Kim, et al.
0

In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets, giving the impression of clear improvements. However, most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; that is, even a random anomaly score can easily turn into a state-of-the-art TAD method. Therefore, the comparison of TAD methods with F1 scores after the PA protocol can lead to misguided rankings. Furthermore, we question the potential of existing TAD methods by showing that an untrained model obtains comparable detection performance to the existing methods even without PA. Based on our findings, we propose a new baseline and an evaluation protocol. We expect that our study will help a rigorous evaluation of TAD and lead to further improvement in future researches.

READ FULL TEXT

page 1

page 2

page 3

page 4

page 5

page 6

page 7

page 8

research
05/15/2023

Evaluation Strategy of Time-series Anomaly Detection with Decay Function

Recent algorithms of time-series anomaly detection have been evaluated b...
research
12/06/2022

Unsupervised Anomaly Detection in Time-series: An Extensive Evaluation and Analysis of State-of-the-art Methods

Unsupervised anomaly detection in time-series has been extensively inves...
research
09/23/2021

An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series

Several techniques for multivariate time series anomaly detection have b...
research
04/15/2021

OneLog: Towards End-to-End Training in Software Log Anomaly Detection

In recent years, with the growth of online services and IoT devices, sof...
research
08/24/2023

Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology

Multivariate Time Series (MVTS) anomaly detection is a long-standing and...
research
12/15/2020

Anomaly Detection and Localization based on Double Kernelized Scoring and Matrix Kernels

Anomaly detection is necessary for proper and safe operation of large-sc...
research
02/15/2019

Street Scene: A new dataset and evaluation protocol for video anomaly detection

Progress in video anomaly detection research is currently slowed by smal...

Please sign up or login with your details

Forgot password? Click here to reset