A Novel Algorithm for Optimized Real Time Anomaly Detection in Timeseries
share
Observations in data which are significantly different from its neighbouring points but cannot be classified as noise are known as anomalies or outliers. These anomalies are a cause of concern and a timely warning about their presence could be valuable. In this paper, we have evaluated and compared the performance of popular algorithms from domains of Machine Learning and Statistics in detecting anomalies on both offline data as well as real time data. Our aim is to come up with an algorithm which can handle all types of seasonal and non-seasonal data effectively and is fast enough to be of practical utility in real time. It is not only important to detect anomalies at the global but also the ones which are anomalies owing to their local surroundings. Such outliers can be termed as contextual anomalies as they derive their context from the neighbouring observations. Also, we require a methodology to automatically determine the presence of seasonality in the given data. For detecting the seasonality, the proposed algorithm takes up a curve fitting approach rather than model based anomaly detection. The proposed model also introduces a unique filter which assess the relative significance of local outliers and removes the ones deemed as insignificant. Since, the proposed model fits polynomial in buckets of timeseries data, it does not suffer from problems such as heteroskedasticity and breakout as compared to its statistical alternatives such as ARIMA, SARIMA and Winter Holt. Experimental results the proposed algorithm performs better on both real time as well as artificial generated datasets.
READ FULL TEXT
