DeepAI AI Chat
Log In Sign Up

AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data

by   Nguyen Ho, et al.

Recent development in computing, sensing and crowd-sourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called Big Data to inform research and the decision-making process are virtually endless. In general, analyses have to be done across multiple data sets in order to bring out the most value of Big Data. A first important step is to identify temporal correlations between data sets. Given the characteristics of Big Data in terms of volume and velocity, techniques that identify correlations not only need to be fast and scalable, but also need to help users in ordering the correlations across temporal scales so that they can focus on important relationships. In this paper, we present AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual information to identify correlations at multiple temporal scales in large time series. Discovered correlations are suggested to users in an order based on the strength of the relationships. Our method supports an adaptive streaming technique that minimizes duplicated computation and is implemented on top of Apache Spark for scalability. We also provide a comprehensive evaluation on the effectiveness and the scalability of AMIC using both synthetic and real-world data sets.


page 1

page 18


A Unified Approach for Multi-Scale Synchronous Correlation Search in Big Time Series – Full Version

The wide deployment of IoT sensors has enabled the collection of very bi...

Toeplitz Least Squares Problems, Fast Algorithms and Big Data

In time series analysis, when fitting an autoregressive model, one must ...

Multifractality in time series is due to temporal correlations

Based on the rigorous mathematical arguments formulated within the Multi...

Anomaly Detection for Industrial Big Data

As the Industrial Internet of Things (IIoT) grows, systems are increasin...

On the Scalability of Big Data Cyber Security Analytics Systems

Big Data Cyber Security Analytics (BDCA) systems use big data technologi...

Information sensitivity functions to assess parameter information gain and identifiability of dynamical systems

A new class of functions, called the `Information sensitivity functions'...