share
I Introduction
Time series forecasting has received lots of attention recently. This is because many phenomena such as stock price, temperature, and weather can be modeled as time series. The fundamental challenge in time series forecasting is that the observations at different points in time are correlated, which makes some of the algorithms that change or permute the order of observations unusable.
Scientists and researchers have done extensive research in time series forecasting, such as [10, 28, 3, 8, 9]. They have borrowed tools from various domains, such as graphical modeling and statistics, to improve forecasting accuracy. For example, in [10] and [28]
, the authors used the hidden Markov model (HMM) to predict time series data. In
[28], the authors showed that the HMM can achieve high accuracy to predict student performance in an educational game. In [3, 8] and [9], the authors used the well-known time series method, namely, autoregressive integrated moving average (ARIMA) to make the forecast of the stock market, electricity prices and energy demands.Although classical methods like ARIMA and HMM proved to be successful in modeling time-series data, however, they exhibit some limitations as follows: 1- HMM follows the one step Markov assumption, which means conditioning on the current state the future state is independent of all the past states. This assumption might be violated for most real-world time series data. 2- HMM considers there are underlying hidden states that are responsible to generate the observations, which might not be correct for many time series forecasting problems. 3- ARIMA and most other classical time series models can only be used for univariate time series; furthermore, it is shown that they cannot model nonlinear time series accurately. [2, 32, 6].
Recent advancements in the processing power of the computers enable researchers to take advantage of the massive volume of data and develop more sophisticated algorithms such as neural networks to model and predict the data. Many researchers have used neural networks in the time series forecasting and showed that they can lead to better prediction than ARIMA model. [6, 32, 26].
Time series clustering is another major topic that receives lots of attention. Extensive research such as [1, 18] and [22] is done in developing accurate and efficient clustering algorithms. In [1] and [18], the authors performed the comprehensive surveys of the time series clustering methods. They described distance-based clustering methods such as dynamic time warping (DTW) as well as feature-based and representation based clustering methods. In [4]
, the authors combined the recurrent neural network, specifically long short term memory (LSTM) and feature-based clustering to improve the time series forecasting accuracy. In particular, they showed that their method outperforms the baseline LSTM model in terms of mean sMAPE accuracy.
Some papers such as [31, 17] and [13] made forecasting using the neural network by combining static and dynamic features. They showed that combining static and dynamic features can outperform the classification accuracy using the static or dynamic features alone.
In this paper, we are proposing to combine anomaly detection, clustering, and forecasting using LSTM for time series data to achieve better prediction. The contribution of this paper is fourfold. First, we perform anomaly detection to identify and replace the outliers in the time series data. This is a major preprocessing step that can significantly affect the prediction and clustering. Second, we introduce some methods to impute the missing values for time series data. Third, we compare distance-based and feature-based clustering in terms of speed and prediction accuracy. Fourth, we introduce multiple architectures to make forecasting using time series data as well as multiple architectures to combine static data and time series data.
The rest of this article is organized as follows: Section II defines the methods and algorithms employed in the later sections. Section III describes the dataset used in the article. Section IV describes the problem formulation. Section V presents and discusses the results. Finally, section VI presents our conclusion.
Ii Algorithms and Methods
Ii-a Anomaly Detection
Anomaly detection is an essential step before applying any machine learning algorithm. This step should be taken before imputing the missing values since the outliers will influence the missing values. Outliers are defined as observations that are significantly different from all other observations in data. If they are not detected and replaced correctly, they will have a negative impact on clustering and forecasting outcomes.
Several algorithms, such as [19] and [24]
are proposed to detect anomalies in time series using the ARMA process and the neural network, specifically LSTM. In this paper, we are using seasonality and trend decomposition based on locally estimated scatterplot smoothing (Loess) to detect the outliers in time series. This is the algorithm developed by Twitter and works by decomposing the time series to seasonality, trend, and random components. It is suitable for seasonal time series and is claimed to work accurately for additive outliers. An example of the implementation of this algorithm can be found in the tsclean package at R.
Ii-B Missing Value Imputation
The second major preprocessing step after detecting and replacing the outliers is imputing the missing values. Depending on the nature of data, several methods exist to impute the missing values. Following techniques are examples of missing value imputation for time series data:
| Time Series Data | |
|---|---|
| Interpolation | Linear, Spline, … |
| Locf | Last Observation Carry Forward |
| Nocb | Next Observation Carry Backward |
| Moving Average | Exponential,Linear, … |
| Mean | Mean, Median, … |
Table 1 lists only some of the well-known methods to impute missing values. It is also worth mentioning that the tsclean package mentioned in the last section performs linear interpolation to impute missing values after anomaly detection. Several algorithms are proposed to impute missing values for the static data such as [5, 29, 20, 11]. In [29]
the authors performed a comparative study of several missing value imputation techniques such as Singular Value Decomposition (SVD) based method (SVDimpute), weighted K-nearest neighbors (KNNimpute), and row average (filling missing values with zeros). They showed that KNNimpute provides a more robust and sensitive method than row average and SVDimpute. In
[20], the authors proposed the SOFT-IMPUTE algorithm to iteratively replace the missing elements with those obtained from a soft-thresholded SVD. The softImpute Package in R described In [11] implements Iterative methods for matrix completion that use nuclear-norm regularization.Ii-C Clustering
Clustering is defined as dividing the data into some groups such that data in the same groups are more similar. It is divided into two major categories of distance-based and feature-based clustering. In distance-based clustering, the distance between data points is considered as similarity measure, and the objective is to have lower intra-cluster distance than inter-cluster distance. The main challenge in distance-based clustering is the notion of the distance itself. Choosing the appropriate distance is dependent on the nature of the dataset. In the case of static and cross-sectional data, euclidean distance is the most popular distance metric, and K-mean and hierarchical algorithms are two popular clustering choices. Hierarchical clustering is divided into two categories of agglomerative and divisive. Agglomerative hierarchical clustering is a more popular choice than divisive hierarchical clustering in which each data point is initially in its own cluster; then, at each iteration, the similar clusters merge with each other until K clusters are formed. The advantage of hierarchical clustering over K-mean is that we do not need to specify the number of clusters in advance.
Dynamic time warping (DTW) is one of the most popular distance metrics used for time series clustering. It is a well-known technique that was originally developed for speech recognition application and is used to find an optimal alignment between two given time-dependent sequences. It is a dynamic programming algorithm that Unlike the Euclidean distance, is not susceptible to distortions in the time-axis. It works by warping the sequences in a nonlinear fashion to match each other [21]. Another advantage of DTW over euclidean distance is that it does not require the sequences to have the same lengths. However,despite its numerous applications, DTW suffers from quadratic computational complexity. This means if m and n represent the length of two sequences, then the computational complexity of finding the DTW distance between them is . Some papers such as [27, 25] proposed algorithms such as PrunedDTW, FastDTW to speedup the classical DTW algorithm.
Feature-based clustering works by first extracting features from data and then constructing the feature matrix from the extracted features. This method has several advantages over distance-based clustering. First, it is faster, as will be shown later in this paper. Second, standard clustering methods such as K-mean and hierarchical clustering can directly be applied to the feature matrix.
In this paper, we are introducing two feature extraction methods for time series data. Method A mainly extracts time series specific features such as entropy, autocorrelation, partial autocorrelation, stability, and holt parameter. These features showed to be very effective for clustering and detecting unusual time series
[30, 16].Method B looks at the time series more like a signal and extracts a more exhaustive list of features such as energy, fast Fourier transform (FFT) coefficients, continuous wavelet transform (CWT) coefficients, variance
[7].Ii-D Forecasting with Neural Networks
Neural networks are drawing lots of attention in recent years. They have been used successfully in various domains such as signal and image processing, control, biology, and finance. Different neural network architectures are used depending on the application and the dataset. For example, if the objective is classification and the input data is a set of images, then a convolutional neural network (CNN) might be the best choice. If the input data is a time series, then recurrent neural network (RNN), specifically LSTM, is a better choice. However, sometimes more than one network type can be used in a given problem. For instance, a multilayer perceptron (MLP) and CNN are also used for time series forecasting. Time series should be reshaped in input and output format to be used by MLP. For example, assume a time series X with length n and the following elements:
. In order to use MLP to forecast the next value, X needs to be reshapes as follows:| Input | Output |
|---|---|
| X(1) | X(2) |
| X(2) | X(3) |
| ⋮ | ⋮ |
| X(n-1) | X(n) |
RNN networks are mainly designed for sequence prediction problem. Unlike MLP, which considers input and output to be independent, RNN networks consist of memory cells that can remember the long term dependencies between elements of a sequence. In theory, RNN can remember arbitrary long time steps, but in practice, they suffer from vanishing gradient problem
[12, 23]. LSTM is designed to address the vanishing gradient problem. It consists of cell states and various gate elements that decide which data in a sequence is necessary to keep or throw away. By doing that, it can pass only relevant information down the long chain of sequences to make predictions.Ii-E Error Metrics
Forecasting error is divided into two categories of scale-dependent and scale-independent. Two well-known scale-dependent measures are mean absolute error (MAE), and root mean square error (RMSE). Minimizing the MAE will lead to forecasts of the median while minimizing the RMSE will lead to forecasts of the mean. Although both MAE and RMSE are widely used error measures, MAE has the advantage that it is easier to compute and understand [14]. MAE and RMSE are defined as follows:
| (1) | |||
| (2) | |||
| (3) |
Mean absolute percentage error (MAPE) is a widely used scale-independent error measure that is defined as follows:
| (4) | |||
| (5) |
Despite being commonly used in practice, it has the following disadvantages: 1- It is undefined when the true value is zero, and it will be a very large value when is close to zero. 2- It puts a heavier weight on the negative errors than the positive ones. 3- It does not make sense when the measurement has an arbitrary zero point [14, 15].
Iii Dataset
In this paper, We are using the synthetic data created as follows: First, we employ ARIMA models with random AR and MA coefficients to generate multivariate time series with length L and a different range of values for different columns. Second, we find the absolute value of each column of the time series to make sure that each column consists of only positive values. Third, we add arbitrary number of columns to the previously generated columns in the last two steps by simulating the piecewise continuous function that takes constant value from 0 to and from to L. Fourth, replace elements of time series by big values larger than the maximum value of each column (measurement) and elements by zeros to simulate the effect of outliers. Finally, additive white Gaussian noise (AWGN) with a specific variance depending on the range of the data is added to each column.
Above method has the following advantages: First, parameters of data such as the dimension of time series (number of measurements in multivariate time series), the variance of the noise, the minimum and the maximum value of time series corresponding to each measurement, and number of outliers can easily be defined and modified in simulation. Second, an arbitrary number of multivariate time series data can easily be created inside a loop.
Generating static data is more straightforward and is done as follows: First, identify the number of features as K. Second, determine the maximum and the minimum value and the type of each feature. Third, generate or sample data for each feature given the range of the values and the type of each feature.
For this paper, we are generating 400 multivariate time series data with three measurement columns. The total number of outliers per each measurement is 10. The length of each measurement is 400 as well. We generate 400 static data with five continuous features. Therefore the static data is a matrix of 400 by 5.
For the multivariate time series, we will be using names corresponding to physical phenomena. For example, the first measurement is oil, then is water, and the third column is gas. For static features, we will call them feature 1, feature 2, … , feature 5.
All the simulations run on Mac pro laptop with 16 GB memory and 2.80 GHz CPU.
Iv Problem Formulation
In this paper, The objective is to predict the cumulative gas value (Third column of time series) given all other data and measurements. We use the first N elements of time series and the static data for training and try to predict the cumulative gas value where . More specifically, we use and K=150, 200, 300, 400.
Neural networks and, more specifically, the LSTM are used for the prediction. We developed several architectures when time series data is only used for training and several other architectures when the combination of time series and static data are used for the training. The followings are the architectures used for training neural networks using the time series measurements.
|
|
|
To combine static and time series data architectures in Fig. 2 are proposed:
|
|
|
|
Architectures in Fig. 1 and 2 are tested against clustered and unclustered data under the following conditions:
-
Only gas measurement (third column of time series data) and the static data are used to predict the future values of gas.
-
All time series measurements or a subset of time series measurements, in addition to static data, are used to predict the future value gas.
V Results and Discussion
In this section, we describe the results for anomaly detection as well as forecasting with and without clustering. The plots in Fig. 3 present some examples of anomaly detection algorithm presented in section II.
|
|
|
In Fig. 3 the blue curve is the original curve, and the red is the curve when outliers are removed. As it is shown, the anomaly detection algorithm described in section II can accurately identify the outliers. This is a crucial preprocessing step to make sure the data used for prediction is reliable.
Table III presents the MAPE, MAE, and RMSE error measures using 5-fold cross-validations for different neural network architectures introduced in the last section when only gas measurement (third column of time series) is used to predict the future values of gas.
| Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|
| Model 1 | RMSE | 9684.88 | 27005.89 | 57649.70 | 71668.95 |
| MAPE | 33.94 | 42.41 | 44.54 | 20.51 | |
| MAE | 6827.62 | 19259.43 | 44179.18 | 53633.80 | |
| Time(s) | 1754 | 1613 | 1420 | 1250 | |
| Model 2 | RMSE | 9736.41 | 25782.56 | 53861.46 | 69246.07 |
| MAPE | 35.53 | 40.50 | 38.69 | 18.88 | |
| MAE | 6716.27 | 17245.11 | 43024.10 | 54552.27 | |
| Time(s) | 3003 | 2521 | 2067 | 2125 | |
| Model 3 | RMSE | 9216.22 | 26161.17 | 50678.83 | 65593.37 |
| MAPE | 35.42 | 40.37 | 38.52 | 18.18 | |
| MAE | 6462.38 | 19731.54 | 45336.42 | 51144.22 | |
| Time(s) | 1900 | 1861 | 1600 | 1550 | |
| Model 4 | RMSE | 10866.09 | 24287.97 | 56205.88 | 73167.68 |
| MAPE | 44.19 | 44.12 | 38.73 | 20.98 | |
| MAE | 8065.32 | 17892.22 | 43378.11 | 57337.14 | |
| Time(s) | 2900 | 2851 | 3421 | 2550 | |
| Model 5 | RMSE | 10313.77 | 26131.29 | 53851.36 | 72557.48 |
| MAPE | 40.17 | 46.08 | 37.63 | 21.07 | |
| MAE | 7533.38 | 19163.40 | 42630.06 | 55765.28 | |
| Time(s) | 1228 | 1888 | 1536 | 1409 | |
| Model 6 | RMSE | 11473.68 | 26938.94 | 54924.93 | 74088.37 |
| MAPE | 45.24 | 56.77 | 38.54 | 22.62 | |
| MAE | 8042.11 | 20872.36 | 44216.48 | 59516.29 | |
| Time(s) | 1800 | 1600 | 1770 | 1662 | |
| Model 7 | RMSE | 16198.66 | 32106.55 | 66425.61 | 76797.46 |
| MAPE | 71.09 | 71.10 | 41.13 | 22.56 | |
| MAE | 12662.59 | 25438.29 | 52732.66 | 61946.76 | |
| Time(s) | 1600 | 1523 | 1327 | 1710 |
In Table III, red color identifies the best RMSE; blue identifies best MAPE, and cyan identifies best MAE values among all architectures. According to Table III, Model 3 (Bidirectional LSTM) overall has the best performance since it gives the best RMSE values for K=150, 300, 400, and best MAPE values for K=200, 300, 400. In terms of time, Model 4 has the highest average time complexity with 2938 (s). Prediction without clustering can have the following problems:
-
Long forecasting time. If the process is interrupted due to insufficient memory or CPU, then training and forecasting should be repeated for all the data again.
-
Since data exhibit various patterns, shapes, and characteristics, training all of them together might lower the prediction performance.
Table IV summarizes the clustering characteristics for various clustering algorithms described in section II part C.
| Number of Measurements | Time(s) | Number of cluster | ||
|---|---|---|---|---|
| Distance-Based | DTW | 1 | 1352.40 | 3 |
| 3 | 1486.57 | 3 | ||
| Feature-Based | Method A | 1 | 12.61 | 2 |
| 3 | 24.48 | 2 | ||
| Method B | 1 | 275.71 | 2 | |
| 3 | 406.52 | 2 |
According to Table IV, feature-based methods outperform the distance-based method in terms of speed; moreover, Method A is faster of the two feature-based methods. The optimal number of clusters in Table IV is determined using several cluster validity indexes (CVI) such as Silhouette, Dunn, and Gamma. Next, we will present the prediction results with DTW clustering when only the gas measurement is used to predict the future value of the gas.
| Cluster 1 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|---|
| Model 1 | RMSE | 13492.88 | 27227.05 | 49960.58 | 71390.70 | |
| MAPE | 14.18 | 17.73 | 15.95 | 15.44 | ||
| MAE | 10493.94 | 19177.90 | 39821.71 | 57437.43 | ||
| Time(s) | 378 | 340 | 400 | 391 | ||
| Model 2 | RMSE | 15664.32 | 29477.30 | 50128.86 | 72614.45 | |
| MAPE | 16.13 | 18.19 | 15.80 | 15.47 | ||
| MAE | 13932.04 | 23122.89 | 38783.94 | 60840.61 | ||
| Time(s) | 687 | 1044 | 971 | 1200 | ||
| Model 3 | RMSE | 14319.32 | 29365.25 | 50414.84 | 72061.09 | |
| MAPE | 14.91 | 17.05 | 16.56 | 16.16 | ||
| MAE | 10814.49 | 24784.66 | 42159.97 | 60357.10 | ||
| Time(s) | 503 | 600 | 726 | 800 | ||
| Model 4 | RMSE | 17292.00 | 32476.94 | 44582.48 | 72725.99 | |
| MAPE | 19.78 | 18.94 | 12.88 | 16.39 | ||
| MAE | 14332.13 | 25061.84 | 38270.20 | 61102.26 | ||
| Time(s) | 1000 | 1080 | 1087 | 1200 | ||
| Model 5 | RMSE | 15303.55 | 31857.90 | 61536.90 | 79902.93 | |
| MAPE | 16.95 | 18.27 | 21.28 | 18.20 | ||
| MAE | 12397.09 | 25548.43 | 49572.39 | 66493.49 | ||
| Time(s) | 821 | 546 | 612 | 782 | ||
| Model 6 | RMSE | 14399.24 | 32669.23 | 54847.72 | 82953.44 | |
| MAPE | 14.89 | 17.93 | 17.62 | 19.19 | ||
| MAE | 11715.45 | 19259.43 | 44179.18 | 53633.80 | ||
| Time(s) | 453 | 460 | 433 | 421 | ||
| Model 7 | RMSE | 19112.97 | 32594.89 | 51854.97 | 75612.36 | |
| MAPE | 19.12 | 16.24 | 15.86 | 16.76 | ||
| MAE | 15384.47 | 25164.28 | 40750.27 | 65085.91 | ||
| Time(s) | 430 | 491 | 455 | 435 | ||
| Cluster 2 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
| Model 1 | RMSE | 4414.00 | 10923.24 | 31035.56 | 52708.65 | |
| MAPE | 33.37 | 43.41 | 40.05 | 20.52 | ||
| MAE | 3640.56 | 9157.58 | 25500.18 | 41605.72 | ||
| Time(s) | 800 | 557 | 800 | 871 | ||
| Model 2 | RMSE | 4419.03 | 11557.68 | 31079.12 | 49957.10 | |
| MAPE | 29.67 | 46.51 | 38.68 | 19.26 | ||
| MAE | 3452.11 | 9547.25 | 25161.26 | 39200.76 | ||
| Time(s) | 1450 | 1200 | 1745 | 1800 | ||
| Model 3 | RMSE | 4375.45 | 10533.33 | 30225.99 | 47035.53 | |
| MAPE | 32.53 | 43.11 | 38.26 | 16.62 | ||
| MAE | 3827.21 | 9393.74 | 24633.34 | 37753.89 | ||
| Time(s) | 952 | 942 | 923 | 968 | ||
| Model 4 | RMSE | 4028.32 | 10457.87 | 34559.18 | 57166.64 | |
| MAPE | 27.26 | 45.30 | 47.08 | 22.70 | ||
| MAE | 3297.35 | 8413.90 | 29013.81 | 44908.45 | ||
| Time(s) | 1379 | 1800 | 1826 | 1913 | ||
| Model 5 | RMSE | 4604.66 | 12290.53 | 33935.11 | 53214.73 | |
| MAPE | 31.24 | 52.95 | 41.64 | 20.69 | ||
| MAE | 3623.62 | 10039.21 | 27668.60 | 42097.45 | ||
| Time(s) | 584 | 713 | 920 | 678 | ||
| Model 6 | RMSE | 4413.09 | 10595.32 | 33016.45 | 54944.10 | |
| MAPE | 33.25 | 44.65 | 39.54 | 19.40 | ||
| MAE | 3562.62 | 8526.28 | 27270.80 | 42646.56 | ||
| Time(s) | 715 | 880 | 979 | 917 | ||
| Model 7 | RMSE | 5012.49 | 12623.80 | 33588.65 | 51467.56 | |
| MAPE | 41.53 | 53.82 | 42.48 | 19.79 | ||
| MAE | 4225.53 | 10208.43 | 27638.36 | 41616.74 | ||
| Time(s) | 712 | 725 | 781 | 796 | ||
| Cluster 3 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
| Model 1 | RMSE | 8035.00 | 20863.14 | 41267.41 | 50736.26 | |
| MAPE | 14.91 | 19.95 | 20.11 | 12.27 | ||
| MAE | 5999.42 | 16170.73 | 34492.70 | 41323.80 | ||
| Time(s) | 542 | 646 | 632 | 753 | ||
| Model 2 | RMSE | 8138.11 | 18217.21 | 45578.03 | 55843.99 | |
| MAPE | 14.35 | 17.24 | 19.37 | 12.15 | ||
| MAE | 5702.11 | 12931.99 | 34985.26 | 44785.75 | ||
| Time(s) | 1020 | 879 | 1225 | 1410 | ||
| Model 3 | RMSE | 9606.88 | 19171.64 | 50842.20 | 65310.41 | |
| MAPE | 17.07 | 20.13 | 19.33 | 17.51 | ||
| MAE | 6507.99 | 14906.23 | 36474.90 | 51834.14 | ||
| Time(s) | 802 | 1007 | 921 | 879 | ||
| Model 4 | RMSE | 9513.50 | 24396.60 | 36394.98 | 62606.17 | |
| MAPE | 19.66 | 19.87 | 17.92 | 15.15 | ||
| MAE | 7497.21 | 16790.39 | 29828.65 | 48750.67 | ||
| Time(s) | 1414 | 1834 | 1821 | 1690 | ||
| Model 5 | RMSE | 8857.02 | 22665.07 | 35642.91 | 59074.92 | |
| MAPE | 17.31 | 18.82 | 16.41 | 14.48 | ||
| MAE | 6841.79 | 16854.21 | 28238.08 | 47178.45 | ||
| Time(s) | 973 | 767 | 718 | 723 | ||
| Model 6 | RMSE | 7891.00 | 16401.70 | 38307.10 | 60533.33 | |
| MAPE | 15.09 | 17.90 | 18.70 | 15.72 | ||
| MAE | 6014.56 | 12895.91 | 31065.94 | 45152.46 | ||
| Time(s) | 800 | 617 | 648 | 712 | ||
| Model 7 | RMSE | 9100.24 | 17191.88 | 39321.63 | 58002.70 | |
| MAPE | 19.70 | 19.57 | 19.51 | 13.52 | ||
| MAE | 7582.63 | 13583.04 | 32742.17 | 44045.32 | ||
| Time(s) | 723 | 711 | 689 | 656 |
According to Table V clustering has the following advantages:
-
Prediction for different clusters can be made at different times, or it can be done parallel, which significantly reduces the forecasting time.
-
For each K, the best architecture for different clusters can be chosen to improve the overall performance. For example, consider K=150 when there is no clustering; the error metrics are as follows: RMSE =9. 216, MAPE = 33.94 and MAE = 6462.38. Now when DTW clustering is used for prediction given that 94 time series are in cluster 1, 188 time series are in cluster 2 and 118 time series are in cluster 3 the error metrics are calculated as follows:
(6) (7) (8) (9) In (6) is the total error, is the error corresponding to each cluster and
is the probability of choosing a cluster. Equation (6) follows the law of total probability.
Comparing the results in (7), (8) and (9) with RMSE, MAPE, and MAE in Table III reveal that clustering can significantly improve the prediction performance.
Tables VI and VII present the forecasting results with feature-based clustering using method A and method B, respectively.
| Cluster 1 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|---|
| Model 1 | RMSE | 4038.76 | 8832.14 | 27142.08 | 45872.40 | |
| MAPE | 34.56 | 38.37 | 38.89 | 18.52 | ||
| MAE | 3204 | 7341.46 | 23202.30 | 37073.63 | ||
| Time(s) | 621 | 691 | 630 | 681 | ||
| Model 2 | RMSE | 4545.92 | 9660.00 | 27239.60 | 44317.91 | |
| MAPE | 32.44 | 41.28 | 36.48 | 17.11 | ||
| MAE | 3400.99 | 7842.32 | 23022.10 | 35806.13 | ||
| Time(s) | 1249 | 1398 | 1100 | 1080 | ||
| Model 3 | RMSE | 3975.74 | 9906.88 | 27657.59 | 42505.00 | |
| MAPE | 33.62 | 46.85 | 40.81 | 17.44 | ||
| MAE | 3306.95 | 8172.18 | 23897.82 | 35207.36 | ||
| Time(s) | 900 | 861 | 812 | 871 | ||
| Model 4 | RMSE | 4328.14 | 10353.43 | 28500.70 | 42600.88 | |
| MAPE | 36.20 | 49.93 | 37.78 | 15.72 | ||
| MAE | 3472.24 | 8346.69 | 24439.28 | 34096.96 | ||
| Time(s) | 1313 | 1814 | 1600 | 1324 | ||
| Model 5 | RMSE | 4949.87 | 9389.69 | 25986.47 | 45082.67 | |
| MAPE | 42.09 | 38.83 | 33.53 | 18.50 | ||
| MAE | 3951.77 | 7696.16 | 21675.31 | 36561.82 | ||
| Time(s) | 671 | 671 | 712 | 650 | ||
| Model 6 | RMSE | 4279.94 | 9743.54 | 27923.25 | 45612.06 | |
| MAPE | 32.34 | 54.16 | 40.99 | 17.29 | ||
| MAE | 3413.10 | 8544.21 | 23223.17 | 36221.62 | ||
| Time(s) | 591 | 660 | 687 | 612 | ||
| Model 7 | RMSE | 4972.13 | 10147.80 | 27145.73 | 46906.73 | |
| MAPE | 38.52 | 45.45 | 38.78 | 18.25 | ||
| MAE | 4040.88 | 8444.79 | 22175.59 | 38482.42 | ||
| Time(s) | 623 | 711 | 601 | 589 | ||
| Cluster 2 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
| Model 1 | RMSE | 11872.12 | 29413.02 | 56085.34 | 65408.21 | |
| MAPE | 28.97 | 29.99 | 25.27 | 15.75 | ||
| MAE | 8838.86 | 22715.02 | 46822.08 | 51364.60 | ||
| Time(s) | 800 | 882 | 1262 | 922 | ||
| Model 2 | RMSE | 14263.18 | 29695.96 | 53459.04 | 60645.00 | |
| MAPE | 33.67 | 34.36 | 23.32 | 13.93 | ||
| MAE | 11584.29 | 24174.64 | 43544.11 | 47841.09 | ||
| Time(s) | 2137 | 1945 | 1462 | 1541 | ||
| Model 3 | RMSE | 14049.78 | 29075.38 | 56893.44 | 58575.78 | |
| MAPE | 35.24 | 29.36 | 25.76 | 13.61 | ||
| MAE | 11626.64 | 23097.41 | 47130.03 | 47059.17 | ||
| Time(s) | 1162 | 1101 | 1053 | 996 | ||
| Model 4 | RMSE | 15978.33 | 32114.92 | 58155.73 | 63148.46 | |
| MAPE | 41.68 | 36.38 | 25.97 | 14.26 | ||
| MAE | 13298.98 | 26018.09 | 47476.40 | 48745.07 | ||
| Time(s) | 1902 | 2084 | 1969 | 2010 | ||
| Model 5 | RMSE | 12947.65 | 29301.81 | 59024.33 | 65776.89 | |
| MAPE | 30.35 | 29.24 | 25.84 | 16.16 | ||
| MAE | 10555.66 | 23042.84 | 46318.66 | 53606.05 | ||
| Time(s) | 1239 | 1300 | 1312 | 1080 | ||
| Model 6 | RMSE | 11963.81 | 28878.83 | 53073.28 | 61825.07 | |
| MAPE | 26.55 | 33.24 | 23.08 | 14.99 | ||
| MAE | 9487.32 | 23477.52 | 42112.35 | 48812.02 | ||
| Time(s) | 1001 | 1211 | 1194 | 951 | ||
| Model 7 | RMSE | 11988.71 | 37235.03 | 61843.47 | 64530.66 | |
| MAPE | 50.88 | 50.18 | 32.29 | 16.38 | ||
| MAE | 9263.22 | 32333.89 | 57271.93 | 53606.42 | ||
| Time(s) | 923 | 1187 | 1015 | 986 |
| Cluster 1 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|---|
| Model 1 | RMSE | 6262.29 | 14602.69 | 40447.69 | 55142.00 | |
| MAPE | 34.51 | 43.28 | 48.18 | 19.65 | ||
| MAE | 4705.40 | 11756.55 | 32875.25 | 43141.80 | ||
| Time(s) | 1211 | 1031 | 876 | 1105 | ||
| Model 2 | RMSE | 5790.1 | 12494.61 | 33793.05 | 52679.51 | |
| MAPE | 32.10 | 43.72 | 42.01 | 19.88 | ||
| MAE | 4670.78 | 10739.64 | 27789.09 | 42660.21 | ||
| Time(s) | 2086 | 2055 | 2010 | 1865 | ||
| Model 3 | RMSE | 4906.85 | 12144.00 | 32997.05 | 49625.67 | |
| MAPE | 31.89 | 42.44 | 37.96 | 17.46 | ||
| MAE | 3591.33 | 10180.60 | 27072.80 | 40025.59 | ||
| Time(s) | 964 | 1353 | 1469 | 1100 | ||
| Model 4 | RMSE | 6212.31 | 12706.26 | 32646.63 | 52751.72 | |
| MAPE | 41.08 | 50.42 | 37.80 | 19.56 | ||
| MAE | 4775.04 | 10641.48 | 26849.83 | 42106.65 | ||
| Time(s) | 2154 | 1772 | 2100 | 2254 | ||
| Model 5 | RMSE | 5403.45 | 14679.94 | 36773.00 | 52649.04 | |
| MAPE | 35.38 | 55.43 | 45.63 | 19.76 | ||
| MAE | 4361.95 | 12294.69 | 30163.46 | 42906.34 | ||
| Time(s) | 1129 | 945 | 1001 | 1122 | ||
| Model 6 | RMSE | 5621.37 | 13893.23 | 32005.10 | 54944.21 | |
| MAPE | 35.07 | 48.77 | 34.76 | 20.83 | ||
| MAE | 4480.75 | 11734.42 | 26004.11 | 43438.74 | ||
| Time(s) | 1122 | 1029 | 983 | 912 | ||
| Model 7 | RMSE | 6515.12 | 15772.53 | 34345.54 | 53317.02 | |
| MAPE | 43.30 | 65.44 | 45.34 | 20.82 | ||
| MAE | 5208.36 | 13215.27 | 27508.65 | 43108.25 | ||
| Time(s) | 987 | 998 | 1112 | 1010 | ||
| Cluster 2 | Architecture | Performance Measure | 150 | 200 | 300 | 400 |
| Model 1 | RMSE | 12059.14 | 29017.86 | 47730.52 | 62920.93 | |
| MAPE | 12.81 | 22.03 | 27.12 | 13.93 | ||
| MAE | 8583.00 | 22420.10 | 39235.51 | 51245.27 | ||
| Time(s) | 730 | 529 | 625 | 565 | ||
| Model 2 | RMSE | 12281.21 | 28399.50 | 52188.18 | 57151.25 | |
| MAPE | 15.99 | 22.09 | 19.81 | 11.44 | ||
| MAE | 8950.43 | 22083.35 | 42345.36 | 45136.57 | ||
| Time(s) | 1048 | 1137 | 1300 | 1323 | ||
| Model 3 | RMSE | 11296.55 | 26270.88 | 48964.51 | 56242.17 | |
| MAPE | 13.36 | 26.19 | 16.36 | 11.75 | ||
| MAE | 7776.63 | 19426.56 | 39846.93 | 45253.69 | ||
| Time(s) | 1100 | 913 | 848 | 874 | ||
| Model 4 | RMSE | 14693.48 | 31948.36 | 46154.93 | 57638.41 | |
| MAPE | 20.96 | 23.20 | 16.05 | 11.26 | ||
| MAE | 11306.93 | 24519.02 | 37135.12 | 44802.18 | ||
| Time(s) | 1283 | 1525 | 1469 | 1198 | ||
| Model 5 | RMSE | 12000.75 | 33271.13 | 52588.69 | 56051.11 | |
| MAPE | 14.56 | 25.98 | 18.75 | 11.49 | ||
| MAE | 8835.99 | 26458.75 | 42617.07 | 43542.71 | ||
| Time(s) | 664 | 661 | 869 | 867 | ||
| Model 6 | RMSE | 13363.42 | 31299.26 | 47829.81 | 57017.62 | |
| MAPE | 19.77 | 22.93 | 14.89 | 12.19 | ||
| MAE | 10311.96 | 23920.97 | 38111.54 | 45107.26 | ||
| Time(s) | 701 | 779 | 923 | 951 | ||
| Model 7 | RMSE | 14424.54 | 30437.25 | 52917.75 | 59926.56 | |
| MAPE | 17.57 | 22.48 | 17.76 | 11.96 | ||
| MAE | 10690.42 | 23874.18 | 43415.56 | 47402.78 | ||
| Time(s) | 923 | 1187 | 1015 | 986 |
According to the Table VI, the total error measures for K=150 for method A is as follows:
| (10) |
| (11) |
| (12) |
The same calculation can be done according to the Table VII for method B and K=150:
| (13) |
| (14) |
| (15) |
Comparing (7) up to (15) reveal that DTW and method B clustering methods provide the best improvement to prediction Errors. Furthermore, according to Table IV, feature-based clustering methods (method A, method B) significantly outperform the DTW method in terms of speed. Therefore if speed and efficiency are the primary concern, method A and method B can be combined with the neural networks to improve efficiency and error. If improving error is the main objective, then method B and DTW are a clear choice.
Now consider the goal is to compare different clustering algorithms based on the improvement they provide to forecasting errors for all the architectures. This means we are not choosing the best model like before, but we consider all the models for each clustering method. For example, to find the RMSE error for model 1 for DTW clustering, we find the total RMSE when model 1 is used for all the clusters. The results of this analysis for DTW, method A, and method B are summarized in Tables VIII, IX, and X, respectively.
| Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|
| Model 1 | RMSE | 7615.73 | 17686.91 | 38501.34 | 56517.08 |
| MAPE | 23.41 | 30.45 | 28.50 | 16.89 | |
| MAE | 5946.97 | 13581.36 | 31518.53 | 45243.00 | |
| Model 2 | RMSE | 8158.80 | 17733.35 | 39832.98 | 57018.20 |
| MAPE | 21.96 | 31.22 | 27.60 | 16.27 | |
| MAE | 6578.64 | 13736.02 | 31260.66 | 45133.94 | |
| Model 3 | RMSE | 8255.53 | 17507.13 | 41052.15 | 58307.62 |
| MAPE | 23.82 | 30.20 | 27.57 | 16.77 | |
| MAE | 6260.05 | 14636.79 | 32245.35 | 47219.31 | |
| Model 4 | RMSE | 8763.41 | 19744.27 | 37456.21 | 62427.74 |
| MAPE | 23.26 | 31.60 | 30.44 | 18.98 | |
| MAE | 7129.48 | 14797.23 | 31429.43 | 49847.45 | |
| Model 5 | RMSE | 8373.34 | 19949.35 | 40925.33 | 61215.21 |
| MAPE | 23.77 | 34.73 | 29.41 | 18.27 | |
| MAE | 6634.74 | 15694.30 | 32983.98 | 49329.41 | |
| Model 6 | RMSE | 7785.81 | 17495.57 | 39707.54 | 63175.11 |
| MAPE | 23.57 | 30.47 | 28.24 | 18.26 | |
| MAE | 6201.85 | 12337.61 | 32363.83 | 45967.80 | |
| Model 7 | RMSE | 11808.89 | 18664.58 | 39572.46 | 59069.45 |
| MAPE | 30.02 | 34.88 | 29.44 | 17.22 | |
| MAE | 9216.95 | 14718.56 | 32225.28 | 47848.42 |
| Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|
| Model 1 | RMSE | 9091.27 | 22106.80 | 45810.48 | 58472.99 |
| MAPE | 30.95 | 32.96 | 30.10 | 16.73 | |
| MAE | 6838.48 | 17257.40 | 38437.05 | 41366.98 | |
| Model 2 | RMSE | 10813.55 | 22583.19 | 44151.13 | 55400.72 |
| MAPE | 33.23 | 36.81 | 27.99 | 15.55 | |
| MAE | 8679.21 | 18376.66 | 38882.59 | 44018.64 | |
| Model 3 | RMSE | 10473.49 | 22270.56 | 46514.71 | 52870.65 |
| MAPE | 34.66 | 35.56 | 31.10 | 14.96 | |
| MAE | 8673.15 | 17798.95 | 38882.59 | 42851.77 | |
| Model 4 | RMSE | 11842.51 | 24389.59 | 47628.19 | 55854.06 |
| MAPE | 39.73 | 41.19 | 30.16 | 14.77 | |
| MAE | 9810.48 | 19744.74 | 39298.22 | 43544.99 | |
| Model 5 | RMSE | 10108.43 | 22233.00 | 47295.88 | 58430.44 |
| MAPE | 34.51 | 32.64 | 28.56 | 16.99 | |
| MAE | 8211.27 | 17594.76 | 37570.27 | 47555.34 | |
| Model 6 | RMSE | 9236.03 | 22085.80 | 44145.01 | 56069.45 |
| MAPE | 28.60 | 40.66 | 29.43 | 15.80 | |
| MAE | 7330.97 | 18176.19 | 35406.69 | 44342.42 | |
| Model 7 | RMSE | 9497.82 | 27619.06 | 49525.77 | 58274.16 |
| MAPE | 46.49 | 48.50 | 34.59 | 17.04 | |
| MAE | 7409.28 | 23853.25 | 44812.72 | 48237.39 |
| Architecture | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|
| Model 1 | RMSE | 8725.95 | 20729.13 | 43542.89 | 58448.04 |
| MAPE | 25.28 | 34.24 | 39.22 | 17.21 | |
| MAE | 6489.63 | 16288.55 | 35578.36 | 46585.77 | |
| Model 2 | RMSE | 8548.82 | 19254.18 | 41610.98 | 54579.99 |
| MAPE | 25.25 | 34.52 | 32.57 | 16.29 | |
| MAE | 6489.63 | 15560.71 | 33975.50 | 43712.66 | |
| Model 3 | RMSE | 7622.47 | 18147.92 | 39783.22 | 52437.68 |
| MAPE | 24.01 | 35.53 | 28.77 | 15.03 | |
| MAE | 5370.08 | 14110.13 | 32501.80 | 42247.53 | |
| Model 4 | RMSE | 9816.80 | 20884.15 | 38387.65 | 54828.56 |
| MAPE | 32.52 | 38.85 | 28.55 | 16.03 | |
| MAE | 7551.09 | 16539.43 | 31221.07 | 43252.25 | |
| Model 5 | RMSE | 8207.30 | 22581.19 | 43494.66 | 54094.91 |
| MAPE | 26.53 | 42.91 | 34.20 | 16.24 | |
| MAE | 6263.41 | 18314.41 | 35456.24 | 43176.79 | |
| Model 6 | RMSE | 8911.74 | 21290.79 | 38730.60 | 55825.40 |
| MAPE | 28.56 | 37.78 | 26.31 | 17.15 | |
| MAE | 6959.01 | 16913.70 | 31149.76 | 44147.86 | |
| Model 7 | RMSE | 9876.62 | 22005.03 | 40076.35 | 56126.07 |
| MAPE | 32.36 | 47.18 | 32.39 | 17.05 | |
| MAE | 7538.23 | 17745.30 | 34269.08 | 44933.42 |
Analyzing Tables VIII, IX, and X reveal that DTW clustering provides better overall improvement to MAPE, RMSE and MAE scores when K=150,200, 300, while for K=400 both methods A and method B outperform DTW. However, comparing Tables III, VIII, IX, and X reveal that regardless of the clustering algorithms, forecasting accuracies can significantly be improved when data clustering is done. For example, consider K=300, with no clustering the best MAPE score is 38.52 for model 3; however, with DTW clustering model 3 achieves MAPE score of 27.57, methods A and B achieve MAPE scores of 31.10 and 28.88 respectively.
V-a Influence of adding more Time Series Measurements
Thus far, we have used the gas measurement along with static data to predict future gas values. However, an important question to ask is, can prediction errors be improved by including more time series measurement?
To answer this question, we will consider all possible combinations of multivariate time series column and compare their prediction errors to Table III. To show the concept and also save simulation time, the idea is tested only against model 1 and model 3.
| Model 1 | Measurements Column | Performance Measure | 150 | 200 | 300 | 400 |
|---|---|---|---|---|---|---|
| First and Second Columns | RMSE | 39450.01 | 56527.14 | 84660.19 | 86700.9 | |
| MAPE | 150.38 | 91.89 | 62.33 | 25.86 | ||
| MAE | 29850.68 | 44804.45 | 69698.23 | 70742.31 | ||
| First and Third Columns | RMSE | 10218.19 | 27168.96 | 58130.94 | 72662.22 | |
| MAPE | 34.36 | 47.14 | 45.89 | 20.66 | ||
| MAE | 7122.69 | 19889.56 | 45096.30 | 55307.80 | ||
| Second and Third Columns | RMSE | 10039.66 | 26831.03 | 60065.97 | 69787.29 | |
| MAPE | 34.53 | 43.16 | 41.94 | 21.14 | ||
| MAE | 7311.21 | 18740.35 | 46414.14 | 55175.58 | ||
| First, Second and Third | RMSE | 10319.54 | 26638.20 | 55663.48 | 72562.50 | |
| MAPE | 36.33 | 42.75 | 45.02 | 20.97 | ||
| MAE | 7257.43 | 19944.56 | 49724.80 | 54406.62 | ||
| Model 3 | Measurements Column | Performance Measure | 150 | 200 | 300 | 400 |
| First and Second Columns | RMSE | 42960.62 | 52977.89 | 94306.01 | 88665.62 | |
| MAPE | 98.84 | 106.53 | 80.84 | 25.47 | ||
| MAE | 30140.50 | 41278.36 | 78561.96 | 72679.35 | ||
| First and Third Columns | RMSE | 10973.27 | 28103.06 | 53204.26 | 71642.94 | |
| MAPE | 34.33 | 48.45 | 39.75 | 20.41 | ||
| MAE | 7886.56 | 20174.22 | 41335.21 | 61775.57 | ||
| Second and Third Columns | RMSE | 10417.47 | 26390.57 | 55478.94 | 71627.27 | |
| MAPE | 34.56 | 47.36 | 39.31 | 21.55 | ||
| MAE | 7036.22 | 20164.46 | 45503.44 | 55921.33 | ||
| First, Second and Third | RMSE | 10287.64 | 26195.90 | 63499.57 | 71540.70 | |
| MAPE | 36.33 | 45.75 | 38.02 | 18.97 | ||
| MAE | 7257.43 | 19944.56 | 49724.80 | 52406.62 |
Comparing Table XI to Table III reveals that adding more time series measurements is not, in general, improving the RMSE, MAPE, and MAE. Such high errors in Table XI corresponding to the first entries of model 1 and model 3 suggest that the first and second columns alone are not good predictors of the third column.
Vi Conclusion
In this paper, we provided a comprehensive analysis of time series forecasting using neural networks. We introduced several architectures to combine static and dynamic measurements to forecast time series using neural networks. Our results reveal that: First, not one architecture is right for predicting all future values of time series, but multiple architectures should be tested, and the best architecture should be determined based on model selection criterion such as cross-validation. Second, adding static data to forecast does not necessarily improve the forecasting errors.
We discussed two primary clustering methods, namely, distance-based and feature-based clustering. We showed that DTW outperforms the feature-based clustering in most architectures; however, feature-based clustering methods outperform the DTW in terms of speed and time complexity and still improve the forecasting accuracies significantly when they are compared to no clustering case.
Finally, we presented several methods, such as anomaly detection and data imputation, to prepare time series data for forecasting and the clustering.
References
- [1] (2015) Time-series clustering–a decade review. Information Systems 53, pp. 16–38. Cited by: §I.
- [2] (2009) Forecasting nonlinear time series with a hybrid methodology. Applied Mathematics Letters 22 (9), pp. 1467–1470. Cited by: §I.
- [3] (2014) Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, pp. 106–112. Cited by: §I.
- [4] (2020) Forecasting across time series databases using recurrent neural networks on groups of similar series: a clustering approach. Expert Systems with Applications 140, pp. 112896. Cited by: §I.
- [5] (2002) A study of k-nearest neighbour as an imputation method.. HIS 87 (251-260), pp. 48. Cited by: §II-B.
- [6] (1992) Forecasting the behavior of multivariate time series using neural networks. Neural networks 5 (6), pp. 961–970. Cited by: §I, §I.
- [7] (2018) Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package). Neurocomputing 307, pp. 72–77. Cited by: §II-C.
- [8] (2003) ARIMA models to predict next-day electricity prices. IEEE transactions on power systems 18 (3), pp. 1014–1020. Cited by: §I.
- [9] (2007) ARIMA forecasting of primary energy demand by fuel in turkey. Energy policy 35 (3), pp. 1701–1708. Cited by: §I.
- [10] (2005) Stock market forecasting using hidden markov model: a new approach. In 5th International Conference on Intelligent Systems Design and Applications (ISDA’05), pp. 192–196. Cited by: §I.
- [11] (2015) Matrix completion and low-rank svd via fast alternating least squares. The Journal of Machine Learning Research 16 (1), pp. 3367–3402. Cited by: §II-B.
- [12] (1998) The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6 (02), pp. 107–116. Cited by: §II-D.
- [13] (2019) Enhanced recurrent neural network for combining static and dynamic features for credit card default prediction. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1572–1576. Cited by: §I.
- [14] (2018) Forecasting: principles and practice. OTexts. Cited by: §II-E.
- [15] (2006) Another look at measures of forecast accuracy. International journal of forecasting 22 (4), pp. 679–688. Cited by: §II-E.
- [16] (2015) Large-scale unusual time series detection. In 2015 IEEE international conference on data mining workshop (ICDMW), pp. 1616–1619. Cited by: §II-C.
-
[17]
(2016)
Combining static and dynamic features for multivariate sequence classification.
In
2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA)
, pp. 21–30. Cited by: §I. - [18] (2005) Clustering of time series data—a survey. Pattern recognition 38 (11), pp. 1857–1874. Cited by: §I.
- [19] (2015) Long short term memory networks for anomaly detection in time series. In Proceedings, pp. 89. Cited by: §II-A.
- [20] (2010) Spectral regularization algorithms for learning large incomplete matrices. Journal of machine learning research 11 (Aug), pp. 2287–2322. Cited by: §II-B.
- [21] (2007) Dynamic time warping. Information retrieval for music and motion, pp. 69–84. Cited by: §II-C.
-
[22]
(1999)
Clustering time series with hidden markov models and dynamic time warping.
In
Proceedings of the IJCAI-99 workshop on neural, symbolic and reinforcement learning methods for sequence learning
, pp. 17–21. Cited by: §I. - [23] (2013) On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310–1318. Cited by: §II-D.
- [24] (2005) Anomaly detection in time series of graphs using arma processes. Asor Bulletin 24 (4), pp. 2. Cited by: §II-A.
- [25] (2007) Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis 11 (5), pp. 561–580. Cited by: §II-C.
- [26] (2018) A comparison of arima and lstm in forecasting time series. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 1394–1401. Cited by: §I.
- [27] (2016) Speeding up all-pairwise dynamic time warping matrix calculation. In Proceedings of the 2016 SIAM International Conference on Data Mining, pp. 837–845. Cited by: §II-C.
- [28] (2019) Predicting student performance in an educational game using a hidden markov model. arXiv preprint arXiv:1904.11857. Cited by: §I.
- [29] (2001) Missing value estimation methods for dna microarrays. Bioinformatics 17 (6), pp. 520–525. Cited by: §II-B.
- [30] (2006) Characteristic-based clustering for time series data. Data mining and knowledge Discovery 13 (3), pp. 335–364. Cited by: §II-C.
- [31] (2019) Integration of static and dynamic analysis for malware family classification with composite neural network. arXiv preprint arXiv:1912.11249. Cited by: §I.
- [32] (2003) Time series forecasting using a hybrid arima and neural network model. Neurocomputing 50, pp. 159–175. Cited by: §I, §I.