One area that strongly requires a technique for multivariate time series analysis is cyber-security for industrial processes (Stu, 2014). Conventional cyber-security tools are used to detect malicious activity at the communication level and the binary execution level. Meanwhile, Industry 4.0 and the IoT era means cyber and physical parts are connected in a single Cyber-Physical System (CPS). To protect a CPS one has to use not only conventional means for cyber-security but also perform communication protocol deep package inspection (DPI). A DPI tool needs to monitor and detect faults inside technological processes by analyzing historical and real-time streaming of industrial data.
Numerous approaches to fault detection (FD) in industrial and other types of multivariate time series have been proposed: classic methods like PCA, DPCA, FDA, DFDA, CVA, PLS (Chiang et al., 2001), SVM and segmentation (Lin et al., 2007; MartГ et al., 2015), change point detection (Matteson and James, 2013), LSTM (Malhotra et al., 2015, 2016; Yadav et al., 2016) .
In this paper for the purpose of monitoring and detecting faults inside a multivariate industrial time series that contains both sensors and controls signals, we evolve LSTM-based approach (Malhotra et al., 2015, 2016).
To validate our approach we needed real object data sets for normal as well as anomalous behavior. Experiments on data sets from several real industrial objects are usually faced with the same problem - the absence of anomalous behavior, or very few examples. To provide realistic data with anomalies we created a mathematical model of part of a real gasoil plant. Having a model, we were able to modify some of the process logic and generate faults. With a self-consistent mathematical model and knowing the causality relations of model variables, we trained and tested an LSTM neural network and deeply investigated the obtained results and adopted LSTM architecture parameters.
2 Data Set Description
We created a Modelica model for a gasoil plant heating loop.
The gasoil heating loop (GHL) model comprises three reservoirs: receiving tank (RT), heating tank (HT) and collector tank (CT). The technological task it to heat gasoil in RT up to 60 degrees Celsius, thus reaching a gasoil viscosity that is enough to transfer it to CT. Heating in the model is performed in portions. A portion of gasoil is heated up to 60 in HT and then pumped back into RT and ralaxing there for some time. This process is repeated till reaching 60 in RT. RT is then emptied into CT. After that, RT is refilled from some inexhaustible source.
For simplicity, we used water as the fluid instead of gasoil. We used Dymola to simulate the model.
Using the GHL model we generated a multivariate time series with 270 variables. In this paper we present the results for a multivariate time series with only 19 variables (GHL, 2016). For the complete 270-variables time series we also applied the same technique of fault detection and obtained the same results except that the time for fitting the model was 30% longer. We selected 19 variables knowing the semantic of data; however, with real object this is not always possible. The most interesting variables of normal behavior are represented in the Figure 2. The first three variables are the sensors of RT level , RT temperature and HT temperature. The last two variables correspond to gasoil source on/off and heater on/off control signals.
In the GHL model we introduced four types of cyber attack to the normal process logic:
unauthorized change of max RT level,
unauthorized change of max HT temperature,
unauthorized change of pump frequency,
unauthorized change of system relaxing time value.
In the current paper we only present the results for fitting and testing the LSTM for anomalies generated by the first type of attack to the max-RT-level set point. By changing the time of attack and the value of the hacked max-RT-level, we generated many anomalous data sets used for fault detection. To train the LSTM we used only a data set with normal behavior.
The generated data has no outlier. When dealing with data from real objects, before learning normal behavior, we perform data preprocessing, thus eliminating outliers and data gaps.
When dealing with cyber attacks at the industrial data level, the main task is to detect anomalous process flows as earlier as possible.
In the generated data set we know the time when the attacker changed the control logic set-point, the start of the sub-process which is influenced by the attacker’s changes, the time when the sub-process crossed the normal behavior condition and the interval when the attack resulted in an incident. The data-driven model has to “see” all of these situations. In a real attack, even if the attacker was able to hide the control logic set-point change event, the data-driven model has to detect a fault at the time-point when the sub-process crosses the normal behavior condition.
Generated multivariate time series consist of high-dimensional complex nonlinear, non-stationary data with non-Gaussian pointwise distribution. Variables have partially probabilistic nature. Correlations of variables have event-based nature because of control data primacy. As will be shown in the next section accurate fitting of this data using parametric data-driven model requires thousands of model parameters, moreover, complete model learning requires data set containing about million of time points ( sec). Meanwhile temporal evolution of hacker-induced anomalies often are very fast (
sec) and rapidly grown into an equipment damage. Under these conditions, online process monitoring using traditional change point methods (operating with testing of several statistical hypothesis) are dramatically complicated by the requirement of fast online estimation of thousand model parameters to make decision about starting of anomalous process behavior. Thus without prior information about anomalies and their representation in process trajectories the most appropriate anomaly detection technique operates with fixed model pre-trained on data set under normal operating conditions. Such technique considers anomalies as a deviation of observed process trajectories from trajectories predicted by the model.
3 LSTM-based Fault Detection
Input data can be described as multivariate time series , where belongs to dimensional space , — number of time points. The proposed fault detection algorithm consists of two parts: forecasting and detection. At first we split the whole time series into equal-sized batches of length denoted as . Here is the batch number and is the number of first time point in the batch. In the forecasting part we predict values for the next batch using already observed measurements . The detection part is based on finding time points where the mean square error (MSE) between the measured and predicted values becomes higher then the precomputed threshold.
3.1 Data Preprocessing
All data points in the presented data set share the same time grid and have significantly varying absolute values. To reduce these variations and unify different dimensions we applied normalization transform on each dimension separately:
are the mean value and standard deviation for each dimension.
In the test set the additional variables labeled as ATTACK, DANGER and FAULT are introduced. They determine different parts of attack evolution. We will use the DANGER series to compare results with the fault-detection algorithm.
3.2 Neural Network Architecture
The choise of optimal network architecture is based on several observations. At first, the most industrial technological processes generate strongly correlated multivariate time series. Furthermore we frequently deal with multiscale processes (see Figure 2
) having fast (long-term) and slow (short-term) sub-processes. In these conditions conventional feed-forward neural networks usually demonstrate a poor results. An accurate data-driven predictive model can be developed using stateful LSTM neural network(Hochreiter and Schmidhuber, 1997; Malhotra et al., 2015; Nanduri et al., 2016). The proposed network architecture includes two stacked LSTM layers with linear output layer (Figure 4). In addition we use a sequence-to-sequence architecture of LSTM network for the forecasting model (Figure 4).
The dropout technique (Srivastava et al., 2014)
is used for regularization. The results for different dropout probability values are shown in Table12012) optimization algorithm is used for training. In Figure 5 an example of the forecasted values for one control variable is shown.
The detection part is based on the MSE between actual data and forecasted values.
To smooth high errors in single points we applied an exponential moving average of MSE where the “half-life” exponential parameter was chosen as doubled batch length (see Figure 7). To achieve a better results in MSE computational experiments we considered only a subset of the aforementioned 19 variables. These are RT_level, RT_temperature, HT_level, HT_temperature, inj_valve_act and heater_act - the most important variables partially represented in Figure 2.
According to discussion in section 2 we will determine process anomalies in terms of forecasting error. The horizontal line in the last subplot (Figure 7) represents a quantile of empirical error distribution. This level is used as a lower boundry for the threshold in the fault-detection algorithm. The decision rule is formulated as follows: if the forecast error is less or equal to the threshold level then the algorithm indicates normal behavior and if the forecast error is greater than the threshold level the algorithm predicts abnormal behavior (fault).
3.3 Quality Metrics
To compute the precision and recall scores for different thresholds we split each test series into equal-sized intervals and check whether MSE is greater than the threshold level. Such a situation is treated as a fault; otherwise, an interval is classified as normal behavior. Figure7 illustrates how the precision, recall and scores depend on threshold level. An interesting practical aspect of the results represented in Figure 7 is that the threshold level may be used as a tunable parameter that can be changed to achieve desired fault positive rate. This aspect can help us to handle the problem of lots of false positive alerts in a monitoring system. The operator of an industrial object can set this parameter to suitable level.
The best score results for different batch size () and dropout probability () are represented in Table 1
3.4 Comparison With Other Methods
The most known methods of industrial fault detection are given in (Chiang et al., 2001). Table 2 shows comparision results of conventional fault detection methods versus the proposed approach tested at 6 aforementioned variables.
As it follows from table 2
such methods as PCA, FDA and СVA show good results in precision but not in recall. The OneClassSVM with radial basis functions as the kernel achieves the best recall but poor precision. The PCA and LSTM show balanced results in both metrics. The LSTM dominates PCA and achieves best averaged () result for described dataset.
4 Conclusion and Future Work
The current paper presents a publicly available dataset for the problem of industrial fault detection. This dataset consists of a multivariate time-series training set and dozens of test sets with different types of faults. Like the Tennessee Eastman process (Ricker, ), the proposed dataset includes both sensor and control, continuous and discreets channels for analysis. The results obtained in section 3 show that the LSTM-based fault-detection approach has advantages over classic fault-detection methods (Chiang et al., 2001). The error threshold level was introduced as a tunable parameter that allows a user to achieve a satisfactory false positive and false negative detection rate.
The fault-detection approach described in section 3 restricts us to a binary decision: the system either operates in normal or abnormal mode. From a practical point of view, such a system has the following disadvantages: alerts cannot be prioritized and interpreted. A possible modification to the proposed approach is to add strict order. Some kind of abnormality measure may help to prioritize alerts triggered by a monitoring system. Such measure may also provide the possibility to use a more complex quality metric such as receiver operating characteristic (ROC). Another possible modification is to add methods for fault diagnosis (Chiang et al., 2001)
to provide not only the moment of time when a fault is detected but also to localize the subset of channels where it was detected. This problem is particularly important in the analysis of large dimension time series.
Another research direction we see in GHL-model improvement in order to reach more realistic data via including stochastic parameters, measurement noise and random outlayers. This will enrich process trajectories and allows us to test low-order statistical parametric model and change point techniques.
- Stu (2014) Stuxnet: Zero victims, 2014. URL https://securelist.com/analysis/publications/67483/stuxnet-zero-victims/.
- GHL (2016) Gasoil heating loop dataset, 2016. URL https://kas.pr/ics-research/dataset_ghl_1.
- Chiang et al. (2001) L H Chiang, E L Russell, and R D Braatz. Fault detection and diagnosis in industrial systems. Measurement Science and Technology, 12(10):1745, 2001. URL http://stacks.iop.org/0957-0233/12/i=10/a=706.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1922.214.171.1245. URL http://dx.doi.org/10.1162/neco.19126.96.36.1995.
- Lin et al. (2007) Jessica Lin, Eamonn Keogh, Li Wei, and Stefano Lonardi. Experiencing sax: a novel symbolic representation of time series. Data Mining and Knowledge Discovery, 15(2):107–144, 2007. ISSN 1573-756X. doi: 10.1007/s10618-007-0064-z. URL http://dx.doi.org/10.1007/s10618-007-0064-z.
Malhotra et al. (2015)
Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, and Puneet Agarwal.
Long short term memory networks for anomaly detection in time series.
23rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning., April 2015.
- Malhotra et al. (2016) Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. CoRR, abs/1607.00148, 2016. URL http://arxiv.org/abs/1607.00148.
- Yadav et al. (2016) Mohit Yadav, Pankaj Malhotra, Lovekesh Vig, K. Sriram, and Gautam Shroff. ODE - augmented training improves anomaly detection in sensor data from machines. CoRR, abs/1605.01534, 2016. URL http://arxiv.org/abs/1605.01534.
- MartГ et al. (2015) Luis MartГ, Nayat Sanchez-Pi, JosГ© Manuel Molina, and Ana Cristina Bicharra Garcia. Anomaly detection based on sensor data in petroleum industry applications. Sensors, 15(2):2774, 2015. ISSN 1424-8220. doi: 10.3390/s150202774. URL http://www.mdpi.com/1424-8220/15/2/2774.
- Matteson and James (2013) David S. Matteson and Nicholas A. James. A nonparametric approach for multiple change point analysis of multivariate data. Journal of the American Statistical Association, 109(505):334–345, 2013. URL https://arxiv.org/abs/1306.4933v2.
Nanduri et al. (2016)
Anvardh Nanduri, M S Candidate, and Lance Sherry.
Anomaly detection in aircraft data using recurrent neural networks (rnn).2016.
- (12) N Lawrence Ricker. Tennessee eastman challenge archive. URL http://depts.washington.edu/control/LARRY/TE/download.html.
- Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html.
- Tieleman and Hinton (2012) T Tieleman and G Hinton. Networks for Machine Learning (lecture 6.5), 2012.