Fault Detection and Identification using Bayesian Recurrent Neural Networks

by   Weike Sun, et al.
Exxon Mobil Corporation

In processing and manufacturing industries, there has been a large push to produce higher quality products and ensure maximum efficiency of processes. This requires approaches to effectively detect and resolve disturbances to ensure optimal operations. While the control system can compensate for many types of disturbances, there are changes to the process which it still cannot handle adequately. It is therefore important to further develop monitoring systems to effectively detect and identify those faults such that they can be quickly resolved by operators. In this paper, a novel probabilistic fault detection and identification method is proposed which adopts a newly developed deep learning approach using Bayesian recurrent neural networks (BRNNs) with variational dropout. The BRNN model is general and can model complex nonlinear dynamics. Moreover, compared to traditional statistic-based data-driven fault detection and identification methods, the proposed BRNN-based method yields uncertainty estimates which allow for simultaneous fault detection of chemical processes, direct fault identification, and fault propagation analysis. The outstanding performance of this method is demonstrated and contrasted to (dynamic) principal component analysis, which are widely applied in the industry, in the benchmark Tennessee Eastman process (TEP) and a real chemical manufacturing dataset.


page 20

page 21

page 23

page 30


Modeling and Soft-fault Diagnosis of Underwater Thrusters with Recurrent Neural Networks

Noncritical soft-faults and model deviations are a challenge for Fault D...

Recurrent neural network based decision support system

Decision Support Systems (DSS) in complex installations play a crucial r...

An improved mixture of probabilistic PCA for nonlinear data-driven process monitoring

An improved mixture of probabilistic principal component analysis (PPCA)...

Probabilistic Bearing Fault Diagnosis Using Gaussian Process with Tailored Feature Extraction

Rolling bearings are subject to various faults due to its long-time oper...

Explainability: Relevance based Dynamic Deep Learning Algorithm for Fault Detection and Diagnosis in Chemical Processes

The focus of this work is on Statistical Process Control (SPC) of a manu...

Bayesian Assessment of a Connectionist Model for Fault Detection

A previous paper [2] showed how to generate a linear discriminant networ...

Plant-wide fault and disturbance screening using combined transfer entropy and eigenvector centrality analysis

Finding the source of a disturbance or fault in complex systems such as ...

1 Introduction

2 Background

3 Methodology for fault detection and identification

4 Case studies

5 Conclusion


  • Qin (2009) S. J. Qin, Data-driven fault detection and diagnosis for complex industrial processes, IFAC Proceedings Volumes 42 (2009) 1115–1125.
  • Ge et al. (2013) Z. Ge, Z. Song, F. Gao, Review of recent research on data-based process monitoring, Industrial & Engineering Chemistry Research 52 (2013) 3543–3562.
  • Yin et al. (2014) S. Yin, S. X. Ding, X. Xie, H. Luo, A review on basic data-driven approaches for industrial process monitoring, IEEE Transactions on Industrial Electronics 61 (2014) 6418–6428.
  • Chiang et al. (2000) L. H. Chiang, E. L. Russell, R. D. Braatz, Fault detection and diagnosis in industrial systems, Springer Science & Business Media, 2000.
  • Jolliffe (2011) I. Jolliffe, Principal component analysis, Springer, 2011.
  • Kourti and MacGregor (1996) T. Kourti, J. F. MacGregor, Multivariate SPC methods for process and product monitoring, Journal of quality technology 28 (1996) 409–428.
  • Ku et al. (1995) W. Ku, R. H. Storer, C. Georgakis, Disturbance detection and isolation by dynamic principal component analysis, Chemometrics and intelligent laboratory systems 30 (1995) 179–196.
  • Lee et al. (2004) J.-M. Lee, C. Yoo, S. W. Choi, P. A. Vanrolleghem, I.-B. Lee, Nonlinear process monitoring using kernel principal component analysis, Chemical engineering science 59 (2004) 223–234.
  • Hsieh (2007) W. W. Hsieh, Nonlinear principal component analysis of noisy data, Neural Networks 20 (2007) 434–443.
  • Zarei et al. (2014) J. Zarei, M. A. Tajeddini, H. R. Karimi, Vibration analysis for bearing fault detection and classification using an intelligent filter, Mechatronics 24 (2014) 151–157.
  • Chine et al. (2016) W. Chine, A. Mellit, V. Lughi, A. Malek, G. Sulligoi, A. M. Pavan, A novel fault diagnosis technique for photovoltaic systems based on artificial neural networks, Renewable Energy 90 (2016) 501–512.
  • Ince et al. (2016) T. Ince, S. Kiranyaz, L. Eren, M. Askar, M. Gabbouj,

    Real-time motor fault detection by 1-D convolutional neural networks,

    IEEE Transactions on Industrial Electronics 63 (2016) 7067–7075.
  • Jia et al. (2016) F. Jia, Y. Lei, J. Lin, X. Zhou, N. Lu, Deep neural networks: A promising tool for fault characteristic mining and intelligent diagnosis of rotating machinery with massive data, Mechanical Systems and Signal Processing 72 (2016) 303–315.
  • Wu and Zhao (2018) H. Wu, J. Zhao, Deep convolutional neural network model based chemical process fault diagnosis, Computers & Chemical Engineering 115 (2018) 185–197.
  • Hu and Jiang (2018) Z. Hu, P. Jiang, An imbalance modified deep neural network with dynamical incremental learning for chemical fault diagnosis, IEEE Transactions on Industrial Electronics 66 (2018) 540–550.
  • Lee et al. (2017) K. B. Lee, S. Cheon, C. O. Kim, A convolutional neural network for fault classification and diagnosis in semiconductor manufacturing processes, IEEE Transactions on Semiconductor Manufacturing 30 (2017) 135–142.
  • Li et al. (2014) W. Li, A. Monti, F. Ponci, Fault detection and classification in medium voltage DC shipboard power systems with wavelets and artificial neural networks, IEEE Transactions on Instrumentation and Measurement 63 (2014) 2651–2665.
  • Zhang and Zhao (2017) Z. Zhang, J. Zhao,

    A deep belief network based fault diagnosis model for complex chemical processes,

    Computers & Chemical Engineering 107 (2017) 395–407.
  • Malhotra et al. (2015) P. Malhotra, L. Vig, G. Shroff, P. Agarwal, Long short term memory networks for anomaly detection in time series,

    in: Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Presses universitaires de Louvain, 2015, pp. 89–94.

  • Patan et al. (2008) K. Patan, M. Witczak, J. Korbicz, Towards robustness in neural network based fault diagnosis, International Journal of Applied Mathematics and Computer Science 18 (2008) 443–454.
  • Moustapha and Selmic (2008) A. I. Moustapha, R. R. Selmic, Wireless sensor network modeling using modified recurrent neural networks: Application to fault detection, IEEE Transactions on Instrumentation and Measurement 57 (2008) 981–988.
  • Talebi et al. (2008) H. A. Talebi, K. Khorasani, S. Tafazoli, A recurrent neural-network-based sensor and actuator fault detection and isolation for nonlinear systems with application to the satellite’s attitude control subsystem, IEEE Transactions on Neural Networks 20 (2008) 45–60.
  • Wang et al. (2016) L. Wang, Z. Zhang, H. Long, J. Xu, R. Liu, Wind turbine gearbox failure identification with deep neural networks, IEEE Transactions on Industrial Informatics 13 (2016) 1360–1368.
  • Nie et al. (2018) L. Nie, Y. Li, X. Kong, Spatio-temporal network traffic estimation and anomaly detection based on convolutional neural network in vehicular ad-hoc networks, IEEE Access 6 (2018) 40168–40176.
  • Miller et al. (1998) P. Miller, R. E. Swanson, C. E. Heckler, Contribution plots: a missing link in multivariate quality control, Applied mathematics and computer science 8 (1998) 775–792.
  • Westerhuis et al. (2000) J. A. Westerhuis, S. P. Gurden, A. K. Smilde, Generalized contribution plots in multivariate statistical process monitoring, Chemometrics and intelligent laboratory systems 51 (2000) 95–114.
  • Wise et al. (1989) B. M. Wise, N. L. Ricker, D. J. Veltkamp, Upset and sensor failure detection in multivariate processes, in: AIChE Annual Meeting, 1989.
  • Gal and Ghahramani (2016a) Y. Gal, Z. Ghahramani, A theoretically grounded application of dropout in recurrent neural networks, in: Advances in Neural Information Processing Systems, 2016a, pp. 1019–1027.
  • Gal and Ghahramani (2016b) Y. Gal, Z. Ghahramani, Dropout as a Bayesian approximation: Representing model uncertainty in deep learning, in: Proceedings of the International Conference on Machine Learning, 2016b, pp. 1050–1059.
  • Elman (1990) J. L. Elman, Finding structure in time, Cognitive science 14 (1990) 179–211.
  • Wu et al. (2016) Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al., Google’s neural machine translation system: Bridging the gap between human and machine translation, arXiv preprint arXiv:1609.08144 (2016).
  • Jozefowicz et al. (2016) R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, Y. Wu, Exploring the limits of language modeling, arXiv preprint arXiv:1602.02410 (2016).
  • Merity et al. (2016) S. Merity, C. Xiong, J. Bradbury, R. Socher, Pointer sentinel mixture models, arXiv preprint arXiv:1609.07843 (2016).
  • Jordan (1997) M. I. Jordan, Serial order: A parallel distributed processing approach, in: Advances in psychology, volume 121, Elsevier, 1997, pp. 471–495.
  • Werbos (1990) P. J. Werbos, Backpropagation through time: what it does and how to do it, Proceedings of the IEEE 78 (1990) 1550–1560.
  • Bengio et al. (1994) Y. Bengio, P. Simard, P. Frasconi, et al., Learning long-term dependencies with gradient descent is difficult, IEEE Transactions on Neural Networks 5 (1994) 157–166.
  • Hochreiter and Schmidhuber (1997) S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (1997) 1735–1780.
  • Cho et al. (2014) K. Cho, B. Van Merriënboer, D. Bahdanau, Y. Bengio,

    On the properties of neural machine translation: Encoder-decoder approaches,

    in: Proceedings of the Eight Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), 2014, pp. 103–111.
  • Graves et al. (2013) A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2013, pp. 6645–6649.
  • Cakir et al. (2015) E. Cakir, T. Heittola, H. Huttunen, T. Virtanen, Polyphonic sound event detection using multi label deep neural networks, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN), IEEE, 2015, pp. 1–7.
  • Pawlowski et al. (2017) N. Pawlowski, A. Brock, M. C. Lee, M. Rajchl, B. Glocker, Implicit weight uncertainty in neural networks, arXiv preprint arXiv:1711.01297 (2017).
  • Fortunato et al. (2017) M. Fortunato, C. Blundell, O. Vinyals, Bayesian recurrent neural networks, arXiv preprint arXiv:1704.02798 (2017).
  • Louizos and Welling (2017) C. Louizos, M. Welling, Multiplicative normalizing flows for variational Bayesian neural networks, in: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, 2017, pp. 2218–2227.
  • Hernández-Lobato and Adams (2015) J. M. Hernández-Lobato, R. Adams, Probabilistic backpropagation for scalable learning of Bayesian neural networks, in: Proceedings of the International Conference on Machine Learning, 2015, pp. 1861–1869.
  • Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15 (2014) 1929–1958.
  • Pham et al. (2014) V. Pham, T. Bluche, C. Kermorvant, J. Louradour, Dropout improves recurrent neural networks for handwriting recognition, in: Proceedings of the International Conference on Frontiers in Handwriting Recognition, IEEE, 2014, pp. 285–290.
  • Pachitariu and Sahani (2013) M. Pachitariu, M. Sahani, Regularization and nonlinearities for neural language models: when are they needed?, arXiv preprint arXiv:1301.5650 (2013).
  • Breunig et al. (2000) M. M. Breunig, H.-P. Kriegel, R. T. Ng, J. Sander, LOF: identifying density-based local outliers, in: Proceedings of the ACM SIGMOD International Conference on Management of Data, ACM, 2000, pp. 93–104.
  • Duda et al. (2001) R. Duda, P. Hart, D. Stork, Pattern classification, Pattern Classification and Scene Analysis: Pattern Classification, Wiley, 2001.
  • Zhu and Braatz (2014) X. Zhu, R. D. Braatz, Two-dimensional contribution map for fault identification [focus on education], IEEE Control Systems Magazine 34 (2014) 72–77.
  • Jackson and Mudholkar (1979) J. E. Jackson, G. S. Mudholkar, Control procedures for residuals associated with principal component analysis, Technometrics 21 (1979) 341–349. doi:10.1080/00401706.1979.10489779.
  • Downs and Vogel (1993) J. J. Downs, E. F. Vogel, A plant-wide industrial process control problem, Computers & chemical engineering 17 (1993) 245–255.
  • De Ketelaere et al. (2015) B. De Ketelaere, M. Hubert, E. Schmitt,

    Overview of PCA-based statistical process-monitoring methods for time-dependent, high-dimensional data,

    Journal of Quality Technology 47 (2015) 318–335.
  • Venkatasubramanian et al. (2003) V. Venkatasubramanian, R. Rengaswamy, K. Yin, S. N. Kavuri, A review of process fault detection and diagnosis: Part I: Quantitative model-based methods, Computers & chemical engineering 27 (2003) 293–311.
  • Franklin et al. (1995) S. B. Franklin, D. J. Gibson, P. A. Robertson, J. T. Pohlmann, J. S. Fralish, Parallel analysis: a method for determining significant principal components, Journal of Vegetation Science 6 (1995) 99–106.
  • Sun and Braatz (2018) W. Sun, R. D. Braatz, Review and comparative study of nonlinear PCA fault detection methods, 2018. AIChE Annual Meeting.
  • Russell et al. (2000) E. L. Russell, L. H. Chiang, R. D. Braatz, Fault detection in industrial processes using canonical variate analysis and dynamic principal component analysis, Chemometrics and intelligent laboratory systems 51 (2000) 81–93.
  • Chiang and Braatz (2003) L. H. Chiang, R. D. Braatz, Process monitoring using causal map and multivariate statistics: fault detection and identification, Chemometrics and intelligent laboratory systems 65 (2003) 159–178.
  • Valle et al. (1999) S. Valle, W. Li, S. J. Qin, Selection of the number of principal components: the variance of the reconstruction error criterion with a comparison to other methods, Industrial & Engineering Chemistry Research 38 (1999) 4389–4401.