1 Introduction
Twophase flow regime is an important concept for severe accident prediction and prevention in twophase flow systems such as the reactor pressure vessel in the nuclear power plant. It serves as an engineering reference that classifies flow characteristics. Many twophase flow models are based on flow regimes. Thus, an accurate prediction on flow regime can be regarded as the first step towards an accurate twophase flow prediction. The analysis of the twophase flow regime and its transitions has quite a long history. Flow regime maps were developed for different flow geometries
[1, 2, 3]. Flow regimes are determined by using twophase parameters that can be experimentally obtained. In the early times, the flow regime maps are either created using experimental data or theoretical approaches [1]. Recently, the flow regime identifications are developed with the help of machine learning techniques.
Over the past years, exclusive work has been done on twophase flow regime identification using machine learning algorithms. Among them, supervised multilayer feedforward neural networks (NN), supported vector machine (SVM)
[4]and selforganized map (SOM)
[5] are the most used algorithms and other generated algorithms are more or less based on these algorithms. [6, 7, 8] They basically require to input all the data to extract the different key features and make predictions. Although these methods are proved to be accurate, certain shortcomings of these methods are: 1) most of the algorithms are determining static flow regime maps that are used as engineering references, while the structures and twophase flow is changing dynamically; 2) the flow regime are only be determined afterwards. However, severe accidents, such as nuclear power plant core melting, often happens at sudden, a fast response and accident prediction is essentially needed. In this paper, a dynamic RNN approach is proposed for the twophase flow regime prediction. A LSTMbased deep RNN is constructed and trained using existing database and the performance is evaluated and analyzed in this paper.2 Recurrent Neural Network
RNN is structurally suitable for a timeseries prediction. Conventional RNN can process time series data temporally and dynamically based on hidden Markov model (HMM), which makes RNN to be able to capture longdistance dependencies. However, RNN could fall into the trouble of gradient vanishing and exploring during model training. Long shortterm memory (LSTM) networks
[9] was created and solved this problem properly by managing the passes of the information. In a LSTM system, the recurrent hidden layers are computing with the selfconnected memory cells and three gates for obtaining the outputs. The key of LSTM in solving the gradient vanishing and exploring is to optionally ignore some of the inputs so that they aren’t used for the updates of parameters in the hidden states.Given a timedependent void fraction sequence = [, , , …, ], the mathematical expressions for the operation of one LSTM hidden cell t, are given as follows,
(1)  
where W is the weights for each parameter at certain state. ht
is the vector sequence of the hidden cell. In this model, the activation functions are sigmoid function,
, and tanh function,. A LSTM system usually contains multiple connected cells among which the outputs from the preceding cell are the inputs of the following cell. The characteristics of the two phase flow is able to pass thought the model. The output of the model is a probability distribution of all the possible flow regimes. The predicted flow regime is the one with the highest probability, obtained as follows,
The prediction on flow regime using LSTMbased RNN has its advantages. Firstly, the input sequence is segmented. Each input node in the sequence represents the state of the flow regime at certain time. Secondly, the relation between the sequence and the output is rather tight since the sequence hardly contains unrelated noises. However, since the mechanisms in related with the transition and development of the two phase flow regime is complicated and the number of dependencies is large, a deep LSTM network is still needed.
This paper follows a similar approach of constructing deep LSTM network with [10]. Their ideas of constructing deep RNN network is as follows: 1) inputhidden; 2) hiddenhidden; 3) hiddenoutput. Based on the ideas, 5 different types of RNN network are constructed by combining different sublayers. In terms of constructing the network, one important consideration is that with the basis of accuracy, the network should respond quickly. This means that the setting of the number of the parameters should balance both the requirement of accuracy and the calculation latency.
3 Experiments
We evaluated the LSTMbased DRNN on the current existing timeseries void fraction signal database [7]. The details on experiment setup, database, and the experimental results are discussed below.
3.1 Database
The twophase flow regime parameters that can describe the flow regime could be classified into two groups. The parameters in the first group directly describe the flow regime characteristics, such as void fraction and interfacial area concentration. The parameters in the second group also include the flow regime characteristics such as local pressure in the system. Since parameters in both groups include the characteristics of flow regime, they could be used as the input parameters for the flow regime prediction.
Time series data containing twophase flow regime characteristics can be obtained using many twophase flow measurement instrumentations. In the lab setup, gamma densitometer is a very accurate and stable method because it is nonintrusive, almost flowregime dependencefree instrumentation. [12] Conductivity probe is another accurate instrumentation yet its setup is relatively difficult. Impedance void meter, as an engineering reference, is also nonintrusive yet its accuracy dependence on the flow regime and void distribution.[13] In terms of industrial application, differential pressure gauge is a convenient and economical choice, yet its accuracy of measurement is not satisfying. All of these mentioned above can provide the time series data that contains the void fraction changing characteristics. The database used in this experiment [7] is collected using impedance meter.
In the database, flow regime is classified into 5 types including bubbly, cap bubbly, slug, churnturbulent, and annular flow. The database contains 200 test conditions in total, and each test condition consists of an impedance signal with a measurement period of 60 seconds and data acquisition frequency of 10kHz. For each test condition, signal is ranging from 0 to 1, with 0 representing full water case and 1 representing full air case, signal fluctuating between 0 and 1 according to the flow regime characteristics. 3
shows the 1 second time signal, probability density functions (PDF) and the cumulative probability density functions (CPDF) to characterize each flow regime. The PDF or CPDF profiles are usually treated as inputs in SOM or SVM methods.
Data augmentation was performed with the original database. Two methods are used in this experiment and they are summarized below.

Since the experiment was performed at steadystate conditions, meaning that the flow regime state is not changing during data collection, data could be segmented into shorter pieces. In the following data sensitivity analysis part, the performance in terms of the length of the data is evaluated and discussed.

The timeseries signals are reversed in order. In this way, the number of data is doubled.
3.2 Experimental setup
The void fraction signal is firstly put into one ReLUNN layer for the primary feature extraction. The ReLUNN layer (or ReLU layer) stands for feedforward neural network layer with ReLU as the activation function. Then it passes through LSTM layers and ReLU layers. The structures of these parts are modified and investigated for the better prediction. The final output stage of the network is a softmax layer with a size of the number of all possible flow regime. The networks are established with tensorflowbased Keras. The optimizer used for all the networks in the experiment is Adam and the loss function is categorical crossentropy.
For the model training, although LSTM unit is featured with good preventing of gradient vanishing and exploration, creating and training and good LSTMbased RNN model require good methods and tricks. Some of the important methodologies used towards building and training a wellestablished RNN model are utilized during the model training.

Gate weight initiation: Gates are the keys for a LSTM unit. Gate initial states can have large effect on the training process. It has been proved that deep neural networks can converge more rapidly using orthogonal initiation that generates a random orthogonal matrix.
[14] 
Dropout and regularization: To prevent from overfitting and better training the model, the following regularization methods are used: early stopping and learning rate reduction. The training time is largely decreased due to these two methods. By using early stopping, the training process will stop if testing accuracy has not been improved for 3 epochs. With learning rate reduction, the learning rate is initialized as 0.01 and it can decease according to the training performance. The minimum learning rate allowed is 0.0001.
3.3 Sensitivity Study
The length of the timeseries data can affect the flow regime prediction. This is obvious because a very short sequence may not include all the key information of the characteristics of the two phase flow. However, if the sequence is very long, the prediction will not be a "realtime" prediction. Therefore, it is essential to determine the proper length of the sequence for the flow regime prediction.
Different flow regime contains different characteristics and the sequence length needed for each characteristic to be presented is different. From 3, the signal variations of typical bubbly and annular twophase flow is small and the PDF curve contains only one peak. This means that the characteristics of these twophase flows are uniform over the time and a short sequence could present these characteristics. In contrast, the signal fluctuation of the flow regimes like slug and churnturbulent twophase flow usually quite large. Thus, these flow regimes usually require a longer sequence. Since our objective is to classify the flow regimes, the length of sequence is determined by the flow regime that requires the longest sequence. Besides, other hydrodynamic parameters such as flow rate should also be considered when determining the sequence length. It is obvious that flow rate determines the speed of the twophase flow passing through the measurement area, thus affecting the sequence length needed.
In this study, the experimental data are segmented into different sequence lengths and they are used separately for training the same model structure. The lengths and the training performances are given in table 1. The selection of these lengths is by considering the types of flow regimes and test conditions included in the database. The model structure used in this section, LTSM2ReLU, From the table, the prediction accuracy of the model increases as the sequence length becomes longer. In terms of the performance tendency, the performance of the model trained using data with 3 seconds sequence length drastically worsen compared with result of 5 seconds. This may not be a general conclusion but it provides a method of determining the input data size.
Seq.Len., (sec.)  Test Accuracy on LSTM2ReLU () 

20  95.6 
10  92.3 
5  86.7 
3  73.5 
3.4 Result and Discussion
Eight different RNNs were evaluated and their performances of each model are summarized in Table 2. These models vary in terms of 3 aspects: the number of hidden layers: both LSTM and ReLU layers; the number of LSTM cells in each hidden layer. Also following [10], the performance of stacking layers is also analyzed with model (LSTM128H2ReLU)2 and (LSTM128H2ReLU)3. The relative prediction time needed for the same number of test cases is also given in the table.
It can be seen that the test accuracy increases as the network becomes deeper. The accuracy can be improved by adding either LSTM layer or ReLU layer, and adding LSTM layer benefits more than adding ReLU layer. Increasing the number of cells in each layer can also improve the accuracy. However, increasing the number of layers or number of cells can also increase the prediction time, which is not good for our general purpose. Comparing the network LSTM128H2ReLU and LSTM128H1ReLU, which we consider the effect of reducing the size of network, the test accuracy greatly drops from 86.7 to 78.5. This is probably because the number of parameters in the network is lower than the minimum requirement of modeling the experiment cases. Besides, the comparison between adding single hidden layers and stacking networks (e.g. 2LSTM128H2ReLU and (LSTM128H2ReLU)2) shows that 2 stacked network doesn’t outperform the 2 intermediate LSTM layer network. However, the author considers that it cannot lead to a solid conclusion that stacking deep networks is not as beneficial as intermediate hidden layers. The result of this paper may be affected by the total number of training data. Further studies can be performed if more data is available.
Network Descriptions  Test Accuracy ()  Relative prediction time 

LSTM128H2ReLU  86.7  1.00 
LSTM256H2ReLU  88.3  1.56 
2LSTM128H2ReLU  91.7  2.18 
3LSTM128H2ReLU  92.3  2.81 
LSTM128H1ReLU  78.5  0.84 
LSTM128H3ReLU  87.1  1.32 
(LSTM128H2ReLU)2  90.2  4.28 
(LSTM128H2ReLU)3  91.1  6.11 
4 Conclusions and future work
The paper developed a methodology of using deepRNNs for flow regime prediction that can achieve both accuracy and fast response. The method could be extended to the prediction with any timeseries database that records the levels and the variations of twophase parameters, such as void fraction and interfacial area concentration, over time.
References
 [1] K Mishima and M Ishii. Flow regime transition criteria for upward twophase flow in vertical tubes. International Journal of Heat and Mass Transfer, 27(5):723–737, 1984.
 [2] K Mishima and T Hibiki. Some characteristics of airwater twophase flow in small diameter vertical tubes. International journal of multiphase flow, 22(4):703–712, 1996.
 [3] D Barnea, Y Luninski, and Y Taitel. Flow pattern in horizontal and vertical two phase flow in small diameter pipes. The Canadian Journal of Chemical Engineering, 61(5):617–620, 1983.
 [4] C Cortes and V Vapnik. Supportvector networks. Machine learning, 20(3):273–297, 1995.
 [5] T Kohonen. The selforganizing map. Proceedings of the IEEE, 78(9):1464–1480, 1990.
 [6] Y Mi, M Ishii, and LH Tsoukalas. Vertical twophase flow identification using advanced instrumentation and neural networks. Nuclear Engineering and Design, 184(23):409–420, 1998.
 [7] Z Dang, Y Zhao, G Wang, P Ju, Q Zhu, X Yang, R Bean, and M Ishii. Investigation of the effect of the electrode distance on the impedance void meter performance in the twophase flow measurement. Experimental Thermal and Fluid Science, 101:283–295, 2019.
 [8] Y Zhou, F Chen, and B Sun. Identification method of gasliquid twophase flow regime based on image multifeature fusion and support vector machine. Chinese Journal of Chemical Engineering, 16(6):832–840, 2008.
 [9] S Hochreiter and J Schmidhuber. Long shortterm memory. Neural computation, 9(8):1735–1780, 1997.
 [10] X Li and X Wu. Constructing long shortterm memory based deep recurrent neural networks for large vocabulary speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4520–4524. IEEE, 2015.
 [11] A Graves, Ar Mohamed, and G Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645–6649. IEEE, 2013.
 [12] C. Eberle, M. Ishii, and S Revankar. A review of gamma densitometer design and measurement in twophase flows, pu/ne92/3. Technical report, Purdue University, 1992.
 [13] G Hewitt. Measurement of two phase flow parameters. Nasa Sti/recon Technical Report A, 79, 1978.
 [14] A M Saxe, J L McClelland, and S Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Comments
There are no comments yet.