I Introduction
Fluid flow around aerodynamic configurations can experience complex nonlinearities with a wide range of spatial and temporal features. These are mainly related to nonlinear convection term and turbulence that often cannot be treated by simple linearization. Many of complex fluid flows evolve on a lowdimensional subspace that may be characterized by dominant spatiotemporal coherent structures. It is of interest in the analysis of unsteady fluid flows to extract dominant features and introduce a reduced model of the complex system based on physically important features. This performs typically through the modal decomposition of a numerical or experimental dataset of the flowfield. However, a problem arises when attempting model reduction of unsteady flows, where long term transient phenomena need to be predicted accurately. Some examples of common unsteady flow features and phenomena are von Kármán shedding, Kelvin–Helmholtz instability, and vortex pairing/merging Taira et al. (2017). Reduced order modeling of the fluid systems is a challenging task due to the existence of these complex phenomena at multiple spatiotemporal scales. It means that they are difficult to reduce to a lowdimensional subspace without losing at least some of these scales. During the last three decades, several efforts in theoretical foundations, numerical investigations, and methodological improvements have made it possible to develop general ideas in reduced order modeling and to tackle several problems arising in fluid dynamics Stankiewicz et al. (2016); Mohebujjaman et al. (2019); Star et al. (2019). Proper orthogonal decomposition (POD) Lumley (1967); Lumey (1970), dynamic mode decomposition (DMD) Schmid (2010), and Koopman analysis Rowley et al. (2009) are some of the wellknown reduced order methods (ROMs) in the field of fluid dynamics. POD was first introduced by Lumley Lumley (1967) to the fluid dynamics community for the study of the turbulent boundary layer. It is a modal decomposition method designed to identify the features (a set of orthogonal modes) of flow most important for reconstructing a dataset. POD modes are not necessarily optimal for modeling dynamical systems while the modes do not depend on the time evolution/dynamics encoded in the data Tu et al. . DMD is a method for analyzing the time evolution of a dynamical system. It is originated in the fluid dynamics community and first was introduced by Schmid and Sesterhenn Schmid and Sesterhenn (2008). Only a year later, Rowley et al. Rowley et al. (2009) presented a technique for describing the global behavior of complex nonlinear flows by decomposing the flow into modes determined from spectral analysis of the Koopman operator Koopman (1931). They showed that DMD could be considered as an algorithm that calculates an approximate Koopman decomposition. Schmid Schmid (2010) followed this with his article in 2010 and demonstrated the DMD capability in providing an accurate decomposition of complex systems into coherent spatiotemporal structures that may be used for shorttime future state prediction and control. DMD is an equationfree datadriven method based on the power of the singular value decomposition (SVD), and it has been employed for modal analysis of a variety of fluid flows Alekseev et al. (2016); Bistrian and Navon (2017).
Deep learning is a subset of machine learning methods based on artificial neural networks, which is capable of extracting hidden information with multiple levels of representation from nonlinear and complex dynamical systems LeCun et al. (2015). Lately, deep learning has made its mark in various areas such as virtual assistants, image and video processing, speech recognition, genetics and disease diagnosis Sainath et al. (2013); Krizhevsky et al. (2012); Abdolhosseini et al. (2019).
In recent years, efficient strategies have been emerged in the field of fluid mechanics based on machine learning (ML) and deep neural networks (DNNs). Complementary information on the use of ML for fluid dynamics can be found in the works of Kutz Kutz (2017) and Brunton et al. Brunton et al. (2020)
. It was shown that the autoencoder network could be efficiently used as a nonlinear principal component analysis (PCA) technique for dimensionality reduction of the fluid flows rather than the linear SVD. Moreover, the ability of recurrent neural networks (RNNs) for prediction of the sequential data brings the idea of using RNNs for learning dynamics and future state estimation of the dynamical systems. The advantage of using deep neural networks rather than the Galerkin projection technique has been stated in several research studies
Pawar,S. et al. (2019). Besides, a type RNNs, the long shortterm memory (LSTM), which has shown attractive potential in the modeling of the sequential problems such as speech modeling and language translation, is used for prediction of the flow evolution, e.g., learning dynamic and prediction of the turbulent shear flows Srinivasan et al. (2019). Moreover, ML strategies have been used for extracting information from highfidelity data to inform lowfidelity numerical simulations for higher accuracy. The specific aim is to use DNNs to build an improved representation of the Reynolds stress anisotropy tensor from highfidelity simulation data
Ling et al. (2016); Duraisamy et al. (2019). Besides, deep neural networks have been used for learning the physics of the flow over aerodynamic structures to hasten the process of geometrical optimization Storti et al. (2019). Recently, deep reinforcement learning has been used for active flow control and drag reduction
Rabault et al. (2019). In brief, dimensionality reduction, feature extraction, superresolution, reducedorder modeling, turbulence closure, shape optimization, and flow control are some of the critical tasks in fluid dynamics that could be enhanced with the implementation of the ML algorithms.
Unsteady flows are, in essence, high dimensional dynamical systems. The use of ML for modelfree prediction of spatiotemporal dynamical systems purely from the system’s past evolution is of interest to both scientists and industries in the field of fluid dynamics. Therefore, the development of nonintrusive reduced order models (NIROMs) is of particular interest in the fluid dynamics community. In the context of ROM, both dimensionality reduction and future state prediction could be advanced from deep learning strategies. Baldi and Hornik Baldi and Hornik (1989)
showed that an autoencoder network with one hidden layer and linear activation functions could closely resemble the standard POD/PCA decomposition. Wang et al.
Wang et al. (2016) used a deep autoencoder architecture for reduced order modeling and dimensionality reduction of distributed parameter systems. Liu et al. Liu et al. (2019) proposed a novel compression method based on generative adversarial network (GAN), which had significant advantages in compression time. More recently, Murata et al. Murata et al. (2020)developed a new nonlinear mode decomposition method based on a convolutional neural network (CNN) autoencoder. Implementation of nonlinear activation functions leads to lower reconstruction errors than POD, and nonlinear embedding considered the key to improving the capability of the model. Although the deep learning method is used to make dimensionality reduction, they do not take advantage of its predictive capability. Moreover, Guastoni et al. used convolutionalnetwork models for the prediction of wallbounded turbulence from wall quantities
Guastoni et al. (2020, 2020).RNNs are of particular interest to fluid mechanics due to their ability in learning and prediction of the sequential data. The renewed interest in RNNs has largely attributed to the development of the LSTM algorithms. Commonly, the Galerkin projection has been used to obtain the evolution equations for the lower order system with POD. However, PODGalerkin method is an intrusive ROM (IROM), and several studies have been conducted using ANNs San et al. (2019); San and Maulik (2018), RNNs Pawar,S. et al. (2019), and LSTM networks Wang et al. (2018); Mohan and Gaitonde (2018); Ahmed et al. (2019) to construct an NIROM based on POD modes.
Besides, datadriven finite dimensional approximations of the Koopman operator has been received significant attention in recent years, in particular, for problems dealing with complex spatiotemporal behaviors such as unsteady fluid flows. DMD, in its original formulation, implicitly utilizes linear observables of the state of the system. Extended DMD Williams et al. (2015) and HankelDMD Arbabi and Mezić (2017) were proposed to include a richer set of observables that spans a Koopman invariant subspace. Recently, several works have been done, introducing fully datadriven approaches for learning Koopman embedding using deep neural networks (DNNs) (Lusch et al., 2018; Takeishi et al., 2017; Li,Qianxiao et al., 2017). Morton et al. Morton et al. (2018) presented a new architecture based on the work by Takeishi et al. Takeishi et al. (2017) to predict the time evolution of a cylinder system. Their approach was grounded in Koopman’s theory and aimed to estimate a set of functions that transform data into a form in which a linear leastsquares regression fits well. However, it has been shown by Khodkar et al. Khodkar et al. (2019) that the linear combination of a finite number of modes may not accurately reproduce the nonlinear characteristics of a chaotic dynamics for a reasonably long period of time. They showed that adding nonlinearities to the linear model as a forcing term leads to an excellent shortterm prediction of several wellknown prototypes of chaos. Moreover, Eivazi et al. Eivazi et al. (2020) showed that the Koopman framework with nonlinear forcing (KNF) provides accurate predictions of the dynamical evolution of the coefficients of a loworder model for nearwall turbulence Moehlis et al. (2004).
To this end, It has been shown that adding nonlinearity to the dimensionality reduction process by using nonlinear activation functions in the autoencoder architecture can effectively enhance the efficiency of the dimensionality reduction process. However, the majority of research studies have been dedicated to the development of reduced order models based on linear compression techniques such as SVD. Moreover, promising capability of the LSTM network in the learning of the complex systems dynamic brings the opportunity to implement LSTM network for future time prediction rather than a linear mapping of the previous observations to the future time instances using DMD algorithm. In this regard, this paper presents a novel reduced order model based on deep neural networks. A deep autoencoder network with nonlinear activation functions is designed and implemented for nonlinear dimensionality reduction and dominant features extraction, and then, LSTM network is used for prediction of the flow evolution. A sequence of the extracted features from the autoencoder network is the input of the LSTM, and the output is the flowfield in the future time step. Train and test sets are acquired from numerical simulation of the flow around a cylinder and an oscillating airfoil. Two test cases are examined for the cylinder. One at a constant Reynolds number of 3900 inducing single frequency vortex shedding, and the other one at gradually decreasing Reynolds number from 3355 to 676, which leads to decay of the vortex shedding behind the cylinder, and consequently, a variable frequency dynamical system. The airfoil oscillates sinusoidally at number of . The performance of the proposed autoencoderLSTM method in future state prediction of the flow is compared with the DMD. Moreover, an autoencoderDMD algorithm is introduced for reduced order modeling, which uses the autoencoder network for dimensionality reduction rather than SVD rank truncation. The main novelties of this research study are listed below:

Introduce a novel datadriven method based on deep neural networks for nonlinear reduced order modeling of complex unsteady fluid flows

Implementation of autoencoder network for nonlinear dimensionality reduction and feature extraction of the fluid systems

Future state estimation of a nonlinear dynamical system using LSTM network from its features extracted by the autoencoder network

Comparison of linear and nonlinear nonintrusive ROM frameworks
Ii Numerical Analysis and Data Sets
Twodimensional NavierStokes equations are solved using finite volume method (FVM). Temporal and spatial properties are discretized with the secondorder implicit scheme and secondorder upwind scheme, respectively. The computational domain is a circle with Otype mesh and velocity inlet and pressure outlet as the boundary conditions. turbulence model is adopted and value is checked to be lower than one. For the unsteady flow over the cylinder, two test cases are examined. First, the flow around a cylinder at Reynolds number of 3900 (cylinder test 1) is simulated where is the kinematic viscosity, is the cylinder diameter, and is the freestream velocity. The computational domain is a circle with a radius of 20D. The Otype grid is generated with 360 equally sized grid point in the azimuthal direction, but with grid stretching in the radial direction with the total cell number of 98878. Figure 1 represents the pressure distribution over the cylinder for the present simulation against the URANS simulation of Young and Ooi (2007), which is shown a good agreement between the results. Simulation is conducted with a time step of 0.2, which results in 357 time steps per each cycle period. Snapshots are extracted from every seven time steps; consequently, 51 snapshots for each cycle. Calculations are performed for 20 complete cycles; four cycles are considered as a training set, and the next four cycles are picked for testing the networks.
At the second test case for the cylinder, freestream velocity is decreased relative to the time from its value corresponding to as , where is the time. The simulation time step is 0.2, and snapshots are collected from the velocity field at every ten time steps. 332 snapshots are extracted from the wake of the cylinder, and 166 snapshots are used as the training set. Variation of lift coefficient besides training and testing data sets are shown in Fig. 2. The gradual decrease of the freestream velocity weakens the vortex shedding and leads to the change in its frequency. Reynolds number varies from 3355 to 676, as it is depicted in Fig. 3.
For the oscillating airfoil test case, the flow around a pitch oscillating NACA0012 airfoil is simulated at Reynolds number of , where is the airfoil chord. Airfoil oscillates sinusoidally about its chord with the reduced frequency of 0.1 according to the LEE and GERONTAKOS (2004) test case. and
represent mean angle of attack and amplitude of oscillations, respectively. Airfoil oscillation is modeled with the sliding mesh technique. The data on the dynamic mesh is then interpolated on a constant mesh using Knearest neighbors (KNN) machine learning algorithm. The principle behind the KNN method is to detect a predefined number of training instances that are close to the new point and compute the value based on the mean of these nearest neighbors’ values. Complementary information about data interpolation using ML techniques can be found in work by Naderi et al.
Hossein et al. (2019). Figure 4 presents the lift coefficient versus angle of attack obtained from present simulation against experimental and numerical data LEE and GERONTAKOS (2004); Gharali and Johnson (2013). For this test case, the data corresponding to two cycles of airfoil’s oscillations are used as the training data set and the next two cycles are predicted.Iii Methodology
To extract dominant features of the unsteady flowfield around the test cases, an autoencoder network is designed and trained with the snapshots obtained from CFD simulation. These features are organized as temporal sequences that are used as the inputs of the LSTM network with the aim of the prediction of the flowfield at future time steps. In contrast with the SVD, which is a basic part of modal decomposition techniques, autoencoder with nonlinear activation functions provides a nonlinear dimensionality reduction. In the following, an elaboration on the networks architecture, inputs and outputs of each network, training and optimization algorithms, and selection of hyperparameters is presented.
iii.1 Autoencoder Neural Network
Autoencoder neural networks are a type of network architecture developed for unsupervised feature extraction and dimensionality reduction. The main incentive for utilizing autoencoder architecture in the context of ROM is to perform a nonlinear dimensionality reduction to possibly enhance the efficiency of the process. In this architecture, the inputs and the outputs are the same with a size larger than the dimension of the hidden layers. The network comprises of two parts, namely, encoder and decoder. The encoder part, , maps the inputs to a latent space,
, with a lower dimension by decrease of the number of neurons in hidden layers. Then, the decoder part,
, projects the latent space back to the original space according to Eq. 1. The middle layer between encoder and decoder is called the bottleneck layer and may be used for the extraction of the most dominant features from the input data. In this study, the input and output data of the autoencoder network are the velocity magnitudes on the computational grid points obtained from CFD simulation of the flowfield over the test cases. The data on the twodimensional CFD grid is first flattened and then fed to the network. The architecture of the autoencoder network, encoder and decoder parts, and the bottleneck layer are depicted in Fig. 5.(1) 
iii.2 Long ShortTerm Memory Network
Long shortterm memory (LSTM) was proposed by Hochreiter and Schmidhuber in 1997 Hochreiter and Schmidhuber (1997). LSTM is an artificial recurrent neural network (RNN) architecture designed for learning of the sequential data such as numerical or experimental time series and to overcome the forgetting problem in conventional RNNs Goodfellow et al. (2016). An LSTM neural network unit is comprised of an input gate, a cell state, a forget gate, and an output gate (Fig. 6). The cell state can transfer important data throughout the processing of the sequence. Therefore the information from the earlier sequences or time steps can move to later time steps, decreasing the impacts of shortterm memory. While the cell state tries to transfer the information, data get appended or eliminated to the cell state by gates. The gates are various neural networks that choose which data is permitted on the cell state.

Forget Gate
Data from the previous time step () and information from the current time step (
) is given through the sigmoid function. Values come out within 0 and 1. If the output is over some trained parameter, then the cell state is entirely reset to 0, mostly forgetting its long term memory.

Input Gate
The input gate decides if the data is required for the long term. Firstly, we transfer the output of the hidden state () and the labeled data input () into a sigmoid function. That determines which values will be updated by changing the amounts to be between 0 and 1. 0 indicates not essential, and 1 means important. The hidden state and current input also are passed into the function to assist in regulating the neural network. Next, the output is multiplied with the sigmoid function output. The sigmoid function output will select which data is critical to hold from the output.

Cell State
The indicates the current cell state of the LSTM neural network. The cell state is an array of numbers that are transferred through all cells in its way. The cell state is multiplied with the forget array. This has the chance of declining values in the cell state if it gets multiplied by numbers close to zero. Next, the output from the input gate and is added, which updates the cell state to a new state. That provides the latest cell state.

Output Gate
The output gate provides the following hidden state array for the next cell (). The former hidden state and the current input is passed toward a sigmoid function. Next, the recently changed cell state is transferred to the function. The output is multiplied by the sigmoid output to determine what data the hidden state should take. This is the shortterm memory character of the LSTM neural network.
iii.3 AutoencoderLSTM method for ROM
In this paper, the power of autoencoder and LSTM networks in nonlinear dimension reduction and learning of the sequential data is leveraged for nonintrusive reduced order modeling of unsteady flows. The schematic of the proposed method is presented in Fig. 7. The first step is to train the autoencoder network using the snapshot data collected from the flowfield with the aim of nonlinear dimension reduction and feature extraction (Fig. 7a). The data on the twodimensional grid is first flattened and then fed to the network. Each time step data
, which is a vector of state variables at time
, is mapped to the latent space through the function as with much lower dimension, . Then, is projected back to the original space to reconstruct the input data, . The data on the latent space, , is then used as the input to train the LSTM network with the aim of future prediction (Fig. 7b). The input of the LSTM network is a sequence of with the length of and the output is the flowfield data at the next time step of the input sequence . Note that in this way, previous time units are used for prediction of the next time step. For testing and using the autoencoderLSTM method for reduced order modeling, an iterative procedure is conducted. An initial sequence is first mapped to the latent space using the encoder () part of the autoencoder. The data on the latent space is fed to the LSTM network to predict the flowfield data for the next time step of the input sequence. The predicted value is then stacked to the input sequence ignoring the first snapshot data to prepare a sequence one step farther in time. The new sequence is fed again to the network to predict the next time step, and the procedure is performed iteratively for future prediction (Fig. 7c). All of the neural network models are created using a machine learning software framework developed by Google Research called TensorFlow
Abadi et al. (2016). The autoencoder and LSTM networks are trained by the feedforward and backpropagation algorithms. For all the training runs in this work, a variant of the stochastic gradient descent algorithm called adaptive moment estimation (Adam)
Kingma and Ba (2015)is utilized with the learning rate of 0.001. Adam has an adaptive learning rate method, which is commonly used to train deep networks. As a standard choice for regression problems, the mean squared error (MSE) is used as the loss function according to
Eq. 3. For all the training, the batch size is equal to 32.Two different metrics are used for error estimation. The first metric is Coefficient of Determination () which is defined in Eq. 2. is commonly between zero and one.
(2) 
The second metric for error calculation is MSE, which is also considered as the loss function for training of the networks (Eq. 3).
(3) 
In Eq. 2 and Eq. 3, is the real data, represents the predicted data, and is the mean of the real data. In addition, is the number of samples, which in this study is the number of nodes (grids) in the computational domain.
iii.4 Hyperparameter Analysis
The number of layers and the number of neurons in the hidden layers of both autoencoder and LSTM networks besides their activation functions are the hyperparameters of the presented method. An analysis of the hyperparameters is conducted, and the results for the most important ones are presented in Fig. 8
for the cylinder test 2. Here, the performance of the autoencoder and LSTM networks in reconstruction and prediction of the testing data is investigated using three different activation functions, i.e., tanh, ReLU, and softplus, three different number of hidden layers for the autoencoder network (1, 2, 3), and three different number of LSTM cells (10, 100, 600) in the LSTM network. 15% of the train data set is considered as the validation set. Results are presented as the validation loss during the training process and
value of the prediction or reconstruction of the testing data. Note that error for the autoencoder network is the error in reconstruction of the flowfield from its dominant features, and the error for the LSTM is the prediction error.a shows the validation loss versus number of epochs for the autoencoder network and
Fig. 8b depicts the obtained for the test data set. It can be seen that the ReLU activation function performs better for the autoencoder. Here, results are reported for the autoencoder network with three hidden layers and . The first and third hidden layers contain 500 neurons. Note that we use linear activation functions for the Input, output, and the bottleneck layers and nonlinear activation functions for other layers. Figure 8c and Fig. 8d present the same results for the LSTM network indicating the better performance of the tanh activation function for the LSTM. Moreover, Fig. 8e and Fig. 8f shows the results for the LSTM networks with various number of LSTM cells. It can be seen that the increase in the number of LSTM cells leads to better predictions. Also, the effect of the number of hidden layers of the autoencoder in the accuracy of the autoencoderLSTM predictions is investigated, and the results are reported in Fig. 8g. It can be observed that the autoencoder with three hidden layers obtains the best results. It also should be noted that to avoid overfitting, the training process is stopped where more training epochs do not lead to farther reduction of the validation loss. Based on these results, the autoencoder network with three hidden layers and ReLU activation function and the LSTM network with one hidden layer containing 600 LSTM cells with tanh activation function are chosen as the network architectures for the autoencoderLSTM method.iii.5 Structure of the Inputs and Outputs
In this section, more detailed information on the shape of the inputs and outputs for each test case is presented. For the cylinder test cases, the snapshots are constructed from the velocity domain ( and ). This leads to a snapshot of 81401 nodes. In this regard, the input and output layers of the autoencoder have 81401 nodes. In the oscillating airfoil test case, snapshots are extracted from the velocity domain ( and ). Each snapshot consists of 99473 nodes, so the input and the output layers of the autoencoder network for this test case have 99473 nodes. Other properties of the network remain unchanged. The input data is normalized to cover both negative and positive values through the equation below:
(4) 
where represents each measured point, is the measured value, and
is the time average and standard deviation of
. represents the normalized value.The inputs of the LSTM network are sequences of the outputs of the bottleneck layer of the autoencoder network () with the size of . At the output, it is of interest to have the predicted velocity field. Therefore the output layer has the same number of nodes as the velocity field dimension, which is 81401 for the cylinder test cases and 99473 for the oscillating airfoil test case. The input of the LSTM network is a threedimensional matrix indicating the size of the input arrays, , the number of time steps in a sequence, , and the number of sequences.
iii.6 DMD with Autoencoder for Dimensionality Reduction
Dynamic mode decomposition is one of the wellknown modelfree reduced order modeling techniques that is based on the measurements of the system rather than the governing equations. In fluid dynamics, it is also a method to decompose complex flows into dominant spatiotemporal coherent structures using the power of the SVD. The first step for utilizing the DMD analysis is to construct the snapshots’ matrix and the lagged matrix :
(5) 
(6) 
The locally linear approximation can be written in terms of these data matrices as below:
(7) 
Therefore, the bestfit operator is given by:
(8) 
where is the MoorePenrose pseudoinverse. The highdimensional matrix is not computed directly; instead, it is first projected onto a lowrank subspace using SVD. The SVD of X can be calculated as:
(9) 
Then, the matrix A is obtained from:
(10) 
Therefore, the projection of the matrix onto the lowranked subspace can be obtained as below, where is the rank of truncation:
(11) 
Eigendecomposition of
obtain a set of eigenvectors
and eigenvalues
, where:(12) 
For each pair of and , a DMD eigenvalue, which is itself, and a DMD mode can be defined as:
(13) 
Finally, the approximate solution of at the future times can be calculated from:
(14) 
where, and is the initial amplitude of each mode. is the initial snapshot, and is the matrix of DMD eigenvectors.
In this work, we also investigate the possibility of performing DMD analysis with the autoencoder network as a tool for dimension reduction rather than SVD. The velocity field around the test cases is reduced to the latent space of . These reduced order vectors can be interpreted as the input snapshots, allowing to construct the input matrices required for the DMD as:
(15) 
Then, a fullranked DMD analysis can be performed on the matrices and to compute the mapping matrix of , which can propagate the data into the future time steps. After performing DMD on the space, the predicted values are passed from the decoder part, , of the autoencoder network to obtain the predicted velocity field for future time steps. Figure 9 illustrates the schematic of this method, which is referred, hereafter, as autoencoderDMD. In this study, DMD analysis is performed with the use of the PyDMD Demo et al. (2018) library in Python.
Iv Results and Discussion
iv.1 Dimension Reduction with Autoencoder
The autoencoder network has been used as a nonlinear dimension reduction technique to project the high dimensional data onto the low dimensional subspace. The velocity field is processed through the lowerdimensional representation at the bottleneck layer and then mapped back to the ambient dimension. Data loss is an inevitable consequence of dimension reduction, and the amount of compression which leads to a reasonable approximation error in the reconstructed data is significant. Therefore, the number of neurons at the bottleneck layer of the autoencoder network,
, is an important hyperparameter and should be chosen based on thorough experiments. Here, autoencoder networks with the size of the bottleneck layer of 3, 5, 25, and 50 are trained and tested with the training and testing data sets of the three test cases. of the testing data is reported in Table 1. It can be seen that even with , the data has been reconstructed with acceptable accuracy. As it is expected, an increase of the dimension at the bottleneck layer leads to an increase in the accuracy of the reconstruction and a decrease of the approximation error.Cylinder test 1  Cylinder test 2  Pitching airfoil  

3  0.999  0.882  0.927 
5  0.999  0.899  0.930 
25  0.999  0.928  0.932 
50  0.999  0.936  0.936 
At the latent space, the autoencoder network extracts the main features of the flowfield, which are adequate for the reconstruction of the velocity field at the output layer. To visualize the main features of the flow for the aforementioned test cases, an input array of is given to the bottleneck layer, and the results are taken from the output layer. For visualizing the features extracted by the first neuron, is considered equal to one and others, , are equal to zero. For the second neuron, is equal to one and others are zero, and so on.
Figures 14, 12 and 10 illustrate the first three features extracted by the autoencoder network (autoencoder modes) from the velocity field for the cylinder test 1, cylinder test 2, and the pitching airfoil, respectively. Figures 15, 13 and 11 depict the DMD modes for the mentioned test cases. It can be seen that the autoencoder modes are different from the modes of decompositionbased model reductions, such as DMD modes. The main reason is the nonlinearity added to the dimension reduction technique using nonlinear activation functions. However, similar to the DMD modes, the autoencoder modes represent the most dominant features of the flowfield. For better representation, DMD modes corresponding to the steady mean flow and the complex conjugate modes are not illustrated. We observed that the extracted features from the autoencoder may contain noise, and we performed a twodimensional Gaussian filter to provide a better representation of the dominant modes.
iv.2 AutoencoderLSTM Method for Reduced Order Modeling of the Unsteady Flows
Here, the performance of the proposed ROM framework, autoencoderLSTM, in the prediction of the future estate of the test cases is compared with the DMD and autoencoderDMD methods. Results are reported for the train and test data sets, where the train data set is reconstructed and the test data set is predicted by the models. We report the performance of the models for the training data sets to also show their skills in the reconstruction of the flowfield from its dominant modes. However, the main focus is on the comparison of the prediction skills of the models. For all the models, the size of the reduced subspace is equal to 50, which means that for the autoencoder network, and for the DMD.
Test cases  AutoencoderLSTM  DMD  

MSE  MSE  
Cylinder 1  Train  0.9999  0.9982  
Test  0.9988  0.9949  
Cylinder 2  Train  0.9999  0.9937  
Test  0.9405  0.0826  
Oscillating airfoil  Train  0.9431  0.9198  
Test  0.9421  0.8981 
and MSE obtained from the models in the prediction of the training and testing data sets are reported in Table 2. It can be seen that the autoencoderLSTM method obtains the of at least 0.9405 in the prediction of the test data set of the cases, which indicates the excellent performance of this method in prediction of the future state of the flow, only from past measurements. For the cylinder test 1, the DMD performance is comparable with the autoencoderLSTM. However, for the cylinder test 2 and oscillating airfoil, which exhibit, respectively, multifrequency and extreme events phenomena, the performance of the autoencoderLSTM method outperforms the DMD method.
Figure 16 shows the velocity magnitude over the cylinder test 1 at the last prediction step for the various models against the real data. Moreover, to depict the performance of the models in the prediction of the velocity evolution through the time, the data for three different points in the wake of the cylinder at and are presented in Fig. 17. The grey area shows the training data that is reconstructed by the models, and the white area shows the predictions of the testing data. It can be seen that for this test case, which is a periodic dynamical system, all of the datadriven models provide accurate results. The best predictions of the test data set are obtained from the autoencoderLSTM method leading to the of 0.9988 against 0.9949 of the DMD.
For the cylinder test 2, the autoencoderLSTM method performs much better than the DMD method in prediction of the velocity variations through the time acquiring of 0.9405 against 0.0826 of the DMD for the testing data (see Table 2). Figure 18 represents the time evolution of the velocity magnitude at three different points in the wake of the cylinder test 2. Again, the grey area shows the data which is inside the training data set, and the white area shows the data which is outside of the training data set. Here, It can be seen that the DMD method is not able to predict the time variations of the velocity for the testing data and leads in variations with higher frequencies while the autoencoderLSTM method performs very well in the prediction of the flow evolution. It also can be observed that the performance of the autoencoderDMD is almost the same as the DMD method.
To illustrate the performance of the models in the prediction of the flowfield, contours of velocity magnitude are shown in Fig. 19 at three time instances corresponding to the three snapshots at the testing data. It can be seen that the autoencoderLSTM method, can acquire trustworthy results in the prediction of the time evolution of a complex fluid system while the wellknown method of DMD is not able to predict the future time instances properly. The inaccuracy of the DMD method is not just related to the SVD rank truncation, where a full ranked DMD analysis of the cylinder test 2 leads to of 0.1776 and MSE of for the testing data; however, autoencoderLSTM provides of 0.9405 and MSE of . The accuracy of the DMD results may be enhanced by the increase of the number of snapshots taken from the flowfield, but here, it can be concluded that the autoencoderLSTM method provides more accurate results than the DMD method for an identical data set.
In the oscillating airfoil test case, dynamic stall occurs when the airfoil’s angle of attack surpasses a specific angle. The dynamic stall is a complex nonlinear phenomenon that follows with strong variations of the flow parameters around the airfoil due to the consecutive vortex shedding. By the increase of the angle of attack, the boundary layer separates from the airfoil surface leading to the dynamic stall phenomena, and by the decrease of the angle of attack through the airfoil oscillations, the boundary layer reattaches to the airfoil surface. Here, it is of value to asses the ability of the datadriven ROMs in the prediction of the dynamic stall. Figure 20 shows the variation of the velocity magnitude versus time for the oscillating airfoil at three different points in the wake. Strong variations in velocity magnitude indicate the dynamic stall and consecutive vortex shedding. It can bee seen that the autoencoderLSTM predicts the velocity variations accurately, which shows the excellent performance of this method in forecasting of an extreme event. The DMD and the autoencoderDMD methods, however, are not able to perfectly capture the dynamic stall phenomena, and it can be seen that even the reconstruction of the train data set is not accurate.
To provide a better insight into the physics of the dynamic stall and the performance of various models in prediction of the flowfield, Fig. 21 presents the contours of velocity magnitude over the airfoil at three time instances after occurrence of the dynamic stall. Here, the excellent performance of the autoencoderLSTM against the DMD and autoencoderDMD methods is evident. It can be seen that the vortex shedding behind the airfoil is very well predicted by the autoencoderLSTM while the predictions of the other methods are inaccurate.
V Conclusion
In this paper, a novel datadriven reduced order method based on the power of deep neural networks is presented with the aim of future estate estimation of the complex unsteady fluid flows. The proposed method is based on the power of the autoencoder neural network in dimensionality reduction and feature extraction and the power of the LSTM network in the prediction of the sequential data. Training and testing data are obtained from CFD simulations using finite volume method. Three test cases are investigated; a cylinder at the constant Reynolds number of 3900, a cylinder for which the Reynolds number is decreased with the time from 3355 to 676, and an oscillating airfoil. In the first step, the autoencoder network has been used as a nonlinear dimension reduction technique to project the high dimensional data onto the low dimensional subspace. In this way, the essential features of the flow feasible for accurate reconstruction of the velocity field are extracted at the bottleneck layer of the autoencoder. Then, sequences of the extracted features implemented as the input of an LSTM network with the aim of future state estimation. The output of the LSTM network is the velocity field at the next time step. Results are compared with the results of the wellknown DMD method. The performance of each method is assessed with the use of the coefficient of determination and MSE. Moreover, for the DMD method, the use of the autoencoder network for dimensionality reduction instead of SVD rank truncation is assessed. Results indicate the excellent potential of deep neural networks for datadriven reduced order modeling of complex unsteady flows. For all the cases, the autoencoderLSTM network obtains the best results in the prediction of the velocity field in future time instances. Results show that the DMD and autoencoderLSTM method can predict the flow evolution of the first test case accurately. However, for the second and third test cases, the autoencoderLSTM method outperforms the DMD method in the prediction of the flow dynamics.
References
 TensorFlow: a system for largescale machine learning. pp. 265–283. Cited by: §III.3.
 Cell identity codes: understanding cell identity from gene expression profiles using deep neural networks. Sci. Rep. 9 (1), pp. 2342. External Links: ISSN 20452322 Cited by: §I.
 A long shortterm memory embedding for hybrid uplifted reduced order models. External Links: 1912.06756 Cited by: §I.
 On linear and nonlinear aspects of dynamic mode decomposition. Int J Numer Methods Fluids 82 (6), pp. 348–371. Cited by: §I.
 Ergodic theory, dynamic mode decomposition, and computation of spectral properties of the koopman operator. SIAM J. Appl. Dyn. Syst. 16 (4), pp. 2096–2126. Cited by: §I.
 Neural networks and principal component analysis: learning from examples without local minima. Neural Netw 2 (1), pp. 53–58. Cited by: §I.
 The method of dynamic mode decomposition in shallow water and a swirling flow problem. Int J Numer Methods Fluids 83 (1), pp. 73–89. External Links: ISSN 02712091 Cited by: §I.
 Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52 (1), pp. 477–508. External Links: ISSN 00664189 Cited by: §I.

PyDMD: Python Dynamic Mode Decomposition.
The Journal of Open Source Software
3 (22), pp. 530. Cited by: §III.6.  Turbulence modeling in the age of data. Annu. Rev. Fluid Mech. 51 (1), pp. 357–377. Cited by: §I.
 Recurrent neural networks and Koopmanbased frameworks for temporal predictions in turbulence. External Links: 2005.02762 Cited by: §I.
 Dynamic stall simulation of a pitching airfoil under unsteady freestream velocity. J Fluids Struct 42, pp. 228–244. External Links: ISSN 08899746 Cited by: Figure 4, §II.
 Deep learning. MIT Press. Cited by: §III.2.
 Convolutionalnetwork models to predict wallbounded turbulence from wall quantities. External Links: 2006.12483 Cited by: §I.
 Prediction of wallbounded turbulence from wall quantities using convolutional neural networks. J Phys Conf Ser 1522, pp. 012022. Cited by: §I.
 Long shortterm memory. Neural Comput 9 (8), pp. 1735–1780. Cited by: §III.2.
 New method for dynamic mode decomposition of flows over moving structures based on machine learning (hybrid dynamic mode decomposition). Phys. Fluids 31 (12), pp. 127102. Cited by: §II.
 A koopmanbased framework for forecasting the spatiotemporal evolution of chaotic dynamics with nonlinearities modeled as exogenous forcings. arXiv preprint arXiv:1909.00076. Cited by: §I.
 Adam: a method for stochastic optimization, in proceedings of the 3rd International Conference for Learning Representations (ICLR’15). San Diego. Cited by: §III.3.
 Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences of the United States of America 17 (5), pp. 315. Cited by: §I.
 ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1097–1105. Cited by: §I.
 Deep learning in fluid dynamics. J. Fluid Mech. 814, pp. 1–4. External Links: ISSN 00221120 Cited by: §I.
 Deep learning. Nature 521, pp. 436. Cited by: §I.
 Investigation of flow over an oscillating airfoil. J. Fluid Mech. 512, pp. 313–341. Cited by: Figure 4, §II.
 Extended dynamic mode decomposition with dictionary learning: a datadriven adaptive spectral decomposition of the koopman operator. Chaos 27 (10), pp. 103111. Cited by: §I.
 Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J. Fluid Mech. 807, pp. 155–166. Cited by: §I.
 A novel in situ compression method for CFD data based on generative adversarial network. Journal of Visualization 22 (1), pp. 95–108. Cited by: §I.
 Stochastic tools in turbulence. Elsevier Science. External Links: ISBN 9780123957726 Cited by: §I.
 The structure of inhomogeneous flow. Atmospheric Turbulence and Radio Wave Propogation, pp. 166–78. Cited by: §I.
 Deep learning for universal linear embeddings of nonlinear dynamics. Nat. Commun. 9 (1), pp. 4950. External Links: ISSN 20411723 Cited by: §I.
 A lowdimensional model for turbulent shear flows. New J. Phys. 6, pp. 56. Cited by: §I.
 A deep learning based approach to reduced order modeling for turbulent flow control using LSTM neural networks. arXiv preprint arXiv:1804.09269. Cited by: §I.
 Physically constrained data‐driven correction for reduced‐order modeling of fluid flows. Int J Numer Methods Fluids 89 (3), pp. 103–122. External Links: ISSN 02712091 Cited by: §I.
 Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.), pp. 9258–9268. Cited by: §I.
 Nonlinear mode decomposition with convolutional neural networks for fluid dynamics. J. Fluid Mech. 882, pp. A13. External Links: ISSN 00221120 Cited by: §I.
 A deep learning enabler for nonintrusive reduced order modeling of fluid flows. Phys. Fluids 31 (8), pp. 085101. Cited by: §I, §I.
 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech. 865, pp. 281–302. Cited by: §I.
 Spectral analysis of nonlinear flows. J. Fluid Mech. 641, pp. 115–127. External Links: ISSN 00221120 Cited by: §I.
 Deep convolutional neural networks for LVCSR. pp. 8614–8618. External Links: ISBN 9781479903566, ISSN 15206149 Cited by: §I.
 An artificial neural network framework for reduced order modeling of transient flows. Commun Nonlinear Sci Numer Simul 77, pp. 271 – 287. External Links: ISSN 10075704 Cited by: §I.
 Neural network closures for nonlinear model order reduction. Adv Comput Math 44 (6), pp. 1717–1750. External Links: ISSN 15729044 Cited by: §I.
 Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656 (November), pp. 5–28. External Links: ISSN 14697645 Cited by: §I.
 Dynamic mode decomposition of numerical and experimental data. 61st Annual Meeting of the APS Division of Fluid Dynamics. American Physical Society (November). Cited by: §I.
 Predictions of turbulent shear flows using deep neural networks. Phys. Rev. Fluids 4, pp. 054603. Cited by: §I.
 Modal decompositionbased global stability analysis for reduced order modeling of 2D and 3D wake flows. Int J Numer Methods Fluids 81 (3), pp. 178–191. External Links: ISSN 02712091 Cited by: §I.
 POD‐identification reduced order model of linear transport equations for control purposes. Int J Numer Methods Fluids 90 (8), pp. 375–388. External Links: ISSN 02712091 Cited by: §I.
 Improving the efficiency of a Savonius wind turbine by designing a set of deflector plates with a metamodelbased optimization approach. Energy 186, pp. 115814. External Links: ISSN 03605442 Cited by: §I.
 Modal analysis of fluid flows: An overview. AIAA J 55 (12), pp. 4013–4041. External Links: 1702.01453, ISSN 00011452 Cited by: §I.
 Learning Koopman invariant subspaces for dynamic mode decomposition. In NIPS 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), pp. 1130–1140. Cited by: §I.
 [50] On dynamic mode decomposition: Theory and applications. J. Comput. Dyn. 1 (2), pp. 391–421. Cited by: §I.
 Deep learningbased model reduction for distributed parameter systems. IEEE Trans. Syst. Man Cybern. Syst. 46 (12), pp. 1664–1674. External Links: ISSN 21682216 Cited by: §I.
 Model identification of reduced order fluid dynamics systems using deep learning. Int J Numer Methods Fluids 86 (4), pp. 255–268. External Links: ISSN 02712091 Cited by: §I.
 A datadriven approximation of the koopman operator: extending dynamic mode decomposition. J Nonlinear Sci 25 (6), pp. 1307–1346. External Links: ISSN 14321467 Cited by: §I.
 Comparative Assessment of LES and URANS for Flow Over a Cylinder at a Reynolds Number of 3900. Proceedings of the 16th Australasian Fluid Mechanics Conference, 16AFMC. Cited by: Figure 1, §II.
Comments
There are no comments yet.