Partial differential equations (PDEs) are widely adopted in engineering fields to explain a variety of phenomena such as heat, diffusion, electrodynamics, fluid dynamics, elasticity, and quantum mechanics. With the rapid development in the sensing and storage capabilities provide engineers to reach more knowledge about these phenomena. The collected massive data from multidimensional systems have the potential to provide better understanding of system dynamics and lead to a discovery of more complex systems.
Exploiting data to discover physical laws has been recently investigated through several studies. Schmidt and Lipson (2009); Bongard and Lipson (2007) applied symbolic regression and Rudy et al. (2017); Schaeffer (2017) proposed sparse regression techniques to explain the nonlinear dynamical systems. Raissi and Karniadakis (2018); Raissi et al. (2017) introduced physics informed neural networks using Gaussian processes. Chen et al. (2018) demonstrated continuous-depth residual networks and continuous-time latent variable models to train ordinary neural networks. Farimani et al. (2017) proposed conditional generative adversarial networks and Long et al. (2017) proposed PDE-Net originated from Wavelet theory.
This study proposes a finite-difference inspired convolutional neural network framework to learn the hidden partial differential equations from the given data and iteratively estimate the future dynamical behavior with only a few parameters. Additionally, we introduce auxiliary time steps to achieve higher accuracy in the solutions.
While first-order methods have been extensively used in training deep neural networks, they struggle to promise the training efficiency. By only considering first-order information, these methods are sensitive to the settings of hyper-parameters, with difficulty in escaping saddle points, and so on. Hessian-free (second-order) Martens (2010)
methods use curvature information, make more progress every iteration, minimize the amount of works of tuning hyper-parameters, and only require Hessian-vector product. In this paper, ADAMKingma and Ba (2014) and TRCG methods Nocedal and Wright (2006) Steihaug (1983)
are used to train the proposed network. The empirical results demonstrate that this particular second-order method is more favorable than ADAM to provide high accuracy results to our engineering application of deep learning.
Let us consider a partial differential equation of the general form
where is the non-linear function of , its partial derivatives in time or space where it is denoted by the subscripts. The objective of the study is to implicitly learn the from the given time-series measurements at specific time instances and predict the behavior of the equation for long time sequences.
For easier interpretation of the approach, the proposed algorithm is explained through the motivation problem. Parabolic evolution equations describe processes that are evolving in time. The heat equation is one of the frequently used examples in physics and mathematics to describe how heat evolves over time in an object Incropera et al. (2007). Let denotes the temperature at point at time . The heat equation has the following form for the 1-D bar of length :
where is a constant and called the thermal conductivity of the material. Thermal equation has some boundary conditions. If boundaries are perfectly insulated, the boundary conditions are reduced to,
The PDE of the heat equation can be solved by using Euler method where x and t are discretized for and to find directional derivatives.
where . When the individual time steps are too from each other, Euler method fails to provide a good solution. The stability criteria is satisfied only when Olsen-Kettle (2011). Additionally, for each prediction step, boundary conditions and values are assumed to be known which is not necessarily true for the real applications. In order to address these challenges, data-driven approach is proposed.
The proposed approach is inspired by the finite difference approximation. Each directional derivative in
direction is defined as trainable finite difference filters by size of three (i.e., one parameter for the left neighbor, one for the point itself and one for the right neighbor). The trainable parameters only include weights without any nonlinear activation function and biases. When there is a higher degree of partial difference, multiple sets of learnable weights are considered during training. At the boundary conditions, the filter size of two is adopted since there is only one neighbor. The main benefit of using such a filter is to reduce the number of parameters of the network and to use more natural and interpretable building blocks for the engineering applications.
In order to increase the accuracy and stability, "artificial" time-steps are introduced to the network (Figure 1), where the function value is computed from the linear combination of the input and the feature maps obtained from the difference approximations. These steps are repeated until the prediction of . Similar idea is also used in residual neural networks He et al. (2016) because of its ease in optimization, however, in our case it is a necessity to obtain solutions for unstable PDEs. The relationship between these iterative updates and Euler discretization is also discussed in the Chen et. al. Chen et al. (2018).
Training might take a considerable amount of time while working with long sequences. The proposed approach addresses this problem by training the architecture with randomized mini-batches. We generate samples from randomly picked time intervals during each iteration where represents a sample from the th time series at time . For comparison purposes, first-order ADAM Kingma and Ba (2014) and second-order TRCG methods Nocedal and Wright (2006) Steihaug (1983) are used to train the proposed network.
TRCG Nocedal and Wright (2006) method uses Steihaug’s Conjugate Gradient (CG) method Steihaug (1983) to approximately solve the trust region subproblem and obtain a searching direction. Compared with the exact Newton’s method, CG requires only the computation of the Hessian-vector products without explicitly storing the Hessian. This feature makes TRCG method a Hessian-free Martens (2010) method and suitable to our deep learning application, where the Hessian matrix can be in an immense size due to its quadratic relationship with the number of parameters. To make TRCG more practical to the proposed network and the datasets, a stochastic mini-batch training is adopted: for every iteration of TRCG, one mini-batch dataset is randomly selected to compute the gradient and for CG to compute the Hessian-vector products and solve the trust region subproblem.
Architecture: The general map of the FD-Net architecture is shown in Figure 2. It shows an example of an artificial time step for a selected time from the sample generated from the PDE. The sample is passed through two sets of trainable finite difference layers and the resultant of each layer is aggregated through a fully connected (FC) layer. Then, the output of the FC layer are mapped into a residual building block to predict the function behavior at time
. The loss function is defined as mean squared error loss between the predicted and true values of the function value. The loss function is penalized more at the boundaries.
4 Numerical Study
A dataset containing samples are generated with varying initial conditions by selecting different
from normal distribution. The the domain of the samples issuch that total the dataset contains values. The dataset is split randomly into train/test sets following an 75/25 ratio. The samples are produced with the parameters , and varying time discretization for stable () and unstable () cases. The boundary conditions and the initial condition of the problem is defined as in (3) and (5), respectively. The optimal solution of the heat conduction problem is adopted from the study Olsen-Kettle (2011) and formulated as following:
5 Results and Discussion
During testing, the function value at time is predicted by using the function value at time . Then, the function value at time is predicted by using the function value at time . These predictions are repeated for the full length of the sequence. The RMSE of the true and predicted sequence is computed for all ’s.
To compare the performance of ADAM and TRCG on training the proposed networks, we conduct experiments with various random seeds and mini-batch sizes on the dataset of the stable case. For Adam, we use two learning rates, and . For each experiment, depending on the mini-batch size, while we allow ADAM to run between to epochs, TRCG is given a small budget, less than epochs.
In spite of the small budget TRCG had, the scale of the testing error in terms of RMSE it achieves at , and ADAM is only able to reduce the error to the scale of . Figure 3 presents an example result from the experiment with the random seed and mini-batch size chosen to be and , and it illustrates the empirical performance of ADAM and TRCG on the proposed network very well. The results demonstrate a relatively slow convergence of ADAM and suggest that, for the proposed network, second-order information is important and the searching directions that TRCG generated seem to capture the information.
The predictions of the testing data is investigated. Figure 4 shows the predictions obtained by the proposed method with TRCG and ADAM, and Euler approaches for the time , , and . As can be seen from the figure, although the function characteristics change drastically in the longer term, the proposed architecture is able to determine the behavior with only a few parameters. The most accurate prediction is achieved when FD-Net with TRCG method.
|FD-Net with TRCG|
|Batch Size||k = 1||k = 10||k = 20||Euler|
Since the prediction at time instance affects the next time prediction, the effect of the error accumulation is tested for the unstable case with different artificial time steps , and . Table 1 shows that the performance of the adopted approach is greater than the Euler approach for the unstable cases and increasing the number of artificial time step increases the accuracy of the method. Although our approach mimics the Euler method when , thus better performance is observed.
Research funding is partially provided by the National Science Foundation through Grant No. CMMI-1351537 by Hazard Mitigation and Structural Engineering program, and by a grant from the Commonwealth of Pennsylvania, Department of Community and Economic Development, through the Pennsylvania Infrastructure Technology Alliance (PITA). Martin Takáč was supported by National Science Foundation grants CCF-1618717, CMMI-1663256 and CCF-1740796.
- Schmidt and Lipson  Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science, 324(5923):81–85, 2009.
- Bongard and Lipson  Josh Bongard and Hod Lipson. Automated reverse engineering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 104(24):9943–9948, 2007.
- Schaeffer  Hayden Schaeffer. Learning partial differential equations via data discovery and sparse optimization. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2197):20160446, 2017.
- Rudy et al.  Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 3(4):e1602614, 2017.
- Raissi et al.  Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations. arXiv preprint arXiv:1711.10566, 2017.
Raissi and Karniadakis 
Maziar Raissi and George Em Karniadakis.
Hidden physics models: Machine learning of nonlinear partial differential equations.Journal of Computational Physics, 357:125–141, 2018.
Chen et al. 
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
Neural ordinary differential equations.In Advances in neural information processing systems, pages 6571–6583. Curran Associates, Inc., 2018.
- Farimani et al.  Amir Barati Farimani, Joseph Gomes, and Vijay S Pande. Deep learning the physics of transport phenomena. arXiv preprint arXiv:1709.02432, 2017.
- Long et al.  Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. arXiv preprint arXiv:1710.09668, 2017.
- Martens  James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, pages 735–742, USA, 2010. Omnipress. ISBN 978-1-60558-907-7. URL http://dl.acm.org/citation.cfm?id=3104322.3104416.
- Kingma and Ba  Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Nocedal and Wright  Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer-Verlag New York, 2 edition, 2006.
- Steihaug  Trond Steihaug. The conjugate gradient method and trust regions in large scale optimization. SIAM Journal on Numerical Analysis, 20(3):626–637, 1983.
- Incropera et al.  Frank P Incropera, Adrienne S Lavine, Theodore L Bergman, and David P DeWitt. Fundamentals of heat and mass transfer. Wiley, 2007.
- Olsen-Kettle  Louise Olsen-Kettle. Numerical solution of partial differential equations. Lecture notes at University of Queensland, Australia, 2011.
- He et al.  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In