Inverse problems in partial differential equations are fundamental in science and mathematics with wide applications in medical imaging, signal processing, computer vision, remote sensing, electromagnetism and more. Classical methods such as finite differences, finite volume and finite elements are numerical discretization-based methods where the domain is divided into a uniform grid or polygon mesh. The differential equation is then reduced to a system of algebraic equations. These methods may have some limitations: the solution is numeric and may suffer from high condition number, highly dependent on the discretization and even the second derivative is sensitive to noise.
In the last few years, deep learning and neural network-based algorithms are extensively used in pattern recognition, image processing, computer vision and more. Recently, the deep learning approach had been adopted to the field of PDEs as well by converting the problem into a machine learning one. InSupervised learning, the network maps an input to an output based on example input-output pairs. This strategy is used in inverse problems, where the input to the network is a set of observations/measurements (e.g. x-ray tomography, ultrasound) and the output is the set of parameters of interest (tissue density etc.) [4, 8, 9]. Unsupervised learning on the other hand is a self-learning mechanism where the natural structure presents within a set of data points is inferred.
Algorithms for forward and inverse problems in partial differential equations via unsupervised learning were recently introduced. The indirect approach utilizes a neural network as a component in the solution. Li et al.  for example, proposed the NETT (Network Tikhonov) approach to inverse problems. NETT considers regularized solutions having small value of a regularizer defined by a trained neural network. Khoo and Ying  introduced a novel neural network architecture, SwitchNet, for solving the wave equation based inverse scattering problems via providing maps between the scatterers and the scattered field. Han et al.  developed a deep learning-based approach that can handle general high-dimensional parabolic PDEs. To this end, the PDEs are reformulated using backward stochastic differential equations. The latter is solved by a temporal discretization and the gradient of the unknown solution at each time step is approximated by neural network.
Direct algorithms solve the forward problem PDEs by directly approximating the solution with a deep neural network. The network parameters are determined by the optimization of a cost function such that the optimal solution satisfies the PDE, boundary conditions and initial conditions. Chiaramonte and Kiener  addressed the forward problem by constructing a one layer network which satisfies the PDE within the domain. The boundary conditions were analytically integrated in the cost function. They demonstrated their algorithm on the Laplace and hyperbolic conservation law PDEs. Sirignano and Spiliopoulos  proposed a deep learning forward problem solver for high dimensional PDEs. Their algorithm was demonstrated on the American option free-boundary equation. Raissi et al.  focused on continuous time models and solved the Burgers and Shrödinger equations.
In this work we focus on the forward and inverse PDEs problems via a direct unsupervised method. Our key contributions are three fold: (1) in the forward part we extend the standard -based fidelity term in the cost function by adding -like norm. Moreover, (2) some regularization terms which impose a-priori knowledge on the solution can be easily incorporated. (3) An important feature of our construction is the ability to handle free-form domain in a mesh free manner. We demonstrate our algorithm by a second order elliptic equation, in particular the Electrical Impedance Tomography (EIT) application.
2 Mathematical Formulation
Let be a bounded open and connected subset of , and be any given symmetric positive definite matrix of functions for . Let be any given n-tuple of functions and let be any given function. A second order operator is said to be in divergence form, if acting on some has the form
where we use the Einstein summation convention. Consider the partial differential problem with Dirichlet boundary conditions
The forward problem solves given the coefficients while the inverse problem determines the coefficients set given .
The proposed algorithm approximates the solutions in both problems by neural networks such that the networks are parameterized by , and the input to the network is . Figure 1 depicts a network architecture of in . The network consists of few fully connected layers with tanh activation and linear sum in the last layer.
The network is trained to satisfy the PDE with the boundary conditions by minimizing a cost function. In the forward problem
and in the inverse problem
The first two terms enforce the solution to satisfy the equation. The first term minimizes the error in sense while the second term minimizes the maximal error. This term is important since the term only forces the equation up to a set of zero measure. The
term takes care of possible outliers. The third term imposes boundary conditions and the last term is a regularizer which can be tailored to the application. There are few advantages of this setting. First, the solutions are smooth analytic functions and are thereforeanalytically differentiable. In addition, this framework enables setting of a prior knowledge on the solution by designing the regularizers and . Lastly, the training procedure is mesh free. In the sequel, we use random points in the domain and its boundary in the course of the optimization of (3) and (4). This means that the solution does not depend upon a coordinate mesh and we can also define in principle an arbitrary regular domain .
3 Application to Electrical Impedance Tomography
Let us address a special case of (1),
We assume that , which guarantees existence and uniqueness of a solution .
The elliptical system (5) was addressed by Siltanen et al.  in the context of Electrical Impedance Tomography (EIT) which is a reconstruction method for the inverse conductivity problem. The function stands for the electrical conductivity density, and is the electrical potential. An electrical current
is applied on electrodes on the surface , where is the angle in polar coordinate system along the domain boundary and is the normal unit. The resulting voltage is measured through the electrodes. The conductivity is determined from the knowledge of the Dirichlet-to-Neumann map or voltage-to-current map
using the D-bar method .
We demonstrate our framework by solving the forward and inverse problem of (5) which is a first step towards a full tomography. Following Mueller and Siltanen , we simulate the voltage measurement by the Finite Element Method (FEM) given two variants of a conductivity phantom on the unit disc. We calculate the FEM solution with different triangle mesh densities such that finer meshes do not improve the numerical solution.
With our suggested method, the forward problem determines the electrical potential in the whole domain , while the inverse problem uses the approximated and calculates the conductivity given that . Throughout the paper we use three different electrical currents where , see Figure 2.
4 Forward Problem
In the forward problem the conductivity and boundary conditions are given for random points set , with sets size of and respectively. A neural network having the architecture shown in Figure 1 approximates . Let
The cost function (3) is then rewritten as
The first term is the norm of the differential operator, the second term is a relaxed version of the infinity norm where we take the mean value of the top-K values of . The third term imposes the boundary conditions and the last term serves as a regularizer of the network parameters.
The network was trained with layers having , and neurons. The algorithm was implemented by TensorFlow  using the ADAM optimizer which is a variant of the SGD algorithm. We used batch size= and a decaying learning rate starting at corresponding to . The learning rate was factored by every epochs. The algorithm parameters were set to , , , , and .
The first phantom is shown in Figure 3. The background has conductivity and the circle has conductivity . The original piecewise constant function was slightly smoothed by a Gaussian kernel.
Figure 4 summarizes the forward problem results for currents and . The top row is the FEM solution which is referred to as ground truth. The middle row depicts the outcome of the trained network, and the bottom row shows the relative error ,
Mean square errors and PSNR are indicated in the figures’ caption.
Figure 5 shows the derivative of with respect to . The top row is the finite difference approximation of the FEM result. The middle row is the analytical derivative of our result, and the bottom row shows the relative error.
We further demonstrate the effect of the norm in the cost function. The left and right images of Figure 6 stand for the derivative of phantom without and with the term respectively. Clearly, this additional norm yields better reconstruction both visually (sharper circle edges) and quantitatively.
We repeated the experiment with an additional phantom, see Figure 7. The impedance values associated with the background, ellipses and circle were set so , and . In this case , learning rate= and all other parameters as before.
5 Inverse Problem
In the inverse problem, the electrical potential is known while is unknown. Since we have a network which approximates , we can evaluate it at any point . The objective function (4) then takes the form
As in the forward problem, the first two terms enforce to satisfy the PDE, where is defined in (6). The third term imposes the boundary conditions, and the fourth regularizes the network parameters. The last term is the total variation regularization () which promotes the solution towards a piecewise constant solution. The network architecture and other parameters are as in the forward problem except for and . Conductivity reconstructions are shown in figures 10 and 11 with
Deep networks by their nature use compositions of simple functions such as matrix multiplication and non-linear activations like sigmoid or tanh. This structure (i) enables the approximation of an arbitrary function and (ii) is inherently differentiable. The network architecture dictates the number of degrees of freedom which in turn enables the expressibility of complex functions. In this work we present a unified framework for the solution of forward and backward problems in partial differential equations. The algorithm relies on direct approximation of the unknown function by a neural network which yields ananalytical smooth solution in a predefined domain. The network is trained to satisfy the PDE and boundary conditions in an unsupervised fashion by the minimization of a cost function. The optimization procedure depends on random points set within the domain and its boundary. The problem is therefore mesh free with free-form domain. We introduce a cost function which is composed of both and fidelity terms and additional regularizers. The algorithm is demonstrated by an elliptic system in applied to Electrical Impedance Tomography for both forward and inverse problems. Promising results were achieved for complex and non monotonic functions. This framework is general and opens up a wide range of applications and extensions for further research.
-  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
-  M. M. Chiaramonte and M. Kiener. Solving differential equations using neural networks, 2017. http://cs229.stanford.edu/proj2013.
-  L.C. Evans. Patial Differential Equations. the American Mathematical Society, 2010.
-  M. Feigin, D. Freedman, and B. W. Anthony. A deep learning framework for single-sided sound speed inversion in medical ultrasound. arXiv 1810.00322v3, 2018.
-  J. Han, A. Jentzen, and E. Weinan. Solving high-dimensional partial differential equations using deep learning. arXiv 1707.02568, 2018.
-  Y. Khoo and L. Ying. Switchnet: A neural network model for forward and inverse scattering problems. arXiv 1810.09676v1, 2018.
-  H. Li, J. Schwab, S. Antholzer, and M. Haltmeier. NETT: Solving inverse problems with deep neural networks. arXiv 1803.00092, 2018.
-  A. Lucas, M. Iliadis, R. Molina, and A. K. Katsaggelos. Using deep neural networks for inverse problems in imaging. IEEE Signal Processing Magazine, 2018.
-  M. T. McCann, K. H. Jin, and M. Unser. A review of convolutional neural networks for inverse problems in imaging. arXiv 1710.04011, 2017.
-  J. Mueller and S. Siltanen. Linear and Nonlinear Inverse Problems with Practical Applications. SIAM, 2012.
-  M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv 1711.10561v1, 2017.
-  S. Siltanen, J. Mueller, and D. Isaacson. An implementation of the reconstruction algorithm of a nachman for the 2d inverse conductivity problem. Inverse Problems, 16(3):681–699, 2000.
-  J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. arXiv 1708.07469v3, 2017.