1 Background and Motivation
In the field of ElectroMagnetics (EM), Boundary Value Problems (BVPs) are the problems for which the EM field in a given region of space is determined from a knowledge of the field over the boundary of the region (BVP, ). Traditional numerical methods such as Finite Difference Time Domain (FDTD) and Finite Element Methods (FEM) are typically employed to solve BVPs. However, these methods involve discretization of the domain to reduce it to higherorder system of linear algebraic equations and solving for them. As such, these methods are not local i.e. they do not give the value of the solution directly at an arbitrary point, where the field needs to be determined, but its value should be extracted from the complete field solution and hence are not amenable to parallel processing.
Neural Network (NN) based numerical method provides an alternate approach to solving BVPs (Lagaris, ),(Mcfall, )
. The principal advantages of the NN based numerical method are the discrete data points where field is computed, can be unstructured and therefore, the issues of meshing (uniform/nonuniform) are not a factor; the solutions are in a differentiable, closed analytic form which avoids the need to interpolate between data points where solutions are obtained using other methods. Also, recent advances in the field of Machine Learning (ML) and Artificial Intelligence (AI) has jump started the design and implementation of computer architectures that are optimized to implement training and inference tasks more efficiently. Since, this NN based numerical method is inherently parallel and hence can be efficiently implemented on parallel architectures, it stands to gain from advances in AI based computer architectures. In this study, NN based field computation is formulated and presented for BVPs with Dirichlet boundary condition and BVPs with mixed boundary conditions on a uniform rectangular grid. Multiple nonuniform training grid distributions are also explored in this study to showcase that this method is not limited by the domain discretization like traditional methods.
2 Problem Statement
NN based method is developed, implemented and investigated for solving Laplace equation with Dirichlet boundary condition and Poisson’s equation with mixed boundary condition. Numerical investigations are carried out to understand efficacy of this method and to provide qualitative understanding of various parameters involved.
3 Formulation
BVPs, where the EM field
is given by the linear second order partial differential equation (PDE), are considered in this study
Lagaris(1) 
subject to boundary condition (B.Cs). Here and
. To solve this BVP using NN based method, a trial form of the solution is constructed, which is written as sum of two parts: the first part satisfies the boundary conditions and contains no adjustable parameters and the second part which involves a feed forward neural network with adjustable parameters is constructed so as not to contribute to boundary conditions.
(2) 
is a feed forward NN with weights and biases . is chosen such that second part does not contribute to boundary conditions.
In such a case, the task of field computation reduces to learning the NN parameters, and , which is done by first transforming the equation (1) to a discretized version and applying it at discretized domain, .
and then training the NN, where the PDE error or cost corresponding to has to become zero.
Note that the computation of this cost function involves not only the network output, but also the derivatives of the output with respect to any of its inputs.
3.1 Neural Network Architecture
The neural network that will be implemented for this solution is a layer network with an input layer ( nodes), a hidden layer ( nodes) and an output layer ( node) as shown in Fig.1
will be used as activation function for the hidden layer.
where and . is used to denote the column of .
3.2 Network Output Derivative Computation
For this neural network, it can be shown that
where and denotes the th order derivative of the sigmoid.
Therefore, first order derivate of , with respect to any input parameter is given by
which can be interpreted as the output of the feedforward neural network of same schematic, where the activation function for the hidden layer is given by the first order derivative of sigmoid, instead of sigmoid and the replaced by .
Similarly, the second order derivative of with respect to can be interpreted as the output of a feedforward neural network with the same architecture, where the activation function for the hidden layer is given by the second order derivative of sigmoid and the replaced by .
For the cost function, which includes network output as well as derivatives of network output, these interpretations become extremely useful during training, when cost function is being optimized.
4 Implementation
The formulated method is implemented to find solution of two BVPs  Laplace equation with Dirichlet boundary condition and Poisson’s equation with mixed boundary condition. In both examples, the domain is chosen to be a square
. Neural Network has been implemented using Tensor Flow framework and optimized using Stochastic Gradient Descent (SGD) with annealing learning rate and regularization.
4.1 Laplace Equation with Dirichlet boundary condition
Electrostatic potential distribution inside a rectangular region where the potential on the boundary is specified, is given by the Laplace equation with Dirichlet boundary condition in a rectangular region janaswamy . NN based method is used to compute the solution and compared with the analytical solution stanford .
(3) 
The boundary conditions are
The analytical solution is
(4) 
The trial solution constructed for NN based method is
In this case, the cost function is given by
The analytical solution (Fig. 2) and the NN based solution (Fig. 3) computed by minimizing the cost function when and are shown below The absolute value of the delta between the two solutions, is also plotted in Fig. 4.
4.2 Poisson’s equation with mixed boundary condition
Electrostatic potential in the presence of charge distribution inside a rectangular region where the potential is specified on a section of the boundary and the gradient of the potential is specified on the rest is considered here.
(5) 
The mixed boundary conditions are
The analytical solution is
(6) 
The trial solution constructed for NN based method is
In this case, the cost function is given by
(7) 
The analytical solution and the NN based solution computed by minimizing the cost function when and are shown below The absolute value of the delta between the two solutions, , with a maximum value of , plotted in Fig. 5 showcases good agreement between the two solutions.
5 Numerical Investigation on Error Properties
Let and represent the set of training and testing dataset respectively. For the case of uniform discretization of domain with a training grid resolution, , and for the nonuniform discretization as well, the term training grid resolution, , is meant to indicate that The testing grid resolution for all the experiments is kept fixed at , therefore the testing dataset is of the size , where the domain is discretized uniformly.
The performance metrics that will be used in this study are the absolute value of the delta between analytical solution and NN based solution
(8) 
and relative error norm as given by
(9) 
gives us information about the spatial distribution of the divergence between analytical and NN based solution, while the paints an aggregate picture over the entire domain.
There is a subtle, but important point that needs to be understood when solving BVPs using NN, training is driven by the cost function (or PDE error), while performance is gauged by the error in the computed field Mcfall .
During the optimization of NN, the PDE is more closely satisfied at the training points for each successful training epoch. But the reduction of PDE error at training points has no obvious guarantee that the NN based solution is converging to the analytical solution, even at the training points. In addition to that, NN with poor generalization because of over fitting may not satisfy the PDE in the regions of the domain away from the training points, and hence NN based solution diverges from analytical solution at such points. Convergence theorem outlines that the difference between the analytical and NN based solutions will decrease everywhere in the domain, when the PDE error is also reduced everywhere in the domain
Mcfall2 . In practice, however, PDE error cannot be guaranteed to decrease everywhere in the domain during training. However, convergence plots such as one shown in Fig. 6 for the Laplace equation when , should be used to monitor the progression of PDE error and during training to make sure both training and test set error decrease along with PDE error during training. Training grid resolution, Number of hidden nodes, Training dataset distribution and size can be tweaked accordingly, if needed to ensure as well as PDE error decreases during training.5.1 Error Dependence on Training Grid Resolution, K
Neural network was trained for various combinations of the number of hidden nodes, and training grid resolution, for the BVP with Laplace equation with Dirichlet boundary condition. For a fixed , the variation of test set error when training grid resolution is shown in Fig.7. As increases from to , the becomes relatively narrower and decreases from to , but plateaus beyond that resolution. Though it is intuitively expected that the test set error improves with increase in the training set size, as the learning model generalizes better, in this particular case, it could also be inferred in the context of the convergence theorem as well. As the PDE error is minimized at more training points, the convergence between analytical and NN based solution gets better.
5.2 Error Dependence on Number of Hidden Nodes, H
To understand the dependence of the test set error on number of hidden nodes, , NN based solution is computed for , for a fixed . Fig.8 shows how the error improves significantly as is increased from to , but stays about the same beyond that. As the computational requirements grow with increase in the number of hidden nodes and/or training grid resolution, and only marginal gains beyond ; Hence, this choice of number of hidden nodes and training grid resolution is deemed appropriate for the problem at hand.
5.3 Error Dependence on NonUniform Training Grid
As mentioned previously, one of the main advantages of NN based method is the training and testing dataset can be totally unstructured and hence they do not suffer from the issue of meshing like other numerical methods. To showcase this, four different grid structures, shown in the Fig.9 where each dot shows a training point in the training dataset, , are investigated in this study. Fig 8(a) shows “Grid1”, the uniform discretization that will be used as baseline in this experiment; Two structured, but different spatial distributions are considered in “Grid2" shown Fig 8(b), which has training points densely distributed closer to the boundary and coarsely away from it and the spatial distribution showing opposite trend in “Grid3", shown in Fig 8(c). Fig 8(d) shows “Grid4", the training dataset generated by uniform random distribution in .
Fig.10 shows the computed test set error for the four grid structures. “Grid4" has the best error performance, followed by “Grid1", while “Grid2" and "Grid3" show similar performance. This may be due to fact that the training dataset distribution “Grid4" closely resembles the test dataset distribution, compared to the other distributions, as the testing grid is a uniform discretization with a grid resolution of . This demonstrates the NN based method’s merits when dealing with unstructured data and its potential to be applied to BVPs with curved or irregular boundaries.
6 Conclusion and Future work
In this study, a NN based numerical method has been formulated, implemented and validated for two examples with Dirichlet and mixed boundary conditions. Numerical experiments were carried out to assess the dependence of the error performance on training grid resolution and number of hidden nodes. To showcase the merit of this method, numerical results for structured and unstructured training dataset are presented as well.
Future efforts could explore application of this method to BVPs with curved or irregular boundaries. Some of the commonly found BVPs in EM are given by PDEs with complex coefficients, which could be the focus of future work as well.
References
 (1) I. E. Lagaris, A. Likas and D. I. Fotiadis, "Artificial neural networks for solving ordinary and partial differential equations," in IEEE Transactions on Neural Networks, vol. 9, no. 5, pp. 9871000, Sep 1998. doi: 10.1109/72.712178
 (2) M. M. Chiaramonte, M. Kiener. Solving differential equations using neural networks. STANFORD UNIVERSITY, 2013. http://cs229.stanford.edu/proj2013/ChiaramonteKienerSolvingDifferentialEquationsUsingNeuralNetworks.pdf.
 (3) SH Kolluru, Preliminary Investigations of a Stochastic Method to solve Electrostatic and Electrodynamic Problems. Masters Thesis, UNIVERSITY OF MASSACHUSETTS AMHERST, August 2008, http://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1261&context=theses
 (4) R. Janaswamy, "Solution of BVPs in electrodynamics by stochastic methods," 2007 IEEE Applied Electromagnetics Conference (AEMC), Kolkata, 2007, pp. 14. doi: 10.1109/AEMC.2007.4638046 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4638046&isnumber=4637999

(5)
J. Han. Deep learningbased numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations
ftp://ftp.math.ucla.edu/pub/camreport/cam1741.pdf  (6) R. Yentis and M. E. Zaghloul, "VLSI implementation of locally connected neural network for solving partial differential equations," in IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, vol. 43, no. 8, pp. 687690, Aug 1996. doi: 10.1109/81.526685
 (7) K. S. McFall and J. R. Mahan, "Artificial Neural Network Method for Solution of Boundary Value Problems With Exact Satisfaction of Arbitrary Boundary Conditions," in IEEE Transactions on Neural Networks, vol. 20, no. 8, pp. 12211233, Aug. 2009. doi: 10.1109/TNN.2009.2020735
 (8) CF. Harrington, Roger. (1961). Time Harmonic Electromagnetic Field. 10.1109/9780470546710.
 (9) K. McFall, An Artificial Neural Network Method for Solving Boundary Value Problems, Germany, Saarbrcken:VDM Verlag,2006. https://smartech.gatech.edu/bitstream/handle/1853/10506/mcfall_kevin_s_200605_phd.pdf