1 Introduction
A wide class of important engineering and physical problems are governed by partial differential equations (PDEs) describing the conservation laws. Extensive research efforts have gone into formulating and solving these governing PDEs. Despite significant progress, major challenges remain related to the computational costs of solving complex PDEs for real life problems like turbulent flows, laminarturbulence transition and climate modeling. To avoid these prohibitively high computational costs, developing accurate and efficient numerical approximations or surrogate models for PDEs has been a key area of research [1, 2, 3, 4]
. Machine learning based models
[5, 6, 7, 8, 9] have the potential to provide a significantly faster alternatives to the traditional methods [10, 11] of surrogate modeling. For accuracy and physical realizability, it is desired that these machine learning based models closely mimic the properties of the governing PDEs.One of the key features of these PDEs is frame invariance, which is an intrinsic property of all equations in classical mechanics from Newton’s second law to Navier Stokes equations. It signifies that the behavior of the physical systems does not depend on the origin or orientation of the reference frame of the observer. A model is frameinvariant to a transformation if the transformation of the input data does not alter the output of the function or model. In the context of fluid mechanics, any scalar variable like pressure or velocity magnitude is independent of any translation or rotation of the reference frame. Specifically, for example, a vectortoscalar constitutive mapping should remain unchanged in the frame rotated by matrix , i.e., the same mapping should be valid for . In other words, for any mapping to be frameinvariant, the mapping should hold for any rotation matrix , with
being the input vector in the new, rotated coordinate system. Invariance with respect to other transformations (e.g., translation of origin or change of reference velocity) can be defined and interpreted similarly. For solid mechanics, the magnitude of the deformation tensor is invariant to the orientation of the reference frame or the origin of the frame – it is an objective quantity regardless of the observer. Clearly, any modeling equations or constitutive relations should faithfully reflect such invariance or symmetries. Furthermore, in the numerical solutions of these PDEs, frame invariance also requires permutational invariance, which ensures independence of the results from the order in which the discretized objects are indexed. Examples of such objects include elements, cells, grid points, or particles, depending on the specific numerical method used. In a hypothetical simple example, assume the scalar
at a given location is a function of the vectors , , and at three cells in the neighborhood of . The output must remain identical whether the input is , , or any other ordering of three vectors. In other words, permutation invariance demands that must only depend on the set as a whole and not on the ordering of its elements. Embedding these invariance properties in machine learning based models can significantly improve the generalizability of these models [12, 13, 14, 15, 16].In this work, we are particularly interested in using neural networkbased solution operators, called neural operators [5, 9], to solve PDEs with a guarantee of frame invariance. Mathematically, given a PDE system, the solution operator maps the input function such as the initial condition, boundary condition, or auxiliary field to the output PDE solution. The solution operator plays an essential role in many applications like surrogate modeling, constitutive modeling, and Bayesian inverse problem. Note that the solution operator is defined with respect to a family of PDEs with different input functions, in contrast to classical methods like finite difference methods and modern machine learning based methods like physicsinformed neural network [17, 18] that aim to determine the solution for a single instance of the PDE. Those methods are computationally expensive when multiple evaluations are required. In contrast, neural operators aim to approximate the solution operator through an explicit mapping starting from the input function so that it operates significantly faster. It is also designed to be meshindependent. Of course, learning a solution operator is significantly more challenging than determining the solution to a PDE in a single instance. But once the network parameters are properly optimized, the neural operator can predict the solution of a new instance of the PDE fast. In this work we examine two such neural operators, graph kernel networks and vector cloud neural networks.
Recently, graph based discretization of the physical domain has found prominence for the development of neural operators and more broadly, scientific simulations [19], using graph neural networks [20]. Graph kernel networks [21, 22]
is a specialized class of graph neural networks for implementing nerual operators. GKN operates on graphstructured data, which is consistent with grids or data structures used in computational methods. A kernel network is used to incorporate nonlocal dependencies from other graph nodes. GKN can exhibit translational invariance if relative spatial information is considered for neighbouring nodes in the graph. In that case global attribute of the graph is invariant to a translation of the graphstructured data in the spatial domain. Similarly, the global arithmetic operation to determine the global attribute of the graphstructured data embeds permutational invariance, which guarantees independence from the order in which graph nodes are indexed. Rotational invariance can be achieved by alternative methods for graph networks. A simpler method, borrowed from computer vision, is to augment the training data with differently rotated coordinate systems. However, such a method is inefficient as it increases the data size and training time (by approximately a factor of over ten as observed in
[12]). Moreover, the data augmentation does not guarantee frame invariance strictly – rather, the trained network only has approximate rotational invariance [23]. Another method to achieve rotational invariance is to use rotational invariant input features. This work will discuss such a method that can be effectively implemented with GKN to achieve frame invariance in a dataefficient manner.The computational complexity of GKN is dependent on the sparsity of the underlying graph structure. When the graph is complete, i.e., every pair of nodes is connected with an edge, the computation cost scales quadratically with the number of nodes or data points in the graph. For some scientific problems, the graph size for capturing nonlocal dependencies can be of the order of hundreds or thousands nodes, and the graph is densely connected, like in the case of high Reynolds number flows. In such cases, GKN does not scale well and can become prohibitively expensive. This computational disadvantage of GKN was also discussed in [21] where a graph kernel network was used to emulate the Green’s function for the Poisson equation.
Vector cloud neural network (VCNN) provides an alternative strategy to develop a neural operator with embedding frame invariance [8]. The nonlocal information of the PDE is represented by a vector cloud, an arbitrarily arranged group of points akin to graphstructured data, with each point (node) having a feature vector attached to it. Input matrix is organised by stacking feature vectors corresponding to every point in the cloud. This vector cloud is mapped to a global scalar attribute of the cloud, through a frameinvariant neural networkbased model. Spatial coordinates for neighbouring points in the cloud are organised relative to the central point of the cloud, which ensures translational invariance. Pairwise projection of feature vectors is used to obtain rotational invariant features. Furthermore, an embedding network is used to obtain a learned basis and the rotational invariant features are projected on this learned basis as where is the number of points in the cloud. Such projections output the average point features on the learned basis and guarantee permutational invariance in the model. Invariant feature matrix is then nonlinearly mapped to the unknown scalar variable of PDE through a fitting network. For predicting the unknown variable in the entire field, VCNN has shown to be computationally more efficient compared to traditional meshbased PDE solvers such as those in finite volume methods as the number of mesh cells in the computational domain scales [8]. Another comparable network architecture with embedded frame invariance, PointNet [24]
, also operates on point clouds to learn pointwise latent features by mapping through a shared embedding network. A global feature is then extracted by using an aggregate function (like maxpooling) to ensure permutational invariance. Rotational invariance is ensured by the use of scalar input features only or together with vector features defined in a rotational equivariant coordinate system
[25]. Considering grid vertices of a CFD domain as a point cloud, PointNet has been used to define an endtoend mapping from the spatial positions of the grid vertices over irregular geometries to the corresponding velocity and pressure fields of the flow [26].In the present work, we use flows over a symmetric airfoil to investigate the frame invariance property and scalability of the neural operators discussed above. As a proof of concept, we perform this investigation for a neural operator developed for a transport equation of a scalar variable. The behavior of such a scalar variable can also be closely related to scalar quantities in turbulence modeling including turbulent kinetic energy (TKE) and turbulent length scale [27]. In this preliminary work, we consider a hypothetical dimensionless concentration tracer , which is transported by a laminar velocity field . Although the originally proposed GKN does not ensure rotational invariance, we have modified it to use rotational invariant input features and map them to the scalar variable . Such results are then compared against the embedded frame invariance of VCNN. The results are analyzed for accuracy and computational efficiency.
The rest of the manuscript is organised as follows. Methodologies used for this work are presented in Section 2. Generation of data for training and testing of the neural operators is discussed. Both neural operators, GKN and VCNN, are also presented while elaborating on their frame invariance properties. Section 3 presents the analysis of frame invariance properties, predictive performance and computational efficiency of both models. Section 4 concludes the paper.
2 Methodology
2.1 Generation of data for training and testing
Flow over a symmetric NACA0012 airfoil in a twodimensional space is considered for this study. The airfoil contour was obtained from a public domain source, UIUC Airfoil Coordinates Database [28]. Cmesh flow domain is used to appropriately distribute the mesh density along the critical areas of flow around an airfoil. For an airfoil with unit chord length, the width of the flow domain extends 5 times the chord length towards the upstream direction and 10 times the chord length towards the downstream direction of the airfoil leading edge. The domain extends 5 times the chord length in the direction normal to the airfoil chord.
The scalar quantity is a hypothetical, dimensionless concentration tracer, which is transported by the velocity field . A simplification is assumed for this study that the velocity field is not affected by the concentration tracer. To compute the scalar field , the transport PDE is solved with a steady, laminar flow field . Resembling the transport equation of other scalar quantities like turbulent kinetic energy, eddy viscosity, etc., the transport PDE for concentration is given as:
(1) 
where indicates the production term, indicates the dissipation term and represents the diffusion coefficient. Both the production and dissipation terms depend on the scalar quantity . The production is further dependent on the mixing length and strainrate magnitude . Both terms are given as:
with 
where and are production and dissipation coefficients, respectively. Further details for this transport equation and solver have been discussed in detail by [8].
Laminar flow field is simulated first to obtain the velocity field . Using this velocity field, scalar field is then computed by solving transport PDE for concentration tracer. Data has been generated for a set of angles of attack (AOA) and divided into training and testing dataset as:
Testing dataset has been organised to interpolate to testing data at AOA =
and extrapolate to testing data at AOA = flow conditions. Furthermore, in the testing dataset, a transformed reference frame is used, which is rotated by a value equal to the AOA in the corresponding flow. Such distinction is demonstrated in Fig. 1. This transformation of the reference frame is specifically implemented to analyze the frame invariance behavior of the neural operators. Furthermore, to avoid redundancy, the data is extracted from a truncated domain by discarding the region which does not have any concentration tracer distribution at any specified flow conditions.transported by flow over a symmetric airfoil (NACA0012). The training dataset is based on Cartesian coordinate system with origin coinciding the airfoil leading edge and
axis aligned with the airfoil chord. Coordinate system for the testing dataset is transformed/rotated by the given angle of attack (AOA), with origin still coinciding with the airfoil leading edge. Red box indicates the truncated flow domain. Data outside the green box is discarded to avoid redundancy.To generate data for nonlocal mapping from a patch of flow field to concentration at the point of interest, the setting is largely adapted from [29, 8]. The flow features’ vector used at each point is chosen to include relative coordinate features , flow velocity and seven additional scalar quantities, collectively denoted by . Note that our convention in this paper is that all vectors are column vectors, and denotes vertical concatenation. Those scalar quantities in include mesh cell volume (), magnitude of strain rate (), boundary cell indicator (), velocity magnitude (), wall distance (), proximity of cloud center (), proximity in local velocity frame (). Detailed descriptions and mathematical definitions of these scalar quantities can be found in [8].
Flow features are considered at randomly selected fixed number of points within the region of influence around the point of interest where concentration has to be determined. Fig. 2 illustrates the regions of influence and respective sampled data points corresponding to different points of interest in the flow domain. The extent of the region of influence, represented by an ellipse, depends on the velocity, diffusion and dissipation coefficients at the point of interest. The halflengths and of the major and minor axis of the ellipse are determined based on the specified relative error tolerance :
(2) 
where and are the diffusion and dissipation coefficients, for which details can be found in [29]. The major axis of the ellipse aligns with the direction of the velocity . Each inputoutput combination for the neural operators then corresponds to an elliptical patch of flow domain, where input data is the stack of feature vectors arranged as input feature matrix and output is the scalar variable at the point of interest in that elliptical patch. Stencil size () represents the number of randomly sampled fixed number of cloud data points from that elliptical patch, or equivalently the amount of information considered from the region of nonlocal influence. Constant stencil size is considered throughout the flow domain. For the locations where the number of data points in the ellipse is smaller than the specified stencil size, the available data points are repeatedly sampled.
2.2 Graph Kernel Network
Graph kernel network is based on graph neural networks, a class of neural networks adapted to leverage the structure and properties of graphs. A graph is an extremely powerful and general representation of data used to express a collection of entities (nodes) and the connections (edges) between them. Depending on the application, a graph can have attributes/features attached to the nodes and/or edges, and in some cases each graph can have a global attribute. Such specialized neural networks have found several practical applications such as antibiotic discovery [30], physics simulations [31], fake news identification [32], traffic prediction [33] and recommendation systems [34].
In the context of PDEs, graph nodes can be associated with the mesh cells and the cells can be connected through edges. To predict at any particular point, one can construct a graph among the neighbouring nodes in the region of influence and assimilate the nonlocal information through corresponding edges. Scalar quantity is the global attribute for each graph which can then be determined by considering the attributes of all the nodes in the graph.
Li et al. [21] proposed GKN to learn mappings between function spaces represented by graphbased data. The mapping is formulated by using a neural networkbased kernel function. Fig. 3 shows a schematic of the GKN architecture in the context of our example. The graph is completely connected, and the connection between two nodes in a graph is attributed by edge features as:
(3) 
which are obtained by concatenating the flow feature vectors of the corresponding nodes. Here and represent two corresponding nodes of the edge. Unlike scalar features , relative coordinates and velocity does not possess rotational invariance, rendering edge features in Eq. (3) dependent on the choice of frame orientation. Therefore, to ensure rotational invariance, we propose here to use rotational invariant edge features by projecting the vectors on to each other and by concatenating the scalars, which are specifically given as:
(4) 
Note that except the difference between Eq. (3) and Eq. (4) in edge features, our modified GKN has the same structure as that in [21], as described below.
Given edge features, kernel integration is computed to approximate mapping from one graphbased representation to another. Each graphbased representation is referenced as a layer () and multilayer GKN enables capturing longrange correlations in graphbased data [21]. Initial graphbased representation in latent space was proposed in [21] to be based on the complete feature vector on each node. However, here we consider only the scalar features for initialization, which possesses rotational invariance. Each GKN layer update, referred to as graph convolution, can then be expressed as:
(5)  
(6) 
where indexes the nodes in the graph representation, is a vector of dimension representing the latent node features of th node in the initial graph representation, is the shorthand notation of , represents the learnable parameters of a fully connected linear layer, represents a matrix of learnable parameters for information passing among nodes, is the total number of layers or depth of GKN and represents the neural network based kernel function. Details of the kernel network architecture have been mentioned in the Appendix B (Table 6). The output of is dimensional and reshaped to a square matrix. Latent features of each node in the graph representation () are updated using kernel network to incorporate the information from all the connected nodes and learnable parameters’ matrix to process the latent features of the node itself in the previous layer. Hence each node in the updated graph representation is connected to every node in the previous graph representation. This graph update is performed for specified number of times enumerated as the depth () of GKN.
For scalar field over an airfoil, GKN is purposed for regiontopoint mapping. In this regard, to obtain a global attribute for the graph representation at layer , a global averaging operation is performed as:
(7) 
where represents learnable parameters and represents the latent features of th node in the last GKN layer (). This global averaging operation also guarantees permutational invariance for GKN. With translational invariance already guaranteed, rotational input features and global averaging operation ensures the frame invariance property of the GKN based model presented here.
2.3 Vector Cloud Neural Network
Zhou et al. [8] recently proposed a frameindependent neural operator, which is able to map to a scalar variable at a point based on the nonlocal information in its neighbourhood. Similar to graph neural networks, such nonlocal information is taken into account as an arbitrarily arranged group of points referred to as vector cloud, with a feature vector attached to each point. Since this nonlocal information can be captured from arbitrarily arranged points, irrespective of the ordering of points in the cloud, VCNN is a powerful tool to deal with unstructured meshes in scientific simulations.
Regiontopoint mapping from input matrix of vector cloud features to the scalar variable is illustrated in Fig. 4 as a schematic figure of VCNN. Input matrix is organised by stacking feature vectors corresponding to every point in the cloud. Each row in is a feature vector on a point. With the usage of relative coordinates and pairwise projections among the feature vectors, a direct mapping of the form can ensure translational and rotational invariance. However to guarantee frame invariance completely, permutational invariance also needs to be embedded. For such purpose, an embedding network is used, as shown in Fig. 4(a). Note that scalar features already possess translational and rotational invariance. These scalar features are mapped through an embedding network to obtain dimensional embedded weights corresponding to each point, and form an embedding matrix corresponding to all the points in the cloud. To introduce permutational invariance, an orderremoving transformation is performed as . Combining it with the pairwise projections among the feature vectors, invariant feature matrix can be obtained as
(8) 
where is taken as a subset of to save computational cost. Equivalently, this can be written as where
(9) 
and similarly defined for . The normalization by the number of points in the cloud allows the training and prediction to use different number of sampled point in the cloud. The obtained invariant feature matrix is then mapped through a fitting network to the scalar variable , as shown in Fig. 4(b).
The combination of embedding network, projection mapping and fitting network thus forms a complete VCNN model that has embedded frame invariance in the model. Detailed architectures of embedding and fitting networks have been mentioned in the Appendix B (Table 5). Besides embedded frame invariance attribute, VCNN has demonstrated reasonable computational efficiency compared to traditional meshbased PDE solvers such as those in finite volume methods for predicting the unknown variable in the entire field, especially as the number of mesh cells in the computational domain scales [8]. With this perspective, the performance and scalability of VCNN to develop a frameinvariant neural operator is compared with that of GKN in the next section.
2.4 Comparative analysis of GKN and VCNN
Both neural operators, GKN and VCNN, presented earlier are compared schematically in Fig. 5. The same form of input matrix is provided for both models. Both models process the data and ensure frame invariance properties in different manners. Different components of each model can be related with respect to their role in ensuring frame invariance and accurately predicting the scalar variable. To extract global nonlocal information, VCNN uses a fullyconnected neural network (Fig. 5(f)) for mapping from invariant feature matrix to the scalar output . In contrast, GKN uses a relatively intricate setting where a neural network basedkernel function incorporates the information from neighbouring points to update the latent graph representation sequentially over several layers, where the mapping between the layers is determined by a nonlinear neural network function operating on the edges . Such an elaborate process of incorporating nonlocal information provides GKN with more flexibility in learning the regiontopoint mapping. However, the edgebased mapping is also the source of scaling of GKN in terms of training time and memory footprint, which will be shown later. In contrast, the high dimension of the edgebased features (or equivalently pairwise project of node vectors) is reduced in the projection operation in the embedding network of the VCNN. In fact, the pairwise project matrix in Eq. (8) is never formed explicitly in the implementation. Another way to see the comparison between two models is by dropping the weights of both models and assuming onelayer graph convolution in GKN with edge features . In this case, if we denote the scalar feature matrix
, the feature extraction part of both models can be summarized as
where is the kernel network (nonlinear function) in GKN operating on edges, while is the nonlinear functions in VCNN operating on the nodes. It is evident that the nonlinearity is applied to edges in GKN and to nodes on VCNN. The comparison is summarized in Table 1.
Element  GKN  VCNN 


Ensure rotational invariance  Edge construction: in Eq. (4)  Pairwise inner product: 
Ensure permutational invariance  Plain node averaging:  Weighted node averaging 
Embody internode interactions  Kernel network (nonlinear) with edge input  Pairwise (linear) projection 
Nonlinearity in network  Graphical convolution with edgedependent (nonlinear) weights  Nodewise embedding network (Fig.4a), fitting network 
Degenerated to one layer 
Not a builtin component of GKN, but can be implemented by constructing rotational invariant edge features
Equivalently written as
3 Results
In this section, we demonstrate and analyze the frame invariance property of both neural operators discussed earlier. Unless otherwise stated, the default stencil size used to report results is 150, which is close to those used in the original works of GKN and VCNN. The results are presented as contours of the scalar field () as well as in terms of prediction error percentage for the testing datasets at different angles of attack with a transformed frame of reference. The prediction error percentage is defined as the normalized discrepancy between predicted scalar field and the corresponding ground truth :
(10) 
where represents the total number of points at which the scalar variable has to be predicted. In the first part of this section, we analyze the predictive performance and frame invariance property of the originally proposed GKN [21]. We demonstrate the effect of using rotational invariant input features for GKN. In the second part of this section, we compare two neural operators, VCNN that has frame invariance embedded in the neural operator and GKN for which frame invariance is achieved by introducing rotational invariant input features. Training cost and predictive performance are also analyzed for both neural operators.
3.1 Frame invariance for GKN
The originally proposed GKN [21] holds permutational and translational invariance properties, which are inherent properties of graph neural networks. However, such a network does not ensure rotational invariance, which can be critical for scientific problems. Reference systems can be differently aligned for different datasets, as shown in Fig. 1. A robust neural operator should have the ability to accurately predict for the dataset with a reference system different from that for the training dataset. For the specified problem setup of predicting concentration field in a flow over an airfoil, the neural operator should be able to accurately predict independent of the reference system of the dataset. In this regard, the frameinvariant characterization of GKN is analyzed here.
To achieve rotational invariance with GKN, we should use rotational invariant input features. The predictive performance of GKN with (Eq. (4)) and without (Eq. (3)) rotational invariant input features is compared in Table 2, which shows the prediction error percentage for the testing dataset with rotated reference system as compared to the training dataset (Fig. 1). It can be observed that the prediction error percentage for the testing dataset is much worse for GKN without rotational invariant input features. For the testing dataset, the reference system is rotated equally to the AOA for the specific flow. In that regard, as the rotational transformation of the reference system increases (with increasing AOA), the prediction error percentage increases from 8.5% at 5°AOA to 27% at 35°AOA. In comparison, GKN with rotational invariant input features predicts with significantly higher accuracy for the testing dataset at all angles of attack. Moreover, it follows a general trend of neural network models, i.e., a better prediction accuracy for interpolating in test conditions than extrapolating in test conditions.



NN Model  Training Error  AOA = 15°  AOA = 25°  AOA = 5°  AOA = 35°  


GKN, no RI  0.27%  16.4%  23.5%  8.5%  27.0%  
GKN, with RI  0.33%  1.7%  1.4%  2.7%  2.6% 




The contours of the predicted scalar field are shown in Fig. 6, where predictions of GKN with and without rotational invariant input features are compared with a true scalar field. In the contours of GKN without rotational invariant input features, erroneously predicted streaks of concentration tracer spread throughout the domain. Moreover for this case, two distinct streams of concentration tracer from upper and lower airfoil surfaces are not clearly distinguishable in the downstream flow. In comparison, the use of rotational invariant input features results in contours that are significantly accurate and imitate those of the true scalar field. These results show that the frame invariance can be achieved with the GKN by introducing rotational invariant input features and such a setup provides a promising candidate for frameinvariant neural operators.
To further highlight the significance of frame invariance for neural operators, GKN without rotational invariance is tested in the case of AOA=, with input data that has been transformed/rotated with arbitrarily selected angles. Fig. 7 shows the scalar field contours for these results. Reference frame of input data is rotated to different angles (,,, or ) and the corresponding results are compared with a true concentration field. As the reference frame of the input data is transformed with larger rotation angles, the predicted output by GKN without rotational invariance becomes more erroneous. For input data transformed to large rotation angles ( and ), the predicted output (Fig. 7(e) and 7(f)) is significantly offscale compared to true scalar field. In contrast, GKN with rotational invariance predicts accurate concentration field, invariant to the transformations applied to the input data.
These results are further analysed by plotting profiles of scalar quantity along a crosssection, as shown in Fig. 8. The crosssection line has been shown in the inset at the right bottom corner of Fig. 8, where the arrow indicates the direction along which profiles have been drawn. Profiles plotted along this crosssection show the discrepancies more recognizably. When no rotational transformation () is applied to the input data, the profile agrees well with the truth data profile. As large rotational transformations are applied to the input data, the predicted output profile clearly deviates from the true data, with erroneous negative values for reference frame rotation of . These results signify the importance of the frameinvariance property of the neural operators for scientific problems where differently aligned reference frames are used in different problem cases.
3.2 Comparison of GKN and VCNN
With two suitable candidates for frameinvariant neural operators solving scientific problems, we present here a comparison of their predictive performance and computational efficiency. The same training and testing dataset setups are used as in the earlier subsection, i.e., a training dataset at three angles of attack (10°, 20°and 30°) and a testing dataset at four angles of attack (5°, 15°, 25° and 35°) with transformed coordinate systems equal to the respective flow angles of attack. Table 3
shows the comparison of predictive performance for GKN and VCNN. The results of both models are compared using the same stencil size (150), i.e., the same number of sampled data points in the nonlocal region of influence. Furthermore, the number of learnable hyperparameters for GKN and for the fitting network of VCNN are approximately equal (
). Results in Table 3 show that the predictive performance of GKN is better than VCNN, despite both learning from the training dataset with comparable accuracy.



NN Model  Training Error  AOA = 15°  AOA = 25°  AOA = 5°  AOA = 35°  


GKN  with RI  0.33%  1.7%  1.4%  2.7%  2.6%  
VCNN  0.31%  3.5%  3.2%  6.8%  5.3% 
Reasons for this relatively inferior performance of VCNN are not fully comprehensible yet, despite both networks using similar information as input data. The difference in the predictive performance might be attributable to the processing of this information. A simple feedforward network is used as a fitting network in VCNN for mapping the invariant input information to the scalar quantity . In GKN, the graph representation is updated for each node by message passing process through kernel function and the resulting node features are updated twice (equal to the depth of the network). With each graph update, features of all the nodes are updated based on the information from their corresponding neighbouring nodes in the graph. The scalar quantity is then predicted by global averaging operation over all the node features. The superior performance of GKN can be attributed to this intricate processing of information. Possible improvement in VCNN requires further investigation and will be a subject of future work.
Qualitative comparison of the predictions made by both models is shown in Fig. 9 and Fig. 10 for interpolating and extrapolating in test conditions, respectively. Contours of the scalar field for both models resemble quite well with those of true scalar field . As expected, more information generally leads to better predictive performance. This analysis is shown in Fig. 11 where the predictive error percentage for the testing dataset is plotted as a function of the stencil size. It can be observed that the predictive performance improves with increasing stencil size. For VCNN, the predictive performance shows a converging behavior with increasing stencil size. For GKN, predictive performance improves with increasing stencil size; however, no observation about the convergence can be made from the given results.




In Fig. 11, although the results for VCNN are shown for the maximum stencil size of the 600, same are shown for GKN only until 225, because training GKN is prohibitively expensive with larger stencil size. The computational cost in terms of memory usage and training time presents a major disadvantage for graph kernel networks. For the scalar field dataset over an airfoil, a comparison is shown in Fig. 12, where the memory usage and training times are plotted with increasing stencil size. For both neural operators, the time cost is calculated based on the use of one NVIDIA V100 (“Volta”) GPU for training. As the stencil size increases, the data size for GKN quadratically increases as it involves arranging all the pairwise combinations of the nodes in the graph and the corresponding edge features. For each stencil size , the number of edges has the order of . This quadratic increase in data size can also be observed in Fig. 12(a) where CPU memory required to preprocess the training dataset is shown with increasing stencil size. In comparison, the memory usage for VCNN increases linearly. With the same stencil size, memory usage for GKN is approximately two orders of magnitude higher than that of VCNN. For the stencil size of 225, the required CPU memory easily exceeds GB to preprocess the training dataset for GKN, compared to the requirement of GB for VCNN. Such an increase in data size is reflected in the training time cost of GKN as well. For VCNN, the linear increase in data size is offset by the parallel computing of GPU. However, the quadratic increase in data size for GKN exceeds the parallel computing capacity of the GPUs, leading to roughly similar order () of increase in training time. With increasing stencil size, the memory requirement and time cost become prohibitively expensive for GKN, which presents a significant disadvantage in comparison with VCNN. On the flip side, for a very small stencil size (), the training time cost of GKN is comparable to that of VCNN, albeit still more expensive in memory. However, such a stencil size range, illustrated by the shaded gray region in Fig. 12, is not a realistic cloud size for many applications in computational physics.
Beyond the CPU memory cost requirement for preprocessing the data, training of GKN using GPUs also presents a challenging requirement for GPU memory. For the current research work, the training dataset was distributed into large number of batches (800) and each batch is moved to GPU for training one by one due to the limitation of GPU memory. With increasing graph size (stencil size), the requirement would be to distribute the training dataset in an even larger number of batches. For many scientific problems, the graph size could be significantly larger than the current problem, which could lead to a situation where GPU memory requirement for even one graph could prove excessive. Such issues of handling large memory costs for graph kernel networks is in itself a topic of research [35, 36, 37, 38]. In this regard, VCNN provides significantly efficient neural operators for scientific problems.
We remark that in [21] the authors proposed to randomly sample times subgraphs with node size from the region to build input–output data pairs. In their cases, the authors observe that parameters and give a good balance between prediction accuracy and computation cost. In the present work, we have chosen and . The general guideline for the choice of and in GKN still remains elusive and requires further investigation. Meanwhile, the idea of sampling smaller subgraphs can also be employed by VCNN to improve the training efficiency; see the discussion in [39].
4 Conclusion
Frameinvariant neural operators for transport partial differential equations (PDEs) have been analyzed. Transport of a scalar quantity in a flow over an airfoil is considered to analyze the frame invariance properties of both neural operators. First, graphical kernel network (GKN) model is considered which leverages the structure and properties of graphs to embed nonlocal dependencies. We propose to use invariant input features for the GKN model to guarantee frame invariance. Secondly, we have considered vectorcloud neural network (VCNN) which has both nonlocal dependencies and frame invariance embedded in the model.
GKN with invariant input features has shown to achieve frame invariance, which is demonstrated for the testing datasets with transformed reference frames. VCNN also demonstrated invariance to transformation in the reference frame; however it showed slightly worse predictive performance as compared to GKN, given the same input data and number of learnable parameters. With increasing stencil size, both models showed improved predictive performance. However, as graphbased networks are intrinsically expensive to train, GKN becomes prohibitively expensive for larger stencil size. In comparison, VCNN demonstrated orders of magnitude better computational efficiency for increasing stencil size. This observation is critical for many scientific problems where large mesh sizes and possibly larger number of feature vectors are used. For such cases, using GKN becomes infeasible and VCNN can be used efficiently as a surrogate model for transport PDEs with embedded invariance properties and nonlocal dependencies.
Despite the satisfactory performance and computational efficiency of VCNN, a future study is required to address its slightly inferior performance to GKN. Since GKN uses relatively intricate processing of the invariant input data, different options can be explored for the fitting network of VCNN to enhance model accuracy.
Acknowledgments
H. Xiao is supported by the U.S. Air Force under agreement number FA86501922204. The U.S. Government is authorised to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The computational resources used for this project were provided by the Advanced Research Computing (ARC) of Virginia Tech, which is gratefully acknowledged.
Appendix A Performance of an improved VCNN
In this appendix we consider a variant of VCNN for potential performance improvement. Pairwise projection among feature vectors is used to determine the invariant feature matrix (Eq. 8). Input feature matrix can also be written as a stack of three submatrices as:
(11) 
where submatrices , and comprises of relative spatial coordinates, velocity vectors and scalar quantities of the point cloud, respectively. Pairwise projection can then be represented as:
(12) 
To retain more information in the invariant feature matrix , the pairwise projection has been segregated in order to determine three different invariant feature matrices as:
(13)  
(14)  
(15) 
where and are based on the pairwise projection of spatial coordinates and velocity vectors, respectively. Three components are then concatenated to determine invariant features (replacing the role of in the original VCNN), which are then mapped to the scalar quantity through the fitting network. Such form closely resembles the determination of rotational invariant edge features (4) of GKN as well.
Since more information is being kept in invariant features with larger dimensions, the number of learnable parameters of the fitting network significantly increases (123000) compared to the earlier presented VCNN with 33000 parameters. Despite the near fourfold increase in model size, the training time per epoch increases only to 4.1 from 3.3. Based on such representation of invariant feature matrix, results have been compared in Table 4. The reduction of prediction error percentage for training and interpolating the testing datasets can be attributed to the increased number of learnable parameters. However, extrapolating testing error percentage is comparable to that of the previously discussed architecture of VCNN.



NN Model  Training Error  AOA = 15°  AOA = 25°  AOA = 5°  AOA = 35°  


GKN  with RI  0.33%  1.7%  1.4%  2.7%  2.6%  
VCNN ()  0.31%  3.5%  3.2%  6.8%  5.3%  
VCNN ()  0.20%  2.3%  2.1%  5.0%  5.2% 
Appendix B Neural network architectures details
In this appendix, architecture details of both neural network models are given. VCNN hyperparameters have been adapted as originally proposed by Zhou et al. [8], for which details for embedding and fitting networks are mentioned separately in Table 5. To make a fair comparison between both neural network models, we have kept the total number of learnable parameters almost equal for both GKN and VCNN. Since nonlocal mapping from a point cloud or a graph to a scalar quantity is performed by the fitting network for VCNN, only the fitting network of VCNN has been considered for equivalency of total learnable parameters. The Adam optimizer [40]
is adopted to train both of the neural networks. The neural network architectures have been implemented in the machine learning framework PyTorch, while also using PyTorch Geometric for implementing GKN.
Embedding network  Fitting network ()  
No. of input neurons 

No. of hidden layers  2  1 
No. of output neurons  1 ()  
Neurons in each layer  (7, 32, 64, 64)  (256, 128, 1) 
Activation functions  ReLU  ReLU, Linear (last layer) 
No. of trainable parameters  19008  33025 
Kernel network  
No. of input neurons  
No. of hidden layers  
No. of output neurons  () 
Neurons in each layer  (16, 64, 96, 256) 
Activation functions  ReLU, Linear (last layer) 
No. of trainable parameters  32577 
References
 [1] A. Quarteroni, G. Rozza, Reduced order methods for modeling and computational reduction, Springer, Heidelberg, 2014.
 [2] D. J. Lucia, P. S. Beran, W. A. Silva, Reducedorder modeling: new approaches for computational physics, Progress in Aerospace Sciences 40 (1) (2004) 51–117.
 [3] B. Peherstorfer, K. Willcox, Datadriven operator inference for nonintrusive projectionbased model reduction, Computer Methods in Applied Mechanics and Engineering 306 (2016) 196–215.
 [4] K. Taira, M. S. Hemati, S. L. Brunton, Y. Sun, K. Duraisamy, S. Bagheri, S. T. M. Dawson, C.A. Yeh, Modal analysis of fluid flows: Applications and outlook, AIAA Journal 58 (3) (2020) 998–1022.
 [5] L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence 3 (2021) 218–229.
 [6] S. Wang, H. Wang, P. Perdikaris, Learning the solution operator of parametric partial differential equations with physicsinformed DeepONets, Science Advances 7 (40) (2021) eabi8605.
 [7] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Fourier neural operator for parametric partial differential equations, in: International Conference on Learning Representations, 2021.
 [8] X.H. Zhou, J. Han, H. Xiao, Frameindependent vectorcloud neural network for nonlocal constitutive modeling on arbitrary grids, Computer Methods in Applied Mechanics and Engineering 388 (2022) 114211.
 [9] N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, Neural operator: Learning maps between function spaces, in: AMS Fall Western Virtual Sectional Meeting, 2021.
 [10] O. C. Zienkiewicz, R. L. Taylor, P. Nithiarasu, J. Zhu, The finite element method, Vol. 3, McGrawhill London, 1977.

[11]
G. Rozza, D. B. P. Huynh, A. T. Patera, Reduced basis approximation and a posteriori error estimation for affinely parametrized elliptic coercive partial differential equations, Archives of Computational Methods in Engineering 15 (3) (2008) 229–275.
 [12] J. Ling, R. Jones, J. Templeton, Machine learning strategies for systems with invariance properties, Journal of Computational Physics 318 (2016) 22–35.
 [13] W. E, J. Han, L. Zhang, Machinelearningassisted modeling, Physics Today 74 (7) (2021) 36–41.

[14]
M. L. Kaandorp, R. P. Dwight, Datadriven modelling of the reynolds stress tensor using random forests with invariance, Computers & Fluids 202 (2020) 104497.
 [15] H. Frezat, G. Balarac, J. Le Sommer, R. Fablet, R. Lguensat, Physical invariance in neural networks for subgridscale scalar flux modeling, Phys. Rev. Fluids 6 (2021) 024607.
 [16] R. Wang, R. Walters, R. Yu, Incorporating symmetry into deep dynamics models for improved generalization, in: International Conference on Learning Representations, 2021.
 [17] M. Raissi, P. Perdikaris, G. Karniadakis, Physicsinformed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707.
 [18] H. Gao, M. J. Zahr, J. Wang, Physicsinformed graph neural galerkin networks: A unified framework for solving pdegoverned forward and inverse problems, arXiv preprint arXiv:2107.12146510 (2021).
 [19] T. Pfaff, M. Fortunato, A. SanchezGonzalez, P.W. Battaglia, Learning meshbased simulation with graph networks, in: International Conference on Learning Representations, 2021.
 [20] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, G. Monfardini, The graph neural network model, IEEE Transactions on Neural Networks 20 (1) (2009) 61–80.
 [21] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Neural operator: Graph kernel network for partial differential equations, arXiv preprint arXiv:2003.03485 (2020).
 [22] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, A. Stuart, K. Bhattacharya, A. Anandkumar, Multipole graph neural operator for parametric partial differential equations, in: Advances in Neural Information Processing Systems, Vol. 33, 2020, pp. 6755–6766.
 [23] C. Lyle, M. van der Wilk, M. Kwiatkowska, Y. Gal, B. BloemReddy, On the benefits of invariance in neural networks, arXiv preprint arXiv:2005.00178 (2020).

[24]
R. Q. Charles, H. Su, M. Kaichun, L. J. Guibas, PointNet: Deep learning on point sets for 3D classification and segmentation, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 77–85.
 [25] X. Sun, Z. Lian, J. Xiao, Srinet: Learning strictly rotationinvariant representations for point cloud classification and segmentation, in: Proceedings of the 27th ACM International Conference on Multimedia, 2019, pp. 980–988.
 [26] A. Kashefi, D. Rempe, L. J. Guibas, A pointcloud deep learning framework for prediction of fluid flow fields on irregular geometries, Physics of Fluids 33 (2) (2021) 027104.
 [27] D. C. Wilcox, Turbulence modeling for CFD, 3rd Edition, DCW Industries, 2006.
 [28] UIUC Applied Aerodynamics Group, UIUC airfoil data site, \(\mathsf{https://mselig.ae.illinois.}\) , [Online; accessed 23February2021] (2020).
 [29] X.H. Zhou, J. Han, H. Xiao, Learning nonlocal constitutive models with neural networks, Computer Methods in Applied Mechanics and Engineering 384 (2021) 113927.
 [30] J. Stokes, K. Yang, K. Swanson, W. Jin, A. CubillosRuiz, N. Donghia, C. MacNair, S. French, L. Carfrae, Z. BloomAckermann, V. Tran, A. ChiappinoPepe, A. Badran, I. Andrews, E. Chory, G. Church, E. Brown, T. Jaakkola, R. Barzilay, J. Collins, A deep learning approach to antibiotic discovery, Cell 181 (2) (2020) 475–483.
 [31] A. SanchezGonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, P. Battaglia, Learning to simulate complex physics with graph networks, in: Proceedings of the 37th International Conference on Machine Learning, Vol. 119 of Proceedings of Machine Learning Research, PMLR, 2020, pp. 8459–8468.
 [32] F. Monti, F. Frasca, D. Eynard, D. Mannion, M. M. Bronstein, Fake news detection on social media using geometric deep learning, arXiv preprint arXiv:1902.06673 (2019).
 [33] W. Jiang, J. Luo, Graph neural network for traffic forecasting: A survey, arXiv preprint arXiv:2101.11174 (2021).
 [34] C. Eksombatchai, P. Jindal, J. Z. Liu, Y. Liu, R. Sharma, C. Sugnet, M. Ulrich, J. Leskovec, Pixie: A system for recommending 3+ billion items to 200+ million users in realtime, International World Wide Web Conferences Steering Committee, 2018, pp. 1775–1784.
 [35] J. Chen, T. Ma, C. Xiao, FastGCN: Fast learning with graph convolutional networks via importance sampling, in: International Conference on Learning Representations, 2018.
 [36] M. Fey, J. E. Lenssen, Fast graph representation learning with PyTorch Geometric, in: ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds, 2019.
 [37] Z. Lin, C. Li, Y. Miao, Y. Liu, Y. Xu, Pagraph: Scaling GNN training on large graphs via computationaware caching, in: Proceedings of the 11th ACM Symposium on Cloud Computing, SoCC ’20, Association for Computing Machinery, New York, NY, USA, 2020, pp. 401–415.
 [38] Z. Jia, S. Lin, M. Gao, M. Zaharia, A. Aiken, Improving the accuracy, scalability, and performance of graph neural networks with roc, in: I. Dhillon, D. Papailiopoulos, V. Sze (Eds.), Proceedings of Machine Learning and Systems, Vol. 2, 2020, pp. 187–198.
 [39] J. Han, X.H. Zhou, H. Xiao, VCNNe: A vectorcloud neural network with equivariance for emulating Reynolds stress transport equations, Unpublished manuscript.
 [40] D. Kingma, J. Ba, Adam: a method for stochastic optimization, in: Proceedings of the International Conference on Learning Representations, 2015.
Comments
There are no comments yet.