Network In Graph Neural Network

1. Introduction

Graph Neural Networks (GNNs) capture local graph structure and feature information in a trainable fashion to derive powerful node representations. They have shown promising success on multiple graph-based machine learning tasks

(ying2018graph; scarselli2008graph; hu2020open) and are widely adopted by various web applications including social network (ying2018graph; rossi2020temporal), recommendation (berg2017graph; fan2019graph; yu2021self), fraud detection (wang2019fdgars; li2019spam; liu2020alleviating), etc. Various strategies have been proposed to improve the expressiveness of GNNs (hamilton2017inductive; velivckovic2017graph; xu2018powerful; schlichtkrull2018modeling).

(a) The test accuracy of GraphSage with different settings.

One natural candidate for improving the performance of a GNN is to increase its parameter size by either expanding the hidden dimension or the number of GNN layers. However, this can result in a large computational cost with only a modest performance gain. As a representative example, Figure 1 displays the performance of GraphSage (hamilton2017inductive) under different settings on the ogbn-products dataset and the corresponding model parameter sizes. From these results, it can be seen that either increasing the hidden dimension or increasing the number of GNN layers increases the model parameter size exponentially, but brings little performance improvement in tersm of test accuracy. For example, in order to improve the accuracy of a 3-layer GraphSage model by 1%, we need to add 2.3 $\times$ more parameters (by increasing the hidden dimension from 256 to 512). Furthermore, with a larger hidden dimension a model is more likely to overfit the training data. On the other head, stacking multiple GNN layers may oversmooth the features of nodes (oono2019graph; chen2020measuring). As shown in Figure 0(a), GrageSage reaches its peak performance with only 3 GNN layers and a hidden dimension of 512.

Inspired by the Network-in-Network architecture (lin2013network), we present Network-in-Graph Neural-Network (NGNN ), a model agnostic methodology that allows arbitrary GNN models to increase their model capacity by making the model deeper. However, instead of adding more GNN layers, NGNN deepens a GNN model by inserting non-linear feedforward neural network layer(s) within each GNN layer. This leads to a much smaller memory footprint than recent alternative deep GNN architectures (li2019deepgcns; li2020deepergcn) and can be applied to all kinds of GNN models with various training methods including full-graph training, neighbor sampling (hamilton2017inductive), cluster-based sampling (chiang2019cluster) and local subgraph sampling (zhang2018link). Thus, it can easily scale to large graphs. Moreover, analysis of NGNN in conjunction with GraphSage on perturbed ogbn-products showed that NGNN is a cheap yet effective way to keep the model stable against either node feature or graph structure perturbations.

In this work, we applied NGNN to GCN (kipf2016semi), GraphSage (hamilton2017inductive), GAT (velivckovic2017graph) and AGDN (sun2020adaptive) and SEAL (zhang2018link). We also combine the proposed technique with different mini-batch training methods including neighbor sampling, graph clustering and local subgraph sampling. We conducted comprehensive experiments on several large-scale graph datasets for both node classification and link prediction leading to the following conclusions (which hold as of the time of this submission):

NGNN improves the performance of GraphSage and GAT and their variants on node classification datasets including ogbn-products, ogbn-arxiv, ogbn-proteins and reddit. It improves the test accuracy by 1.6% on the ogbn-products datasets for GraphSage. Furthermore, NGNN with AGDN+BoT+self-KD+C&S (huang2020combining) achieves the forth place on the ogbn-arxiv leaderboard¹¹1https://ogb.stanford.edu/docs/leader_nodeprop/ and NGNN with GAT+BoT (wang2021bag) achieves second place on the ogbn-proteins leaderboard with many fewer model parameters.
NGNN improves the performance of SEAL, GCN and GraphSage and their variants on link prediction datasets including ogbl-collab, ogbl-ppa and ogbl-ppi. For example, it increases the test hits@100 score by 7.08% on the ogbl-ppa dataset for SEAL, which outperforms all the state-of-the-art approaches on the ogbl-ppa leaderboard²²2https://ogb.stanford.edu/docs/leader_linkprop/ by a substantial margin. Furthermore, NGNN achieves an improvement of the test hits@20 score by 6.22% on the ogbl-ppi dataset for GraphSage+EdgeAttr, which also takes the first place on the ogbl-ppi learderboard.
NGNN improves the performance of GraphSage and GAT under different training methods including full-graph training, neighbor sampling, graph clustering, and subgraph sampling.
NGNN is a more effective way of improving the model performance than expanding the hidden dimension. It takes less parameter size and less training time to get better performance than simply doubling the hidden dimension.

In summary, we present NGNN , a method that deepens a GNN model without adding extra GNN message-passing layers. We show that NGNN significantly improves the performance of vanilla GNNs on various datasets for both node classification and link prediction. We demonstrate the generality of NGNN by applying them to various GNN architectures.

2. Related Work

Deep models have been widely studied in various domains including computer vision

(simonyan2014very; he2016deep)

, natural language processing

(brown2020language), and speech recognition (zhang2017very). VGG (simonyan2014very)

investigates the effect of the convolutional neural network depth on its accuracy in the large-scale image recognition setting. It demonstrates the depth of representations is essential to the model performance. But when the depth grows, the accuracy will not always grow. Resnet

(he2016deep)

eases the difficulties on training the deep model by introducing residual connections between input and output layers. DenseNet

(huang2017cvpr)

takes this idea a step further by adding connections across layers. GPT-3

(brown2020language) presents an autoregressive language model with 96 layers that achieves SOTA performance on various NLP tasks. Even so, while deep neural networks have achieved great success in various domains, the use of deep models in graph representation leaning is less well-established.

Most recent works (li2019deepgcns; li2020deepergcn; li2021training) attempt to train deep GNN models with a large number of parameters and achieved SOTA performance. For example, DeepGCN (li2019deepgcns) adapts the concept of residual connections, dense connections, and dilated convolutions (yu2015multi) to training very deep GCNs. However DeepGCN and its successor DeeperGCN (li2020deepergcn) have large memory footprints during model training which can be subject to current hardware limitations. RevGNN (li2021training) explored grouped reversible graph connections to train a deep GNN and has a much smaller memory footprint. However, RevGNN can only work with full-graph training and cluster-based mini-batch training, which makes it difficult to work with other methods designed for large scale graphs such as neighbor sampling (hamilton2017inductive) and layer-wise sampling (chen2018fastgcn). In contrast, NGNN deepens a GNN model by inserting non-linear feedforward layer(s) within each GNN layer. It can be applied to all kinds of GNN models with various training methods including full-graph training, neighbor sampling (hamilton2017inductive), layer-wise sampling (chen2018fastgcn) and cluster-based sampling (chiang2019cluster).

Xu et al. (xu2018powerful)

used Multilayer Perceptrons (MLPs) to learn the injective functions of the Graph Isomophism Network (GIN) model and showed its effectiveness on graph classification tasks. But they did not show whether adding an MLP within GNN layers works effectively across wide-ranging node classification and link prediction tasks. Additionally, You et al.

(you2020design) mentioned that adding MLPs within GNN layer could benefit the performance. However, they did not systematically analyze the reason for the performance improvement introduced by extra non-linear layers, nor evaluate with numerous SOTA GNN architectures on large-scale graph datasets for both node classification and link prediction tasks.

(a) Gaussian noise is concatenated with node features.

3. Building Network in Graph Neural Network Models

3.1. Preliminaries

A graph is composed of nodes and edges $G = (V, E)$ , where $V = (v_{1}, \dots, v_{N})$ is the set of $N$ nodes and $E \subseteq V \times V$ is the set of edges. Furthermore, $A \in {0, 1}^{N \times N}$ denotes the corresponding adjacency matrix of $G$ . Let $X \in R^{N \times D}$ be the node feature space such that $X = (x_{1}, \dots, x_{N})^{T}$ where $x_{i}$ represents the node feature of $v_{i}$ . Formally, the $(l + 1)$ -th layer of a GNN is defined as:³³3We omit edge features for simplicity.

(1)

h^{(l + 1)} = σ (f_{w} (G, h^{l})),

where the function $f_{w} (G, h^{l})$ is determined by learnable parameters $w$ and $σ (\cdot)$

is an optional activation function. Additionally,

h^{l}

represents the embeddings of the nodes in the

l

-th layer, and

h^{l} = X

when

l = 1

. With an

L

-layer GNN, the node embeddings in the last layer

h^{L}

are used by downstream tasks like node classification and link prediction.

3.2. Basic NGNN Design

Inspired by the network-in-network architecture (lin2013network), we deepen a GNN model by inserting non-linear feedforward neural network layer(s) within each GNN layer. The $(l + 1)$ -th layer in NGNN is thus constructed as:

(2)

h^{(l + 1)} = σ (g_{n g n n} (f_{w} (G, h^{l}))) .

The calculation of $g_{n g n n}$ is defined layer-wise as:

(3)

\begin{matrix} g_{n g n n}^{1} = σ (f_{w} (G, h^{l}) w^{1}) \dots g_{n g n n}^{k} = σ (g_{n g n n}^{k - 1} w^{k}) \end{matrix}

where $w_{1}, \dots, w_{k}$ are learnable weight matrices, $σ (\cdot)$ is an activation function, and $k$ is the number of in-GNN non-linear feedforward neural network layers. The first in-GNN layer takes the output of $f_{w}$

as input and performs the non-linear transformation.

3.3. Discussion

In this section, we demonstrate that a NGNN architecture can better handle both noisy node features and noisy graph structures relative to its vanilla GNN counterpart.

Figure 3. The model performance of GraphSage, GraphSage with one additional non-linear layer in each GNN layer (NGNN-GraphSage-1) and GraphSage with two additional non-linear layers in each GNN layer (NGNN-GraphSage-2) on ogbn-product with randomly added noise edges.

Remark 1 ().

GNNs work well when the input features consist of distinguishable true features and noise. But when the true features are mixed with noise, GNNs can struggle to filter out the noise, especially as the noise level increases.

GNNs follow a neural message passing scheme (gilmer2017neural) to aggregate information from neighbors of a target node. In doing so, they can perform noise filtering and learn from the resulting signal when the noise is in some way distinguishable from true features, such as when the latter are mostly low-frequency (nt2019revisiting). However, when the noise level becomes too large and is mixed with true features, it cannot easily be reduced by GNNs (huang2019residual). Figure 2 demonstrates this scenario. Here we randomly added Gaussian noise $N = N (0, σ)$ to node features in ogbn-products data, where $σ$

is the standard deviation ranging from 0.1 to 5.0. We adopt two different methods for adding noise: 1)

X = [X | N]

as shown in Figure 1(a), where

|

is a concatenation operation, and 2)

X = [X + N]

as shown in Figure 1(b). We trained GraphSage models (using the DGL (wang2020deep)

implementation) under five different settings: 1) baseline GraphSage with default 3-layer structure and hidden dimension of 256, denoted as GraphSage; 2) GraphSage with the hidden dimension increased to 512, denoted as GraphSage-512; 3) 4-layer GraphSage, denoted as GraphSage-4layer; 4) GraphSage with one additional non-linear layer in each GNN layer, denoted as NGNN-GraphSage-1 and 5) GraphSage with two additional non-linear layers in each GNN layer, denoted as NGNN-GraphSage-2. In all cases we used ReLU as the activation function. As shown in Figure

1(a), GraphSage performs well when the noise is highly distinguishable from the true features. But the performance starts dropping when the noise is mixed with the true features and decays faster when

σ

becomes larger than 1.0 as shown in Figure 1(b).

The same scenario happens with the gfNN model (nt2019revisiting), which is formed by transforming input node features via muliplications of the adjacency matrix followed by application of a single MLP block. This relatively simple model was shown to be more noise tolerant than GCN and SGC (wu2019simplifying); however, the performance of gfNN turns out to be much lower than the baseline GraphSage model in our experiments, so we do not present results here.

Remark 2 ().

NGNN is a cheap yet effective way to form a GNN architecture that is stable against node feature perturbations.

One potential way to improve the denoising capability of a GNN model is to increase the parameter count via a larger hidden dimension. As shown in Figure 1(b), GraphSage-512 does perform better than the baseline GraphSage. But it is also more expensive as its parameter size (675,887) is $3.27 \times$ larger than that of baseline GraphSage (206,895). And it is still not as effective as either NGNN model, both of which use considerably fewer parameters (see below) and yet have more stable performance as the noise level increases.

An alternative strategy for increasing the model parameter count is to add more GNN layers. As shown in Figure 1(b), by adding one more GNN layer, GraphSage-4layer does outperform baseline GraphSage when $σ$ is smaller than 4.0. However, as a deeper GNN potentially aggregates more noisy information from its $L$ hop neighbors (zeng2020deep), the performance of GraphSage-4layer drops below baseline GraphSage when $σ$ is 5.0.

In contrast to the above two methods, NGNN-GraphSage achieves much better performance as shown in Figure 1(b) with fewer parameters (272,687 for NGNN-GraphSage-1 and 338,479 for NGNN-GraphSage-2) than GraphSage-512 and without introducing new GNN layers. It can help maintain model performance when $σ$ is smaller than 1.0 and slow the downward trend when $σ$ is larger than 1.0 compared to the other three counterparts.

Remark 3 ().

NGNN with GNNs can also keep the model stable against graph structure perturbation.

We now show that by applying NGNN to a GNN, it can better deal with graph structure data perturbation. For this purpose, we randomly added $K$ edges to the original graph of ogbn-products, where $K$ is the ratio of newly added noise edges to the existing edges. For example $K = 0.01$ means we randomly added 618.6K edges.⁴⁴4The graph of ogbn-product has 61,859,140 edges. We trained 3-layer GraphSage models with a hidden dimension of 256 under three different settings: 1) vanilla GraphSage, denoted as GraphSage; 2) GraphSage with one additional non-linear layer in each GNN layer, denoted as NGNN -GraphSage-1 and 3) GraphSage with two additional non-linear layers in each GNN layer, denoted as NGNN -GraphSage-2. Figure 3 shows the results. It can be seen that, NGNN can help preserve the model performance when $K$ is smaller than 0.01 and ease the trend of performance downgrade after $K$ is larger than 0.01 comparing to vanilla GraphSage.

4. Experiments

We next provide experimental evidence to show that NGNN works will with various GNN architectures for both node classification and link prediction tasks in Sections 4.2 and 4.3. We also show that NGNN works with different training methods in Section 4.4. Finally, we discuss the impact of different NGNN settings in Sections 4.5 and 4.6.

4.1. Evaluation Setup

Datasets

We conducted experiments on seven datasets, including ogbn-products, ogbn-arxiv and ogbn-proteins from ogbn (hu2020open) and reddit⁵⁵5http://snap.stanford.edu/graphsage/ for node classification, and ogbl-collab, ogbl-ppa and ogbl-citaiton2 from ogbl (hu2020open) for link prediction. The detailed statistics are summarized in Table 1.

Node Classification
Datasets	# Nodes	# Edges
ogbn-products	2,449,029	61,859,140
ogbn-arxiv	169,343	1,166,243
ogbn-proteins	132,524	39,561,252
reddit	232,965	114,615,892
Link Prediction
ogbl-collab	235,868	1,285,465
ogbl-ppa	576,289	30,326,273
ogbl-ddi	4,267	1,334,889

Table 1. Datasets statistics.

Node classification task
GNN Model	Description
GraphSage	Vanilla GraphSage with neighbor sampling.
GraphSage-Cluster	Vanilla GraphSage with cluster based sampling (chiang2019cluster).
GAT-FLAG	GAT with FLAG (kong2020flag) enhancement.
GAT+BoT	GAT with bag of tricks (wang2021bag).
AGDN+BoT	AGDN with bag of tricks.
AGDN+BoT+self-KD+C&S	AGDN with bag of tricks, knowledge distillation and correct&smooth(huang2020combining).
Link prediction task
SEAL-DGCNN	Vanilla SEAL using DGCNN (zhang2018end) as the backbone GNN.
GCN-full	Vanilla GCN with full graph training.
GraphSage-full	Vanilla Sage with full graph training.
GraphSage+EdgeAttr	GraphSage with edge attribute.

GraphSage+EdgeAttr comes from https://github.com/lustoo/OGB_link_prediction

Table 2. Baseline GNN models used in evaluation.

Dataset		ogbn-products	ogbn-arxiv	ogbn-proteins	reddit
Eval Metric		Accuracy(%)	Accuracy(%)	ROC-AUC(%)	Accuracy(%)
GraphSage	Vanilla	78.27 $\pm$ 0.45	71.15 $\pm$ 1.66	75.67 $\pm$ 1.72	96.19 $\pm$ 0.08
GraphSage	NGNN	79.88 $\pm$ 0.34	71.77 $\pm$ 1.18	76.30 $\pm$ 0.96	96.21 $\pm$ 0.04
GraphSage-Cluster	Vanilla	78.72 $\pm$ 0.63	56.57 $\pm$ 1.56	67.45 $\pm$ 1.21	95.27 $\pm$ 0.09
GraphSage-Cluster	NGNN	78.91 $\pm$ 0.59	56.76 $\pm$ 1.08	68.12 $\pm$ 0.96	95.34 $\pm$ 0.09
GAT-NS	Vanilla	79.23 $\pm$ 0.16	72.10 $\pm$ 1.12	81.76 $\pm$ 0.17	96.12 $\pm$ 0.02
GAT-NS	NGNN	79.67 $\pm$ 0.09	71.88 $\pm$ 1.10	81.91 $\pm$ 0.21	96.45 $\pm$ 0.05
GAT-FLAG	Vanilla	80.75 $\pm$ 0.14	71.56 $\pm$ 1.11	81.81 $\pm$ 0.15	95.27 $\pm$ 0.02
GAT-FLAG	NGNN	80.99 $\pm$ 0.09	71.74 $\pm$ 1.10	81.84 $\pm$ 0.11	95.68 $\pm$ 0.03

Any score difference between vanilla GNN and NGNNGNN that is greater than 0.5% is highlighted with boldface.

Table 3. Performance of NGNN on ogbn-products, ogbn-arxiv, ogbn-proteins and reddit.

Dataset	Model		Accuracy(%)
ogbn-arxiv	AGDN+BoT	Vanilla	74.03 $\pm$ 0.15
ogbn-arxiv	AGDN+BoT	NGNN	74.25 $\pm$ 0.17
ogbn-arxiv	AGDN+BoT+	Vanilla	74.28 $\pm$ 0.13
ogbn-arxiv	self-KD+C&S	NGNN	74.34 $\pm$ 0.14
ogbn-proteins	GAT+BoT	Vanilla	87.73 $\pm$ 0.18
ogbn-proteins	GAT+BoT	NGNN	88.09 $\pm$ 0.1

Table 4. Performance (as measured by classification accuracy and ROC-AUC for ogbn-arxiv and ogbn-proteins, respectively) of NGNN combined with bag of tricks on ogbn-arxiv and ogbn-proteins.

	Metric (%)	ogbl-collab		ogbl-ppa		ogbl-ddi
	Metric (%)	Vanilla GNN	NGNN	Vanilla GNN	NGNN	Vanilla GNN	NGNN
SEAL-	hit@20	45.76 $\pm$ 0.72	46.19 $\pm$ 0.58	16.10 $\pm$ 1.85	20.82 $\pm$ 1.76	30.75 $\pm$ 2.12	31.93 $\pm$ 3.00
DGCNN	hit@50	54.70 $\pm$ 0.49	54.82 $\pm$ 0.20	32.58 $\pm$ 1.42	37.25 $\pm$ 0.98	43.99 $\pm$ 1.11	42.39 $\pm$ 3.23
	hit@100	60.13 $\pm$ 0.32	60.70 $\pm$ 0.18	49.36 $\pm$ 1.24	56.44 $\pm$ 0.99	51.25 $\pm$ 1.60	49.63 $\pm$ 3.65
GCN-full	hit@10	35.94 $\pm$ 1.60	36.69 $\pm$ 0.82	4.00 $\pm$ 1.46	5.64 $\pm$ 0.93	47.82 $\pm$ 5.90	48.22 $\pm$ 7.00
	hit@50	49.52 $\pm$ 0.70	51.83 $\pm$ 0.50	14.23 $\pm$ 1.81	18.44 $\pm$ 1.88	79.56 $\pm$ 3.83	82.56 $\pm$ 4.03
	hit@100	55.74 $\pm$ 0.44	57.41 $\pm$ 0.22	20.21 $\pm$ 1.92	26.78 $\pm$ 0.92	87.58 $\pm$ 1.33	89.48 $\pm$ 1.68
GraphSage-	hit@10	32.59 $\pm$ 3.56	36.83 $\pm$ 2.56	3.68 $\pm$ 1.02	3.52 $\pm$ 1.24	54.27 $\pm$ 9.86	60.75 $\pm$ 4.94
full	hit@50	51.66 $\pm$ 0.35	52.62 $\pm$ 1.04	15.02 $\pm$ 1.69	15.55 $\pm$ 1.92	82.18 $\pm$ 4.00	84.58 $\pm$ 1.89
	hit@100	56.91 $\pm$ 0.72	57.96 $\pm$ 0.56	23.56 $\pm$ 1.58	24.45 $\pm$ 2.34	91.94 $\pm$ 0.64	92.58 $\pm$ 0.88
GraphSage+	hit@20	-	-	-	-	87.06 $\pm$ 4.81	93.28 $\pm$ 1.61
EdgeAttr	hit@50	-	-	-	-	97.98 $\pm$ 0.42	98.39 $\pm$ 0.21
	hit@100	-	-	-	-	98.98 $\pm$ 0.16	99.21 $\pm$ 0.08

The evaluation metrics used in ogb learderboard are hit@50 for ogbl-collab, hit@100 for ogbl-ppa and hit@20 for ogbl-ddi.
Any hit score difference between vanilla GNN and NGNN GNN that is greater than 1% is highlighted with boldface.
The evaluation metrics used for ogbl-ddi when profiling GCN-full and GraphSage-full are hit@20, hit@50 and hit@100.

Table 5. Performance of NGNN on ogbl-collab, ogbl-ppa and ogbl-ppi. We use the hit@20, hit@50 and hit@100 as the evaluation metrics.

We evaluated the effectiveness of NGNN by applying it to various GNN models including GCN (kipf2016semi), Graphsage (hamilton2017inductive), Graph Attention Network (GAT) (velivckovic2017graph), Adaptive Graph Diffusion Networks (AGDN) (sun2020adaptive), and SEAL (zhang2020revisiting) and their variants. Table 2 presents all the baseline models. We directly followed the implementation and configuration of each baseline model from the OGB (hu2020open) leaderboard and added non-linear layer(s) into each GNN layer for NGNN . Table 11 presents the detail configuration of each model. All models were trained on a single V100 GPU with 32GB memory. We report average performance over 10 runs for all models except SEAL related models. As training SEAL models is very expensive, we took 5 runs instead.

4.2. Node classification

Firstly, we analyzed how NGNN improves the performance of GNN models on node classification tasks. Table 3 presents the overall results. It can be seen that NGNN-based models outperform their baseline models in most of the cases. Notably, NGNN tends to performs well with GraphSage. It improves the test accuracy of GraphSage on ogbn-products and ogbn-arxiv by 1.61 and 0.62 respectively. It also improves the ROC-AUC score of GraphSage on ogbn-proteins by 0.63. But as the baseline performance of reddit dataset is quite high, not surprisingly, the overall improvement of NGNN is not significant.

We further analysis the performance of NGNN combined with bag of tricks (wang2021bag) on ogbn-arxiv and ogbn-proteins in Table 4. It can be seen that NGNN-based models outperform their vanilla counterparts. NGNN with AGDN+BoT+self-KD+C&S even achieves the first place over all the methods with no extension to the input data on the ogbn-arxiv leaderboard as of the time of this submission (The forth place on the entire ogbn-arxiv leaderboard). NGNN with GAT+BoT also achieves the second place on the ogbn-proteins leaderboard with 5.83 times fewer parameters compared with the current leading method RevGNN-Wide.⁶⁶6NGNN -GAT+Bot has 11,740,552 parameters while RevGNN-Wide has 68,471,608 parameters.

4.3. Link prediction

Secondly, we analyzed how NGNN improves the performance of GNN models on link prediction tasks. Table 5 presents the results on the ogbl-collab, ogbl-ppa and ogbl-ppi datasets. As shown in the tables, the performance improvement of NGNN over SEAL models is significant. NGNN improves the hit@20, hit@50 and hit@100 of SEAL-DGCNN by 4.72%, 4.67% and 7.08% respectively on ogbl-ppa. NGNN with SEAL-DGCNN achieves the first place on the ogbn-ppa leaderboard with an improvement of hit@100 by 5.82% over the current leading method MLP+CN&RA&AA ⁷⁷7https://github.com/lustoo/OGB_link_prediction. Furthermore, NGNN with GraphSage+EdgeAttr achieves the first place on the ogbl-ddi leaderboard with an improvement of hit@20 by 5.47% over the current leading method vanilla GraphSage+EdgeAttr. As GraphSage+EdgeAttr only provided the performance on ogbl-ppi, we do not compare its performance on other datasets. NGNN also works with GCN and GraphSage on link prediction tasks. As shown in the tables, It improves the performance of GCN and GraphSage in all cases. In particular, it improves the hit@20, hit@50 and hit@100 of GCN by 1.64%, 4.21% and 6.57% respectively on ogbl-ppa.

Sampling	full-graph	neighbor	cluster-based
Methods	full-graph	sampling	sampling
GraphSage	78.27	78.70	78.72
GraphSage-NGNN	79.88	79.11	78.91
GAT	80.75	79.23	71.41
GAT-NGNN	80.99	79.67	76.76

Table 6. Test accuracy (%) of GraphSage and GAT with and without NGNN trained with different training methods on ogbn-products.

Model	GraphSage
Hidden-size	128	256	512
baseline	77.44	78.27	79.37
NGNN -1layer	77.39	79.53	79.12
NGNN -2layer	78.79	79.88	79.94
NGNN -4layer	78.79	79.52	79.88
Model	GAT
Hidden-size	64	128	256
baseline	68.41	79.23	75.26
NGNN -1layer	69.72	79.67	77.53
NGNN -2layer	69.86	78.26	78.76
NGNN -4layer	69.41	78.23	78.61

Table 7. Test accuracy (%) of GraphSage and GAT with different number of non-linear layers added into GNN layers on ogbn-products.

Model	GraphSage
Hidden-size	128	256	512
baseline	70,703	206,895	675,887
NGNN -1layer	87,215	272,687	938,543
NGNN -2layer	103,727	338,479	1,201,199
NGNN -4layer	136,751	470,063	1,726,511
Model	GAT
Hidden-size	64	128	256
baseline	510,056	1,543,272	5,182,568
NGNN -1layer	514,152	1,559,656	5,248,104
NGNN -2layer	518,248	1,576,040	5,313,640
NGNN -4layer	526,440	1,608,808	5,444,712

Table 8. The parameter size of each model in Table 7.

Model	GraphSage (secs)
Hidden-size	128	256	512
baseline	18.71 $\pm$ 0.41	20.71 $\pm$ 0.74	25.40 $\pm$ 0.35
NGNN -1layer	19.57 $\pm$ 1.04	20.98 $\pm$ 0.47	29.07 $\pm$ 0.69
NGNN -2layer	19.25 $\pm$ 0.87	21.36 $\pm$ 0.48	30.01 $\pm$ 0.13
NGNN -4layer	19.79 $\pm$ 0.72	24.41 $\pm$ 0.38	32.33 $\pm$ 0.19

Table 9.

The single training epoch time of GraphSage with different number of non-linear layers added into GNN layers on ogbn-products.

4.4. NGNN with Different Training Methods

Finally, we presents the effectiveness of using NGNN with different training methods including full-graph training, neighbor sampling and cluster-based sampling. Table 6 presents the results. It can be seen that NGNN improves the performance of GraphSage and GAT with all kinds of training methods on ogbn-products. It is worth mentioning that NGNN also works with local subgraph sampling method proposed by SEAL (zhang2018link) as shown in Section 4.3.

4.5. Effectiveness of Multiple NGNN Layers

We studied the effectiveness of adding multiple non-linear layers to GNN layers on ogbn-products using GraphSage and GAT. Table 7 presents the results. The baseline model is a three-layer GNN model. We applied 1, 2 or 4 non-linear layers to each hidden GNN layer denoted as NGNN-1layer, NGNN-2layer and NGNN-4layer respectively. The GAT models use eight attention heads and all heads share the same NGNN layer(s). Table 7 presents the result. As shown in the table, NGNN-2layer always performed best with different hidden sizes in most of the cases. This reveals that adding non-linear layers can be effective, but the effect may vanish significantly when we continuously add more layers. The reason is straightforward, given that adding more non-linear layers can eventually cause overfitting.

We also observe that deeper models can achieve better performance with many fewer trainable parameters than wider models. Table 8 presents the model parameter size of each model. As shown in the table, the parameter size of GraphSage with NGNN-2layer and a hidden size of 256 is 338,479 which is 2 $\times$ smaller than the parameter size of vanilla GraphSage with a hidden-size of 512, i.e., 675,887. And its performance is much better than vanilla GraphSage with a hidden size of 512.

Furthermore, we also observe that adding NGNNlayers only slightly increase the model training time. Table 9 presents the single training epoch time of GraphSage under different configurations. As shown in the table, the epoch time of GraphSage with NGNN-2layer and a hidden size of 256 is only 3.1% longer than that of vanilla GraphSage with the same hidden size. However the corresponding parameter size is 1.63 $\times$ larger.

	GraphSage	GAT
baseline	78.27	79.23
NGNN -all	79.88	79.49
NGNN -input	79.81	78.87
NGNN -hidden	79.91	79.68
NGNN -output	78.60	78.45

Table 10. Test accuracy (%) of GraphSage and GAT when applying NGNN on different GNN layers on ogbn-product.

4.6. Effectiveness of Applying NGNN to Different GNN Layers

Finally, we studied the effectiveness of applying NGNN to only the input GNN layer (NGNN-input), only the hidden GNN layers (NGNN-hidden), only the output GNN layer (NGNN-output) and all the GNN layers on the ogbn-products dataset using GraphSage and GAT. The baseline model is a three-layer GNN model. The hidden dimension size is 256 and 128 for GraphSage and GAT respectively. Table 10 presents the results. As the table shows, only applying NGNN to the output GNN layer brings little or no benefit. While applying NGNN to hidden and input GNN layers can improve the model performance, especially applying NGNN to hidden layers. It demonstrates that the benefit of NGNN mainly comes from adding additional non-linear layers into the input and hidden GNN layers.

Dataset	Model	hidden size	layers	aggregation	NGNN position	NGNN setting
Node classification tasks
ogbn-product	GraphSage	256	3	mean	hidden-only	1-relu+1-sigmoid
ogbn-product	GraphSage-cluster	256	3	mean	hidden-only	1-relu+1-sigmoid
ogbn-product	GAT-flag	256	3	sum	hidden-only	1-relu+1-sigmoid
ogbn-product	GAT-ns	256	3	sum	hidden-only	1-relu+1-sigmoid
ogbn-arxiv	GraphSage	256	3	mean	hidden-only	1-relu+1-sigmoid
ogbn-arxiv	GraphSage-cluster	256	3	mean	hidden-only	1-relu+1-sigmoid
ogbn-arxiv	GAT-flag	256	3	sum	hidden-only	1-relu+1-sigmoid
ogbn-arxiv	GAT+BoT	120	6	sum	hidden-only	2-relu
ogbn-arxiv	AGDN+BoT	256	3	GAT-HA	hidden-only	1-relu
ogbn-arxiv	AGDN+BoT+self-KD+C&S	256	3	GAT-HA	hidden-only	1-relu
ogbn-protein	GraphSage	256	3	mean	hidden-only	1-relu
ogbn-protein	GraphSage-cluster	256	3	mean	hidden-only	1-relu
ogbn-protein	GAT-flag	256	3	sum	hidden-only	1-relu
ogbn-protein	GAT-ns	256	3	sum	hidden-only	1-relu
reddit	GraphSage	256	3	mean	hidden-only	1-relu+1-sigmoid
reddit	GraphSage-cluster	256	3	mean	hidden-only	1-relu+1-sigmoid
reddit	GAT-flag	256	3	sum	hidden-only	1-relu+1-sigmoid
reddit	GAT-ns	256	3	sum	hidden-only	1-relu+1-sigmoid
Link prediction tasks
ogbl-collab	Seal-DGCNN	256	3	sum	all-layers	1-relu
ogbl-collab	GCN-full	256	3	mean	hidden-only	2-relu
ogbl-collab	GraphSage-full	256	3	mean	hidden-only	2-relu
ogbl-ppa	Seal-DGCNN	32	3	sum	all-layers	1-relu
ogbl-ppa	GCN-full	256	3	mean	all-layers	2-relu
ogbl-ppa	GraphSage-full	256	3	mean	all-layers	2-relu
ogbl-ppi	Seal-DGCNN	32	3	sum	hidden-layers	1-relu
ogbl-ppi	GCN-full	256	2	mean	input-only	1-relu
ogbl-ppi	GraphSage-full	256	2	mean	input-only	1-relu
ogbl-ppi	GraphSage+EdgeAttr	512	2	mean	all-layers	2-relu

Table 11. Model settings of NGNN models. The column of NGNN position presents where we put the non-linear layers. hidden-only means only applying NGNN to the hidden GNN layers, input-only means only applying NGNN to the input layer, all-layer means applying NGNN to all the GNN layers.. The column of NGNN setting presents how we organize each NGNN layer. For example, 1-relu+1-sigmoid means NGNN contains one feedforward neural network with ReLU as its activation function followed by another feedforward neural network with Sigmoid as its activation function and 2-relu means NGNN contains two feedforward neural network layers with ReLU as the activation function of each layer.

5. Conclusion and Future Work

We present NGNN, a model agnostic methodology that allows arbitrary GNN models to increase their model capacity by inserting non-linear feedforward neural network layer(s) inside GNN layers. Moreover, unlike existing deep GNN approaches, NGNN does not have large memory overhead and can work with various training methods including neighbor sampling, graph clustering and local subgraph sampling. Empirically, we demonstrate that NGNN can work with various GNN models on both node classification and link prediction tasks and achieve state-of-the-art results. Future work includes evaluating NGNN on more GNN models and investigating whether NGNN can work on broader graph-related prediction tasks. We also plan to explore methodologies to make a single GNN layer deeper in the future.

Network In Graph Neural Network

Authors

Structure Enhanced Graph Neural Networks for Link Prediction

NDGGNET-A Node Independent Gate based Graph Neural Networks

Link Prediction Based on Graph Neural Networks

Analyzing the Performance of Graph Neural Networks with Pipe Parallelism

NENET: An Edge Learnable Network for Link Prediction in Scene Text

GROWL: Group Detection With Link Prediction

Integrating Transductive And Inductive Embeddings Improves Link Prediction Accuracy

1. Introduction

2. Related Work

3. Building Network in Graph Neural Network Models

3.1. Preliminaries

3.2. Basic NGNN Design

3.3. Discussion

Remark 1 ().

Remark 2 ().

Remark 3 ().

4. Experiments

4.1. Evaluation Setup

Datasets

4.2. Node classification

4.3. Link prediction

4.4. NGNN with Different Training Methods

4.5. Effectiveness of Multiple NGNN Layers

4.6. Effectiveness of Applying NGNN to Different GNN Layers

5. Conclusion and Future Work

References

Network In Graph Neural Network

Authors

Related Research

Structure Enhanced Graph Neural Networks for Link Prediction

NDGGNET-A Node Independent Gate based Graph Neural Networks

Link Prediction Based on Graph Neural Networks

Analyzing the Performance of Graph Neural Networks with Pipe Parallelism

NENET: An Edge Learnable Network for Link Prediction in Scene Text

GROWL: Group Detection With Link Prediction

Integrating Transductive And Inductive Embeddings Improves Link Prediction Accuracy

This week in AI

1. Introduction

2. Related Work

3. Building Network in Graph Neural Network Models

3.1. Preliminaries

3.2. Basic NGNN Design

3.3. Discussion

Remark 1 ().

Remark 2 ().

Remark 3 ().

4. Experiments

4.1. Evaluation Setup

Datasets

4.2. Node classification

4.3. Link prediction

4.4. NGNN with Different Training Methods

4.5. Effectiveness of Multiple NGNN Layers

4.6. Effectiveness of Applying NGNN to Different GNN Layers

5. Conclusion and Future Work

References