DeepAI AI Chat
Log In Sign Up

Graph Neural Tangent Kernel: Convergence on Large Graphs

by   Sanjukta Krishnagopal, et al.
berkeley college

Graph neural networks (GNNs) achieve remarkable performance in graph machine learning tasks but can be hard to train on large-graph data, where their learning dynamics are not well understood. We investigate the training dynamics of large-graph GNNs using graph neural tangent kernels (GNTKs) and graphons. In the limit of large width, optimization of an overparametrized NN is equivalent to kernel regression on the NTK. Here, we investigate how the GNTK evolves as another independent dimension is varied: the graph size. We use graphons to define limit objects – graphon NNs for GNNs, and graphon NTKs for GNTKs, and prove that, on a sequence of growing graphs, the GNTKs converge to the graphon NTK. We further prove that the eigenspaces of the GNTK, which are related to the problem learning directions and associated learning speeds, converge to the spectrum of the GNTK. This implies that in the large-graph limit, the GNTK fitted on a graph of moderate size can be used to solve the same task on the large-graph and infer the learning dynamics of the large-graph GNN. These results are verified empirically on node regression and node classification tasks.


page 1

page 2

page 3

page 4


Increase and Conquer: Training Graph Neural Networks on Growing Graphs

Graph neural networks (GNNs) use graph convolutions to exploit network i...

Fast Graph Neural Tangent Kernel via Kronecker Sketching

Many deep learning tasks have to deal with graphs (e.g., protein structu...

On the Universality of Graph Neural Networks on Large Random Graphs

We study the approximation power of Graph Neural Networks (GNNs) on late...

Triple Sparsification of Graph Convolutional Networks without Sacrificing the Accuracy

Graph Neural Networks (GNNs) are widely used to perform different machin...

G-Signatures: Global Graph Propagation With Randomized Signatures

Graph neural networks (GNNs) have evolved into one of the most popular d...

Pre-training Graph Neural Networks with Kernels

Many machine learning techniques have been proposed in the last few year...

Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation

Consistency training is a popular method to improve deep learning models...