
Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
Graph Neural Networks (GNNs) have been studied through the lens of expre...
read it

Learning Graph Neural Networks with Approximate Gradient Descent
The first provably efficient algorithm for learning graph neural network...
read it

Increase and Conquer: Training Graph Neural Networks on Growing Graphs
Graph neural networks (GNNs) use graph convolutions to exploit network i...
read it

Towards Understanding Learning in Neural Networks with Linear Teachers
Can a neural network minimizing crossentropy learn linearly separable d...
read it

Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
While graph kernels (GKs) are easy to train and enjoy provable theoretic...
read it

Fast Learning of Graph Neural Networks with Guaranteed Generalizability: Onehiddenlayer Case
Although graph neural networks (GNNs) have made great progress recently ...
read it

A Note on OverSmoothing for Graph Neural Networks
Graph Neural Networks (GNNs) have achieved a lot of success on graphstr...
read it
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
We study how neural networks trained by gradient descent extrapolate, i.e., what they learn outside the support of the training distribution. Previous works report mixed empirical results when extrapolating with neural networks: while multilayer perceptrons (MLPs) do not extrapolate well in certain simple tasks, Graph Neural Network (GNN), a structured network with MLP modules, has shown some success in more complex tasks. Working towards a theoretical explanation, we identify conditions under which MLPs and GNNs extrapolate well. First, we quantify the observation that ReLU MLPs quickly converge to linear functions along any direction from the origin, which implies that ReLU MLPs do not extrapolate most nonlinear functions. But, they can provably learn a linear target function when the training distribution is sufficiently "diverse". Second, in connection to analyzing successes and limitations of GNNs, these results suggest a hypothesis for which we provide theoretical and empirical evidence: the success of GNNs in extrapolating algorithmic tasks to new data (e.g., larger graphs or edge weights) relies on encoding taskspecific nonlinearities in the architecture or features.
READ FULL TEXT
Comments
There are no comments yet.