GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

09/07/2020
by   Tianle Cai, et al.
1

Normalization plays an important role in the optimization of deep neural networks. While there are standard normalization methods in computer vision and natural language processing, there is limited understanding of how to effectively normalize neural networks for graph representation learning. In this paper, we propose a principled normalization method, Graph Normalization (GraphNorm), where the key idea is to normalize the feature values across all nodes for each individual graph with a learnable shift. Theoretically, we show that GraphNorm serves as a preconditioner that smooths the distribution of the graph aggregation's spectrum, leading to faster optimization. Such an improvement cannot be well obtained if we use currently popular normalization methods, such as BatchNorm, which normalizes the nodes in a batch rather than in individual graphs, due to heavy batch noises. Moreover, we show that for some highly regular graphs, the mean of the feature values contains graph structural information, and directly subtracting the mean may lead to an expressiveness degradation. The learnable shift in GraphNorm enables the model to learn to avoid such degradation for those cases. Empirically, Graph neural networks (GNNs) with GraphNorm converge much faster compared to GNNs with other normalization methods, e.g., BatchNorm. GraphNorm also improves generalization of GNNs, achieving better performance on graph classification benchmarks.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/15/2021

Graph Neural Networks with Learnable Structural and Positional Representations

Graph neural networks (GNNs) have become the standard learning architect...
06/16/2022

ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization

Graph Neural Networks (GNNs) have attracted much attention due to their ...
05/20/2022

FairNorm: Fair and Fast Graph Neural Network Training

Graph neural networks (GNNs) have been demonstrated to achieve state-of-...
09/24/2020

Learning Graph Normalization for Graph Neural Networks

Graph Neural Networks (GNNs) have attracted considerable attention and h...
05/05/2021

Scalable Graph Neural Network Training: The Case for Sampling

Graph Neural Networks (GNNs) are a new and increasingly popular family o...
11/03/2021

How Neural Architectures Affect Deep Learning for Communication Networks?

In recent years, there has been a surge in applying deep learning to var...
03/29/2021

Regular Polytope Networks

Neural networks are widely used as a model for classification in a large...