GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

by   Tianle Cai, et al.

Normalization plays an important role in the optimization of deep neural networks. While there are standard normalization methods in computer vision and natural language processing, there is limited understanding of how to effectively normalize neural networks for graph representation learning. In this paper, we propose a principled normalization method, Graph Normalization (GraphNorm), where the key idea is to normalize the feature values across all nodes for each individual graph with a learnable shift. Theoretically, we show that GraphNorm serves as a preconditioner that smooths the distribution of the graph aggregation's spectrum, leading to faster optimization. Such an improvement cannot be well obtained if we use currently popular normalization methods, such as BatchNorm, which normalizes the nodes in a batch rather than in individual graphs, due to heavy batch noises. Moreover, we show that for some highly regular graphs, the mean of the feature values contains graph structural information, and directly subtracting the mean may lead to an expressiveness degradation. The learnable shift in GraphNorm enables the model to learn to avoid such degradation for those cases. Empirically, Graph neural networks (GNNs) with GraphNorm converge much faster compared to GNNs with other normalization methods, e.g., BatchNorm. GraphNorm also improves generalization of GNNs, achieving better performance on graph classification benchmarks.


page 1

page 2

page 3

page 4


Graph Neural Networks with Learnable Structural and Positional Representations

Graph neural networks (GNNs) have become the standard learning architect...

Demystifying Structural Disparity in Graph Neural Networks: Can One Size Fit All?

Recent studies on Graph Neural Networks(GNNs) provide both empirical and...

ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization

Graph Neural Networks (GNNs) have attracted much attention due to their ...

FairNorm: Fair and Fast Graph Neural Network Training

Graph neural networks (GNNs) have been demonstrated to achieve state-of-...

Learning Graph Normalization for Graph Neural Networks

Graph Neural Networks (GNNs) have attracted considerable attention and h...

How Neural Architectures Affect Deep Learning for Communication Networks?

In recent years, there has been a surge in applying deep learning to var...

Graph Representation Learning via Hard and Channel-Wise Attention Networks

Attention operators have been widely applied in various fields, includin...

Please sign up or login with your details

Forgot password? Click here to reset