
A Hierarchy of Graph Neural Networks Based on Learnable Local Features
Graph neural networks (GNNs) are a powerful tool to learn representation...
read it

Learning Graph Normalization for Graph Neural Networks
Graph Neural Networks (GNNs) have attracted considerable attention and h...
read it

Scalable Graph Neural Network Training: The Case for Sampling
Graph Neural Networks (GNNs) are a new and increasingly popular family o...
read it

Discriminability of SingleLayer Graph Neural Networks
Network data can be conveniently modeled as a graph signal, where data v...
read it

A Generalization of Transformer Networks to Graphs
We propose a generalization of transformer neural network architecture f...
read it

Regular Polytope Networks
Neural networks are widely used as a model for classification in a large...
read it

Graph Information Vanishing Phenomenon inImplicit Graph Neural Networks
One of the key problems of GNNs is how to describe the importance of nei...
read it
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
Normalization plays an important role in the optimization of deep neural networks. While there are standard normalization methods in computer vision and natural language processing, there is limited understanding of how to effectively normalize neural networks for graph representation learning. In this paper, we propose a principled normalization method, Graph Normalization (GraphNorm), where the key idea is to normalize the feature values across all nodes for each individual graph with a learnable shift. Theoretically, we show that GraphNorm serves as a preconditioner that smooths the distribution of the graph aggregation's spectrum, leading to faster optimization. Such an improvement cannot be well obtained if we use currently popular normalization methods, such as BatchNorm, which normalizes the nodes in a batch rather than in individual graphs, due to heavy batch noises. Moreover, we show that for some highly regular graphs, the mean of the feature values contains graph structural information, and directly subtracting the mean may lead to an expressiveness degradation. The learnable shift in GraphNorm enables the model to learn to avoid such degradation for those cases. Empirically, Graph neural networks (GNNs) with GraphNorm converge much faster compared to GNNs with other normalization methods, e.g., BatchNorm. GraphNorm also improves generalization of GNNs, achieving better performance on graph classification benchmarks.
READ FULL TEXT
Comments
There are no comments yet.