DeepAI AI Chat
Log In Sign Up

ABC: Aggregation before Communication, a Communication Reduction Framework for Distributed Graph Neural Network Training and Effective Partition

by   Junwei Su, et al.
The University of Hong Kong

Graph Neural Networks(GNNs) are a family of neural models tailored for graph-structure data and have shown superior performance in learning representations for graph-structured data. However, training GNNs on large graphs remains challenging and a promising direction is distributed GNN training, which is to partition the input graph and distribute the workload across multiple machines. The key bottleneck of the existing distributed GNNs training framework is the across-machine communication induced by the dependency on the graph data and aggregation operator of GNNs. In this paper, we study the communication complexity during distributed GNNs training and propose a simple lossless communication reduction method, termed the Aggregation before Communication (ABC) method. ABC method exploits the permutation-invariant property of the GNNs layer and leads to a paradigm where vertex-cut is proved to admit a superior communication performance than the currently popular paradigm (edge-cut). In addition, we show that the new partition paradigm is particularly ideal in the case of dynamic graphs where it is infeasible to control the edge placement due to the unknown stochastic of the graph-changing process.


page 1

page 2

page 3

page 4


Distributed Graph Neural Network Training: A Survey

Graph neural networks (GNNs) are a type of deep learning models that lea...

Scalable Graph Neural Network Training: The Case for Sampling

Graph Neural Networks (GNNs) are a new and increasingly popular family o...

RETEXO: Scalable Neural Network Training over Distributed Graphs

Graph neural networks offer a promising approach to supervised learning ...

Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

Despite the recent success of Graph Neural Networks (GNNs), training GNN...

Discriminability of Single-Layer Graph Neural Networks

Network data can be conveniently modeled as a graph signal, where data v...

Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks

In this paper, we study a novel meta aggregation scheme towards binarizi...

GNN at the Edge: Cost-Efficient Graph Neural Network Processing over Distributed Edge Servers

Edge intelligence has arisen as a promising computing paradigm for suppo...