DeepAI AI Chat
Log In Sign Up

ABC: Aggregation before Communication, a Communication Reduction Framework for Distributed Graph Neural Network Training and Effective Partition

12/11/2022
by   Junwei Su, et al.
The University of Hong Kong
0

Graph Neural Networks(GNNs) are a family of neural models tailored for graph-structure data and have shown superior performance in learning representations for graph-structured data. However, training GNNs on large graphs remains challenging and a promising direction is distributed GNN training, which is to partition the input graph and distribute the workload across multiple machines. The key bottleneck of the existing distributed GNNs training framework is the across-machine communication induced by the dependency on the graph data and aggregation operator of GNNs. In this paper, we study the communication complexity during distributed GNNs training and propose a simple lossless communication reduction method, termed the Aggregation before Communication (ABC) method. ABC method exploits the permutation-invariant property of the GNNs layer and leads to a paradigm where vertex-cut is proved to admit a superior communication performance than the currently popular paradigm (edge-cut). In addition, we show that the new partition paradigm is particularly ideal in the case of dynamic graphs where it is infeasible to control the edge placement due to the unknown stochastic of the graph-changing process.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/01/2022

Distributed Graph Neural Network Training: A Survey

Graph neural networks (GNNs) are a type of deep learning models that lea...
05/05/2021

Scalable Graph Neural Network Training: The Case for Sampling

Graph Neural Networks (GNNs) are a new and increasingly popular family o...
02/25/2023

RETEXO: Scalable Neural Network Training over Distributed Graphs

Graph neural networks offer a promising approach to supervised learning ...
11/16/2021

Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

Despite the recent success of Graph Neural Networks (GNNs), training GNN...
10/17/2020

Discriminability of Single-Layer Graph Neural Networks

Network data can be conveniently modeled as a graph signal, where data v...
09/27/2021

Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks

In this paper, we study a novel meta aggregation scheme towards binarizi...
10/31/2022

GNN at the Edge: Cost-Efficient Graph Neural Network Processing over Distributed Edge Servers

Edge intelligence has arisen as a promising computing paradigm for suppo...