Communication-Efficient Sampling for Distributed Training of Graph Convolutional Networks

01/19/2021
by   Peng Jiang, et al.
0

Training Graph Convolutional Networks (GCNs) is expensive as it needs to aggregate data recursively from neighboring nodes. To reduce the computation overhead, previous works have proposed various neighbor sampling methods that estimate the aggregation result based on a small number of sampled neighbors. Although these methods have successfully accelerated the training, they mainly focus on the single-machine setting. As real-world graphs are large, training GCNs in distributed systems is desirable. However, we found that the existing neighbor sampling methods do not work well in a distributed setting. Specifically, a naive implementation may incur a huge amount of communication of feature vectors among different machines. To address this problem, we propose a communication-efficient neighbor sampling method in this work. Our main idea is to assign higher sampling probabilities to the local nodes so that remote nodes are accessed less frequently. We present an algorithm that determines the local sampling probabilities and makes sure our skewed neighbor sampling does not affect much the convergence of the training. Our experiments with node classification benchmarks show that our method significantly reduces the communication overhead for distributed GCN training with little accuracy loss.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/21/2022

BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling

Graph Convolutional Networks (GCNs) have emerged as the state-of-the-art...
research
11/17/2019

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

Graph convolutional networks (GCNs) have recently received wide attentio...
research
10/29/2017

Stochastic Training of Graph Convolutional Networks

Graph convolutional networks (GCNs) are powerful deep neural networks fo...
research
12/09/2020

Distributed Training of Graph Convolutional Networks using Subgraph Approximation

Modern machine learning techniques are successfully being adapted to dat...
research
03/20/2022

PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication

Graph Convolutional Networks (GCNs) is the state-of-the-art method for l...
research
12/18/2021

Time-Aware Neighbor Sampling for Temporal Graph Networks

We present a new neighbor sampling method on temporal graphs. In a tempo...
research
10/24/2019

Communication-Efficient (Weighted) Reservoir Sampling

We consider communication-efficient weighted and unweighted (uniform) ra...

Please sign up or login with your details

Forgot password? Click here to reset