Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

11/17/2019
by   Difan Zou, et al.
10

Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a large graph, however, is still a challenge. Original full-batch GCN training requires calculating the representation of all the nodes in the graph per GCN layer, which brings in high computation and memory costs. To alleviate this issue, several sampling-based methods have been proposed to train GCNs on a subset of nodes. Among them, the node-wise neighbor-sampling method recursively samples a fixed number of neighbor nodes, and thus its computation cost suffers from exponential growing neighbor size; while the layer-wise importance-sampling method discards the neighbor-dependent constraints, and thus the nodes sampled across layer suffer from sparse connection problem. To deal with the above two problems, we propose a new effective sampling algorithm called LAyer-Dependent ImportancE Sampling (LADIES). Based on the sampled nodes in the upper layer, LADIES selects their neighborhood nodes, constructs a bipartite subgraph and computes the importance probability accordingly. Then, it samples a fixed number of nodes by the calculated probability, and recursively conducts such procedure per layer to construct the whole computation graph. We prove theoretically and experimentally, that our proposed sampling algorithm outperforms the previous sampling methods in terms of both time and memory costs. Furthermore, LADIES is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/10/2019

GraphSAINT: Graph Sampling Based Inductive Learning Method

Graph Convolutional Networks (GCNs) are powerful models for learning rep...
research
09/14/2018

Adaptive Sampling Towards Fast Graph Representation Learning

Graph Convolutional Networks (GCNs) have become a crucial tool on learni...
research
10/24/2022

(LA)yer-neigh(BOR) Sampling: Defusing Neighborhood Explosion in GNNs

Graph Neural Networks have recently received a significant attention, ho...
research
01/19/2021

Communication-Efficient Sampling for Distributed Training of Graph Convolutional Networks

Training Graph Convolutional Networks (GCNs) is expensive as it needs to...
research
03/03/2021

On the Importance of Sampling in Learning Graph Convolutional Networks

Graph Convolutional Networks (GCNs) have achieved impressive empirical a...
research
09/25/2022

Flattened Graph Convolutional Networks For Recommendation

Graph Convolutional Networks (GCNs) and their variants have achieved sig...
research
01/30/2018

FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling

The graph convolutional networks (GCN) recently proposed by Kipf and Wel...

Please sign up or login with your details

Forgot password? Click here to reset