Accurate, Efficient and Scalable Training of Graph Neural Networks

10/05/2020
by   Hanqing Zeng, et al.
0

Graph Neural Networks (GNNs) are powerful deep learning models to generate node embeddings on graphs. When applying deep GNNs on large graphs, it is still challenging to perform training in an efficient and scalable way. We propose a novel parallel training framework. Through sampling small subgraphs as minibatches, we reduce training workload by orders of magnitude compared with state-of-the-art minibatch methods. We then parallelize the key computation steps on tightly-coupled shared memory systems. For graph sampling, we exploit parallelism within and across sampler instances, and propose an efficient data structure supporting concurrent accesses from samplers. The parallel sampler theoretically achieves near-linear speedup with respect to number of processing units. For feature propagation within subgraphs, we improve cache utilization and reduce DRAM traffic by data partitioning. Our partitioning is a 2-approximation strategy for minimizing the communication cost compared to the optimal. We further develop a runtime scheduler to reorder the training operations and adjust the minibatch subgraphs to improve parallel performance. Finally, we generalize the above parallelization strategies to support multiple types of GNN models and graph samplers. The proposed training outperforms the state-of-the-art in scalability, efficiency and accuracy simultaneously. On a 40-core Xeon platform, we achieve 60x speedup (with AVX) in the sampling step and 20x speedup in the feature propagation step, compared to the serial implementation. Our algorithm enables fast training of deeper GNNs, as demonstrated by orders of magnitude speedup compared to the Tensorflow implementation. We open-source our code at https://github.com/GraphSAINT/GraphSAINT.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/28/2018

Accurate, Efficient and Scalable Graph Embedding

The Graph Convolutional Network (GCN) model and its variants are powerfu...
research
07/14/2022

Subgraph Frequency Distribution Estimation using Graph Neural Networks

Small subgraphs (graphlets) are important features to describe fundament...
research
06/29/2019

Planting Trees for scalable and efficient Canonical Hub Labeling

Point-to-Point Shortest Distance (PPSD) query is a crucial primitive in ...
research
02/17/2021

Fast Graph Learning with Unique Optimal Solutions

Graph Representation Learning (GRL) has been advancing at an unprecedent...
research
10/17/2020

RECEIPT: REfine CoarsE-grained IndePendent Tasks for Parallel Tip decomposition of Bipartite Graphs

Tip decomposition is a crucial kernel for mining dense subgraphs in bipa...
research
10/19/2018

Towards Efficient Large-Scale Graph Neural Network Computing

Recent deep learning models have moved beyond low-dimensional regular gr...
research
05/27/2022

Learning to Solve Combinatorial Graph Partitioning Problems via Efficient Exploration

From logistics to the natural sciences, combinatorial optimisation on gr...

Please sign up or login with your details

Forgot password? Click here to reset