DeepAI AI Chat
Log In Sign Up

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

by   Shichang Zhang, et al.
Snap Inc.

Graph Neural Networks (GNNs) have recently become popular for graph machine learning and have shown great results on wide node classification tasks. Yet, GNNs are less popular for practical deployments in the industry owing to their scalability challenges incurred by data dependency. Namely, GNN inference depends on neighbor nodes multiple hops away from the target, and fetching these nodes burdens latency-constrained applications. Existing inference acceleration methods like pruning and quantization can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation (MAC) operations. However, their improvements are limited given the data dependency is not resolved. Conversely, multi-layer perceptrons (MLPs) have no dependency on graph data and infer much faster than GNNs, even though they are less accurate than GNNs for node classification in general. Motivated by these complementary strengths and weaknesses, we bring GNNs and MLPs together via knowledge distillation (KD). Our work shows that the performance of MLPs can be improved by large margins with GNN KD. We call the distilled MLPs Graph-less Neural Networks (GLNNs) as they have no inference graph dependency. We show that GLNN with competitive performance infer faster than GNNs by 146X-273X and faster than other acceleration methods by 14X-27X. Meanwhile, under a production setting involving both transductive and inductive predictions across 7 datasets, GLNN accuracies improve over stand alone MLPs by 12.36 6/7 datasets. A comprehensive analysis of GLNN shows when and why GLNN can achieve competitive results to GNNs and suggests GLNN as a handy choice for latency-constrained applications.


page 1

page 2

page 3

page 4


Should Graph Neural Networks Use Features, Edges, Or Both?

Graph Neural Networks (GNNs) are the first choice for learning algorithm...

Efficient Graph Neural Network Inference at Large Scale

Graph neural networks (GNNs) have demonstrated excellent performance in ...

NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs

While Graph Neural Networks (GNNs) have demonstrated their efficacy in d...

Degree-Quant: Quantization-Aware Training for Graph Neural Networks

Graph neural networks (GNNs) have demonstrated strong performance on a w...

Self-supervised Smoothing Graph Neural Networks

This paper studies learning node representations with GNNs for unsupervi...

A^2Q: Aggregation-Aware Quantization for Graph Neural Networks

As graph data size increases, the vast latency and memory consumption du...

Dissecting Graph Neural Networks on Graph Classification

Graph Neural Nets (GNNs) have received increasing attentions, partially ...