Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication

by   Ajay Jaiswal, et al.

Graphs are omnipresent and GNNs are a powerful family of neural networks for learning over graphs. Despite their popularity, scaling GNNs either by deepening or widening suffers from prevalent issues of unhealthy gradients, over-smoothening, information squashing, which often lead to sub-standard performance. In this work, we are interested in exploring a principled way to scale GNNs capacity without deepening or widening, which can improve its performance across multiple small and large graphs. Motivated by the recent intriguing phenomenon of model soups, which suggest that fine-tuned weights of multiple large-language pre-trained models can be merged to a better minima, we argue to exploit the fundamentals of model soups to mitigate the aforementioned issues of memory bottleneck and trainability during GNNs scaling. More specifically, we propose not to deepen or widen current GNNs, but instead present a data-centric perspective of model soups tailored for GNNs, i.e., to build powerful GNNs by dividing giant graph data to build independently and parallelly trained multiple comparatively weaker GNNs without any intermediate communication, and combining their strength using a greedy interpolation soup procedure to achieve state-of-the-art performance. Moreover, we provide a wide variety of model soup preparation techniques by leveraging state-of-the-art graph sampling and graph partitioning approaches that can handle large graph data structures. Our extensive experiments across many real-world small and large graphs, illustrate the effectiveness of our approach and point towards a promising orthogonal direction for GNN scaling. Codes are available at: <>.


page 1

page 2

page 3

page 4


A Unified Lottery Ticket Hypothesis for Graph Neural Networks

With graphs rapidly growing in size and deeper graph neural networks (GN...

Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study

Training deep graph neural networks (GNNs) is notoriously hard. Besides ...

Memory-Based Graph Networks

Graph neural networks (GNNs) are a class of deep models that operate on ...

Revisiting Embeddings for Graph Neural Networks

Current graph representation learning techniques use Graph Neural Networ...

Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling

Graph neural networks (GNNs) have been widely applied to learning over g...

The Split Matters: Flat Minima Methods for Improving the Performance of GNNs

When training a Neural Network, it is optimized using the available trai...

Combining Label Propagation and Simple Models Out-performs Graph Neural Networks

Graph Neural Networks (GNNs) are the predominant technique for learning ...

Please sign up or login with your details

Forgot password? Click here to reset