DeepAI
Log In Sign Up

Wide Graph Neural Networks: Aggregation Provably Leads to Exponentially Trainability Loss

03/03/2021
by   Wei Huang, et al.
7

Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data. However, it is well known that deep GCNs will suffer from over-smoothing problem, where node representations tend to be indistinguishable as we stack up more layers. Although extensive research has confirmed this prevailing understanding, few theoretical analyses have been conducted to study the expressivity and trainability of deep GCNs. In this work, we demonstrate these characterizations by studying the Gaussian Process Kernel (GPK) and Graph Neural Tangent Kernel (GNTK) of an infinitely-wide GCN, corresponding to the analysis on expressivity and trainability, respectively. We first prove the expressivity of infinitely-wide GCNs decaying at an exponential rate by applying the mean-field theory on GPK. Besides, we formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate. Additionally, we extend our theoretical framework to analyze residual connection-resemble techniques. We found that these techniques can mildly mitigate exponential decay, but they failed to overcome it fundamentally. Finally, all theoretical results in this work are corroborated experimentally on a variety of graph-structured datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/04/2020

Simple and Deep Graph Convolutional Networks

Graph convolutional networks (GCNs) are a powerful deep learning approac...
05/27/2019

On Asymptotic Behaviors of Graph CNNs from Dynamical Systems Perspective

Graph Convolutional Neural Networks (graph CNNs) are a promising deep le...
01/14/2021

BiGCN: A Bi-directional Low-Pass Filtering Graph Neural Network

Graph convolutional networks have achieved great success on graph-struct...
07/25/2019

The Truly Deep Graph Convolutional Networks for Node Classification

Existing Graph Convolutional Networks (GCNs) are shallow---the number of...
10/29/2022

Clenshaw Graph Neural Networks

Graph Convolutional Networks (GCNs), which use a message-passing paradig...
10/28/2021

On Provable Benefits of Depth in Training Graph Convolutional Networks

Graph Convolutional Networks (GCNs) are known to suffer from performance...