PyTorch implementation of graph ConvNets, NIPS’16
In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words' embedding, represented by graphs. We present a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs.READ FULL TEXT VIEW PDF
PyTorch implementation of graph ConvNets, NIPS’16
Chainer implementation of 'Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering' (https://arxiv.org/abs/1606.09375)
Convolutional neural networks pro:LeCunBottouBengioHaffner98MNIST offer an efficient architecture to extract highly meaningful statistical patterns in large-scale and high-dimensional datasets. The ability of CNNs to learn local stationary structures and compose them to form multi-scale hierarchical patterns has led to breakthroughs in image, video, and sound recognition tasks art:LeCunBengioHinton15DL . Precisely, CNNs extract the local stationarity property of the input data or signals by revealing local features that are shared across the data domain. These similar features are identified with localized convolutional filters or kernels, which are learned from the data. Convolutional filters are shift- or translation-invariant filters, meaning they are able to recognize identical features independently of their spatial locations. Localized kernels or compactly supported filters refer to filters that extract local features independently of the input data size, with a support size that can be much smaller than the input size.
User data on social networks, gene data on biological regulatory networks, log data on telecommunication networks, or text documents on word embeddings are important examples of data lying on irregular or non-Euclidean domains that can be structured with graphs, which are universal representations of heterogeneous pairwise relationships. Graphs can encode complex geometric structures and can be studied with strong mathematical tools such as spectral graph theory book:Chung97Spectral .
A generalization of CNNs to graphs is not straightforward as the convolution and pooling operators are only defined for regular grids. This makes this extension challenging, both theoretically and implementation-wise. The major bottleneck of generalizing CNNs to graphs, and one of the primary goals of this work, is the definition of localized graph filters which are efficient to evaluate and learn. Precisely, the main contributions of this work are summarized below.
Spectral formulation. A spectral graph theoretical formulation of CNNs on graphs built on established tools in graph signal processing (GSP). art:ShumanNarangFrossardOrtegaVandergheynst13ReviewSPG .
Strictly localized filters. Enhancing art:BrunaZarembaSzlamLeCun13DLgraphs , the proposed spectral filters are provable to be strictly localized in a ball of radius , i.e. hops from the central vertex.
Low computational complexity. The evaluation complexity of our filters is linear w.r.t. the filters support’s size and the number of edges . Importantly, as most real-world graphs are highly sparse, we have and for the widespread -nearest neighbor (NN) graphs, leading to a linear complexity w.r.t the input data size
. Moreover, this method avoids the Fourier basis altogether, thus the expensive eigenvalue decomposition (EVD) necessary to compute it as well as the need to store the basis, a matrix of size. That is especially relevant when working with limited GPU memory. Besides the data, our method only requires to store the Laplacian, a sparse matrix of non-zero values.
Efficient pooling. We propose an efficient pooling strategy on graphs which, after a rearrangement of the vertices as a binary tree structure, is analog to pooling of 1D signals.
Experimental results. We present multiple experiments that ultimately show that our formulation is (i) a useful model, (ii) computationally efficient and (iii) superior both in accuracy and complexity to the pioneer spectral graph CNN introduced in art:BrunaZarembaSzlamLeCun13DLgraphs
. We also show that our graph formulation performs similarly to a classical CNNs on MNIST and study the impact of various graph constructions on performance. The TensorFlowabadi_tensorflow_2016 code to reproduce our results and apply the model to other data is available as an open-source software.111https://github.com/mdeff/cnn_graph
Generalizing CNNs to graphs requires three fundamental steps: (i) the design of localized convolutional filters on graphs, (ii) a graph coarsening procedure that groups together similar vertices and (iii) a graph pooling operation that trades spatial resolution for higher filter resolution.
There are two strategies to define convolutional filters; either from a spatial approach or from a spectral approach. By construction, spatial approaches provide filter localization via the finite size of the kernel. However, although graph convolution in the spatial domain is conceivable, it faces the challenge of matching local neighborhoods, as pointed out in art:BrunaZarembaSzlamLeCun13DLgraphs . Consequently, there is no unique mathematical definition of translation on graphs from a spatial perspective. On the other side, a spectral approach provides a well-defined localization operator on graphs via convolutions with a Kronecker delta implemented in the spectral domain art:ShumanNarangFrossardOrtegaVandergheynst13ReviewSPG . The convolution theorem book:Mallat99wavelets
defines convolutions as linear operators that diagonalize in the Fourier basis (represented by the eigenvectors of the Laplacian operator). However, a filter defined in the spectral domain is not naturally localized and translations are costly due to themultiplication with the graph Fourier basis. Both limitations can however be overcome with a special choice of filter parametrization.
We are interested in processing signals defined on undirected and connected graphs , where is a finite set of vertices, is a set of edges and is a weighted adjacency matrix encoding the connection weight between two vertices. A signal
defined on the nodes of the graph may be regarded as a vectorwhere is the value of at the node. An essential operator in spectral graph analysis is the graph Laplacian book:Chung97Spectral , which combinatorial definition is where is the diagonal degree matrix with , and normalized definition is where
is the identity matrix. Asis a real symmetric positive semidefinite matrix, it has a complete set of orthonormal eigenvectors , known as the graph Fourier modes, and their associated ordered real nonnegative eigenvalues , identified as the frequencies of the graph. The Laplacian is indeed diagonalized by the Fourier basis such that where
. The graph Fourier transform of a signalis then defined as , and its inverse as art:ShumanNarangFrossardOrtegaVandergheynst13ReviewSPG . As on Euclidean spaces, that transform enables the formulation of fundamental operations such as filtering.
As we cannot express a meaningful translation operator in the vertex domain, the convolution operator on graph is defined in the Fourier domain such that , where is the element-wise Hadamard product. It follows that a signal is filtered by as
A non-parametric filter, i.e. a filter whose parameters are all free, would be defined as
where the parameter is a vector of Fourier coefficients.
There are however two limitations with non-parametric filters: (i) they are not localized in space and (ii) their learning complexity is in , the dimensionality of the data. These issues can be overcome with the use of a polynomial filter
where the parameter is a vector of polynomial coefficients. The value at vertex of the filter centered at vertex is given by , where the kernel is localized via a convolution with a Kronecker delta function . By (art:HammondVandergheynstGribonval11GraphWav, , Lemma 5.2), implies , where is the shortest path distance, i.e. the minimum number of edges connecting two vertices on the graph. Consequently, spectral filters represented by -order polynomials of the Laplacian are exactly -localized. Besides, their learning complexity is , the support size of the filter, and thus the same complexity as classical CNNs.
While we have shown how to learn localized filters with parameters, the cost to filter a signal as is still high with operations because of the multiplication with the Fourier basis . A solution to this problem is to parametrize as a polynomial function that can be computed recursively from , as multiplications by a sparse costs . One such polynomial, traditionally used in GSP to approximate kernels (like wavelets), is the Chebyshev expansion art:HammondVandergheynstGribonval11GraphWav . Another option, the Lanczos algorithm art:SusnjaraPerraudinKressnerVandergheynst15Lanczos , which constructs an orthonormal basis of the Krylov subspace , seems attractive because of the coefficients’ independence. It is however more convoluted and thus left as a future work.
Recall that the Chebyshev polynomial of order may be computed by the stable recurrence relation with and . These polynomials form an orthogonal basis for , the Hilbert space of square integrable functions with respect to the measure . A filter can thus be parametrized as the truncated expansion
of order , where the parameter is a vector of Chebyshev coefficients and is the Chebyshev polynomial of order evaluated at , a diagonal matrix of scaled eigenvalues that lie in . The filtering operation can then be written as , where is the Chebyshev polynomial of order evaluated at the scaled Laplacian . Denoting , we can use the recurrence relation to compute with and . The entire filtering operation then costs operations.
The output feature map of the sample is given by
where the are the input feature maps and the vectors of Chebyshev coefficients
are the layer’s trainable parameters. When training multiple convolutional layers with the backpropagation algorithm, one needs the two gradients
where is the loss energy over a mini-batch of samples. Each of the above three computations boils down to sparse matrix-vector multiplications and one dense matrix-vector multiplication for a cost of
operations. These can be efficiently computed on parallel architectures by leveraging tensor operations. Eventually,only needs to be computed once.
The pooling operation requires meaningful neighborhoods on graphs, where similar vertices are clustered together. Doing this for multiple layers is equivalent to a multi-scale clustering of the graph that preserves local geometric structures. It is however known that graph clustering is NP-hard art:BuiJonesGraphPartNPhard
and that approximations must be used. While there exist many clustering techniques, e.g. the popular spectral clusteringart:VonLuxburg07Tutorial , we are most interested in multilevel clustering algorithms where each level produces a coarser graph which corresponds to the data domain seen at a different resolution. Moreover, clustering techniques that reduce the size of the graph by a factor two at each level offers a precise control on the coarsening and pooling size. In this work, we make use of the coarsening phase of the Graclus multilevel clustering algorithm art:DhillonGuanKulis07Graclus , which has been shown to be extremely efficient at clustering a large variety of graphs. Algebraic multigrid techniques on graphs art:RonSafroBrandt11MultigridGraph and the Kron reduction art:ShumanFarajiVandergheynst16PyramTrans are two methods worth exploring in future works.
Graclus art:DhillonGuanKulis07Graclus , built on Metis art:KarypisKumar98Metis , uses a greedy algorithm to compute successive coarser versions of a given graph and is able to minimize several popular spectral clustering objectives, from which we chose the normalized cut art:ShiMalik00NCut . Graclus’ greedy rule consists, at each coarsening level, in picking an unmarked vertex and matching it with one of its unmarked neighbors that maximizes the local normalized cut . The two matched vertices are then marked and the coarsened weights are set as the sum of their weights. The matching is repeated until all nodes have been explored. This is an very fast coarsening scheme which divides the number of nodes by approximately two (there may exist a few singletons, non-matched nodes) from one level to the next coarser level.
Pooling operations are carried out many times and must be efficient. After coarsening, the vertices of the input graph and its coarsened versions are not arranged in any meaningful way. Hence, a direct application of the pooling operation would need a table to store all matched vertices. That would result in a memory inefficient, slow, and hardly parallelizable implementation. It is however possible to arrange the vertices such that a graph pooling operation becomes as efficient as a 1D pooling. We proceed in two steps: (i) create a balanced binary tree and (ii) rearrange the vertices. After coarsening, each node has either two children, if it was matched at the finer level, or one, if it was not, i.e the node was a singleton. From the coarsest to finest level, fake nodes, i.e. disconnected nodes, are added to pair with the singletons such that each node has two children. This structure is a balanced binary tree: (i) regular nodes (and singletons) either have two regular nodes (e.g. level 1 vertex 0 in Figure 2
) or (ii) one singleton and a fake node as children (e.g. level 2 vertex 0), and (iii) fake nodes always have two fake nodes as children (e.g. level 1 vertex 1). Input signals are initialized with a neutral value at the fake nodes, e.g. 0 when using a ReLU activation with max pooling. Because these nodes are disconnected, filtering does not impact the initial neutral value. While those fake nodes do artificially increase the dimensionality thus the computational cost, we found that, in practice, the number of singletons left by Graclus is quite low. Arbitrarily ordering the nodes at the coarsest level, then propagating this ordering to the finest levels, i.e. nodehas nodes and as children, produces a regular ordering in the finest level. Regular in the sense that adjacent nodes are hierarchically merged at coarser levels. Pooling such a rearranged graph signal is analog to pooling a regular 1D signal. Figure 2 shows an example of the whole process. This regular arrangement makes the operation very efficient and satisfies parallel architectures such as GPUs as memory accesses are local, i.e. matched nodes do not have to be fetched.
The emerging field of GSP aims at bridging the gap between signal processing and spectral graph theory book:Chung97Spectral ; art:BelkinNiyogi05LaplaBeltrami ; art:VonLuxburg07Tutorial , a blend between graph theory and harmonic analysis. A goal is to generalize fundamental analysis operations for signals from regular grids to irregular structures embodied by graphs. We refer the reader to art:ShumanNarangFrossardOrtegaVandergheynst13ReviewSPG for an introduction of the field. Standard operations on grids such as convolution, translation, filtering, dilatation, modulation or downsampling do not extend directly to graphs and thus require new mathematical definitions while keeping the original intuitive concepts. In this context, the authors of art:HammondVandergheynstGribonval11GraphWav ; art:CoifmanLafon06DifMap ; pro:GavishNadlerCoifman10GraphHaar revisited the construction of wavelet operators on graphs and techniques to perform mutli-scale pyramid transforms on graphs were proposed in art:ShumanFarajiVandergheynst16PyramTrans ; art:RamEladCohen11TreeWavelets . The works of pro:TsitsveroBarbarossa15Uncert ; pro:PasdeloupAlamiGriponRabbat15Uncert ; art:PerraudinRicaudShumanVandergheynst16Uncert redefined uncertainty principles on graphs and showed that while intuitive concepts may be lost, enhanced localization principles can be derived.
The Graph Neural Network framework scarselli_gnn_2009 , simplified in li_ggsnn_2015 , was designed to embed each node in an Euclidean space with a RNN and use those embeddings as features for classification or regression of nodes or graphs. By setting their transition function as a simple diffusion instead of a neural net with a recursive relation, their state vector becomes . Their point-wise output function can further be set as instead of another neural net. The Chebyshev polynomials of degree can then be obtained with a -layer GNN, to be followed by a non-linear layer and a graph pooling operation. Our model can thus be interpreted as multiple layers of diffusions and node-local operations.
The works of pro:GregorLeCun10LRF ; pro:CoatesNg11LRF introduced the concept of constructing a local receptive field to reduce the number of learned parameters. The idea is to group together features based upon a measure of similarity such as to select a limited number of connections between two successive layers. While this model reduces the number of parameters by exploiting the locality assumption, it did not attempt to exploit any stationarity property, i.e. no weight-sharing strategy. The authors of art:BrunaZarembaSzlamLeCun13DLgraphs used this idea for their spatial formulation of graph CNNs. They use a weighted graph to define the local neighborhood and compute a multiscale clustering of the graph for the pooling operation. Inducing weight sharing in a spatial construction is however challenging, as it requires to select and order the neighborhoods when a problem-specific ordering (spatial, temporal, or otherwise) is missing.
A spatial generalization of CNNs to 3D-meshes, a class of smooth low-dimensional non-Euclidean spaces, was proposed in masci2015geodesic . The authors used geodesic polar coordinates to define the convolution on mesh patches, and formulated a deep learning architecture which allows comparison across different manifolds. They obtained state-of-the-art results for 3D shape recognition.
The first spectral formulation of a graph CNN, proposed in art:BrunaZarembaSzlamLeCun13DLgraphs , defines a filter as
where is the cubic B-spline basis and the parameter is a vector of control points. They later proposed a strategy to learn the graph structure from the data and applied the model to image recognition, text categorization and bioinformatics art:HenaffBrunaLeCun15DLgraphs . This approach does however not scale up due to the necessary multiplications by the graph Fourier basis . Despite the cost of computing this matrix, which requires an EVD on the graph Laplacian, the dominant cost is the need to multiply the data by this matrix twice (forward and inverse Fourier transforms) at a cost of operations per forward and backward pass, a computational bottleneck already identified by the authors. Besides, as they rely on smoothness in the Fourier domain, via the spline parametrization, to bring localization in the vertex domain, their model does not provide a precise control over the local support of their kernels, which is essential to learn localized filters. Our technique leverages on this work, and we showed how to overcome these limitations and beyond.
In the sequel, we refer to the non-parametric and non-localized filters (2) as Non-Param, the filters (7) proposed in art:BrunaZarembaSzlamLeCun13DLgraphs as Spline and the proposed filters (4) as Chebyshev. We always use the Graclus coarsening algorithm introduced in Section 2.2 rather than the simple agglomerative method of art:BrunaZarembaSzlamLeCun13DLgraphs . Our motivation is to compare the learned filters, not the coarsening algorithms.
We use the following notation when describing network architectures: FC denotes a fully connected layer with hidden units, P
denotes a (graph or classical) pooling layer of size and stride, GC and C denote a (graph) convolutional layer with feature maps. All FC, C and GC layers are followed by a ReLU activation . The final layer is always a softmax regression and the loss energy is the cross-entropy with an regularization on the weights of all FC layers. Mini-batches are of size .
|Proposed graph CNN||GC32-P4-GC64-P4-FC512||99.14|
To validate our model, we applied it to the Euclidean case on the benchmark MNIST classification problem pro:LeCunBottouBengioHaffner98MNIST , a dataset of 70,000 digits represented on a 2D grid of size . For our graph model, we construct an -NN graph of the 2D grid which produces a graph of nodes ( pixels and 192 fake nodes as explained in Section 2.3) and edges. Following standard practice, the weights of a -NN similarity graph (between features) are computed as
where is the 2D coordinate of pixel .
This is an important sanity check for our model, which must be able to extract features on any graph, including the regular 2D grid. Table 1
shows the ability of our model to achieve a performance very close to a classical CNN with the same architecture. The gap in performance may be explained by the isotropic nature of the spectral filters, i.e. the fact that edges in a general graph do not possess an orientation (like up, down, right and left for pixels on a 2D grid). Whether this is a limitation or an advantage depends on the problem and should be verified, as for any invariance. Moreover, rotational invariance has been sought: (i) many data augmentation schemes have used rotated versions of images and (ii) models have been developed to learn this invariance, like the Spatial Transformer Networksjaderberg2015spatial . Other explanations are the lack of experience on architecture design and the need to investigate better suited optimization or initialization strategies.
The LeNet-5-like network architecture and the following hyper-parameters are borrowed from the TensorFlow MNIST tutorial222 https://www.tensorflow.org/versions/r0.8/tutorials/mnist/pros
: dropout probability of 0.5, regularization weight of, initial learning rate of 0.03, learning rate decay of 0.95, momentum of 0.9. Filters are of size and graph filters have the same support of
. All models were trained for 20 epochs.
To demonstrate the versatility of our model to work with graphs generated from unstructured data, we applied our technique to the text categorization problem on the 20NEWS dataset which consists of 18,846 (11,314 for training and 7,532 for testing) text documents associated with 20 classes art:Joachims9620NEWS . We extracted the 10,000 most common words from the 93,953 unique words in this corpus. Each document is represented using the bag-of-words model, normalized across words. To test our model, we constructed a 16-NN graph with (8) where is the word2vec embedding pro:MikolovChenCorradoDean13word2vec of word , which produced a graph of nodes and edges. All models were trained for 20 epochs by the Adam optimizer art:KingmaBa14AdamOpt with an initial learning rate of 0.001. The architecture is GC32 with support . Table 2
shows decent performances: while the proposed model does not outperform the multinomial naive Bayes classifier on this small dataset, it does defeat fully connected networks, which require much more parameters.
|Dataset||Architecture||Non-Param (2)||Spline (7) art:BrunaZarembaSzlamLeCun13DLgraphs||Chebyshev (4)|
|Proposed graph CNN||GC32-P4-GC64-P4-FC512||1600||200||8.00x|
Table 3 reports that the proposed parametrization (4) outperforms (7) from art:BrunaZarembaSzlamLeCun13DLgraphs as well as non-parametric filters (2) which are not localized and require parameters. Moreover, Figure 3 gives a sense of how the validation accuracy and the loss converges w.r.t. the filter definitions.
Figure 2 validates the low computational complexity of our model which scales as while art:BrunaZarembaSzlamLeCun13DLgraphs scales as . The measured runtime is the total training time divided by the number of gradient steps. Table 4 shows a similar speedup as classical CNNs when moving to GPUs. This exemplifies the parallelization opportunity offered by our model, who relies solely on matrix multiplications. Those are efficiently implemented by cuBLAS, the linear algebra routines provided by NVIDIA.
|Architecture||8-NN on 2D Euclidean grid||random|
For any graph CNN to be successful, the statistical assumptions of locality, stationarity, and compositionality regarding the data must be fulfilled on the graph where the data resides. Therefore, the learned filters’ quality and thus the classification performance critically depends on the quality of the graph. For data lying on Euclidean space, experiments in Section 4.1 show that a simple -NN graph of the grid is good enough to recover almost exactly the performance of standard CNNs. We also noticed that the value of does not have a strong influence on the results. We can witness the importance of a graph satisfying the data assumptions by comparing its performance with a random graph. Table 5 reports a large drop of accuracy when using a random graph, that is when the data structure is lost and the convolutional layers are not useful anymore to extract meaningful features.
While images can be structured by a grid graph, a feature graph has to be built for text documents represented as bag-of-words. We investigate here three ways to represent a word : the simplest option is to represent each word as its corresponding column in the bag-of-words matrix while, another approach is to learn an embedding for each word with word2vec pro:MikolovChenCorradoDean13word2vec or to use the pre-learned embeddings provided by the authors. For larger datasets, an approximate nearest neighbors algorithm may be required, which is the reason we tried LSHForest pro:BawaCondieGanesan05LSHForest on the learned word2vec embeddings. Table 6 reports classification results which highlight the importance of a well constructed graph.
In this paper, we have introduced the mathematical and computational foundations of an efficient generalization of CNNs to graphs using tools from GSP. Experiments have shown the ability of the model to extract local and stationary features through graph convolutional layers. Compared with the first work on spectral graph CNNs introduced in art:BrunaZarembaSzlamLeCun13DLgraphs , our model provides a strict control over the local support of filters, is computationally more efficient by avoiding an explicit use of the Graph Fourier basis, and experimentally shows a better test accuracy. Besides, we addressed the three concerns raised by art:HenaffBrunaLeCun15DLgraphs : (i) we introduced a model whose computational complexity is linear with the dimensionality of the data, (ii) we confirmed that the quality of the input graph is of paramount importance, (iii) we showed that the statistical assumptions of local stationarity and compositionality made by the model are verified for text documents as long as the graph is well constructed.
Future works will investigate two directions. On one hand, we will enhance the proposed framework with newly developed tools in GSP. On the other hand, we will explore applications of this generic model to important fields where the data naturally lies on graphs, which may then incorporate external information about the structure of the data rather than artificially created graphs which quality may vary as seen in the experiments. Another natural and future approach, pioneered in art:HenaffBrunaLeCun15DLgraphs , would be to alternate the learning of the CNN parameters and the graph.
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems.2016.
Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 37–45, 2015.
On the Degrees of Freedom of Signals on Graphs.In Signal Processing Conference (EUSIPCO), pages 1506–1510, 2015.