The large amounts of data and rich interactions characteristic of complex networks such as those observed in brain imaging and social networks motivate the need for rich representations for use in learning. Convolutional neural networks offer a means of learning rich representations of data by composing convolutions, pooling, and nonlinear activation functions. However, in Bruna et al. , it is argued that the success of convolutional neural networks for images, video, and speech can be attributed to the special statistical properties of these domains (i.e. sampled on a regular grid; local similarity, translation invariance, and multi-scale structure) and does not generalize to data with arbitrary graph structure.
This motivates a generalization of the convolution to data supported on an arbitrary graph, and in Bruna et al. 
, a graph-specific convolution is defined consistent with the graph Fourier transform proposed in Shumanet al. 
. In this formulation, the eigenvectors of the graph Laplacian form the basis for linear operations as well as imparting the topology of the graph. A learning algorithm then optimizes the eigenvalues of the linear operator. Application of this learning representation in so-called graph convolutional networks has yielded state-of-the-art results in applications such as network analysis, computer graphics, and medical imaging.
However, as identified in Bronstein et al. , this procedure has limitations, and this chapter aims to address two of them. First, it is not clear how to apply these techniques to stochastic (time-varying) processes on graphs, and second, the graph signal processing approach advocated by Shuman et al.  works only for undirected graphs. These limitations severely reduce the domains in which such learning representations can apply. This chapter aims to address these two gaps by proposing a theoretical framework for designing and learning graph-specific linear operators that act on stochastic processes on directed (or undirected) graphs. Together with pooling operations and nonlinear activation functions, it is hypothesized that such linear operators would lead to learning rich representations of stochastic processes on graphs. Throughout, consideration is given to learning complexity.
The paper proceeds as follows. Section 2 discusses work related to filtering and linear modeling of stochastic processes supported on graphs. Section 3 establishes some preliminary notation and theory from harmonic analysis. Sec. 4 motivates the learning of covariant linear operations and proposes the theoretical framework for designing them. Section 5 proposes the use of functional calculus to design and learn covariant representations with arbitrarily low complexity. Section 6 then compares the proposed approach to an alternative approach (Sandryhaila and Moura ) for an example problem.
2 Related Work
Learning representations for graph structured data has featured in two recent review articles, that of Bronstein et al.  and Hamilton et al. . These reviews discuss defining convolutional neural networks with graph-specific linear operators as in Bruna et al. . In Bruna et al. , the learnable parameters comprise the eigenvalues of a linear operator with eigenvectors fixed by the graph Laplacian. In Defferrard et al.  and Kipf and Welling , the learnable parameters are instead the coefficients of a polynomial on the graph Laplacian, reducing the learning complexity and leading to superior results in application.
Graph signal processing as proposed in Sandryhaila and Moura [15, 17] and Shuman et al.  extends the traditional tools of time-series signal processing to scalar functions supported on the nodes of a graph. Theoretical extensions for transforms, sampling, and filtering have been established and applied in various domains. For a recent review, see Ortega et al. . Analysis of stochastic processes supported on a graph is a special case of graph signal processing first addressed in Sandryhaila and Moura . In this work, the authors propose a generalization of their graph signal processing approach to multi-variate observations which can be modeled in a factor graph. This same idea underlies the work of Loukas and Foucard  and Grassi et al.  which specifically address time-varying graph signals.
Let be a graph with nodes and edges . We consider stochastic processes which take values on , indexed by time in , i.e.
a sequence of vector-valued functions of, . The image of this function should be thought of as representing an attribute of the vertices of the graph at the indexed time. That attribute could be the action of posting or liking a message by an individual in a social network or the recorded activity at an electrode or brain region.
We consider a particular subset of stochastic processes on this graph, those which are square summable,
. We want to find generalizations of the convolution on these functions, that is to say bounded linear transformations,. A linear transformation is called bounded if
Bounded linear transformations are denoted . Although maps infinite vector-valued sequences into infinite vector-valued sequences, its action on is similar to a matrix-vector product. For any , there exists a unique kernel function such that
More specifically, we consider bounded linear transformations which have a Laurent structure, those for which for all . This gives a bi-infinite block Toeplitz structure,
where . A Laurent operator is seen to be a generalization of the convolution on , where for , . Much like convolution is diagonalized by the Fourier transform, Laurent operators act multiplicatively after a Fourier transform. For , we define the Fourier transform as
Then, for a Laurent and ,
We make our analysis complete by defining
where . The associated Fourier transform for is
Now, we can say that is bijective and unitary, and the same for . Moreover, .
4 Learning robust representations of
The goal of this paper is to present a framework for learning robust representations of stochastic processes on graphs. Important to learning and generalization is invariance or covariance to particular group actions. For example, convolutional neural networks depend on the covariance of convolution to translation and invariance of pooling to small deformations [11, 12]. This motivates defining linear operators on with such symmetries.
Here, we consider covariance to an arbitrary group generator, Laurent,
where . could be thought of as the weighted adjacency matrix (as in Sandryhaila and Moura ) or the Laplacian of (as in Shuman et al. ) with an additional dimension of time. Thus, for where is the kernel function of can be understood as the weighted edge between nodes and at a temporal distance of . Moreover, it defines the mechanism by which nodes interact in time and space.
For a given , we want to find a parameterization of to satisfy Eq. (10). Given a parameterization of and a model which depends on
, the learning problem is to estimate the parameters ofconditioned on observed data. The learning complexity then depends on the parameterization of .
where and the following conditions hold for all and .
Eq. (11) is known as the Jordan spectral representation, and it is unique . We can use this result to parameterize in the frequency domain because Eq. (10) is equivalent to almost everywhere on . Therefore, if
for for , then satisfies Eq. (10).
As a function of , there is very little that can be said about the multiplicity of the eigenvalues (i.e. ) and the associated invariant subspaces (i.e. and ) for an arbitrary . Much stronger results exist for holomorphic on an annulus, for which is satisfied for for and for and constants . Such a restriction is satisfied by assuming that interaction between nodes beyond a sufficient temporal distance is negligible. We assume from now on that is indeed a holomorphic matrix-valued function on an appropriate annulus. Then, , , and are holomorphic functions for all and . Moreover, almost everywhere on (see  for a full discussion of analytic perturbation theory).
With these assumptions, a linear operation that is covariant to an arbitrary graph structure is defined by parameters (where scales with ), for .
This result generalizes the spectral construction (Eq. 3.2) of Bruna et al. , where now is a function of time.
5 Learning robust representations of with arbitrarily low complexity
Our learning framework entails defining parameterizations of which satisfy Eq. (10) for some and estimating the parameters of conditioned on data observations. By Eq. (13), the learning problem has complexity , which is linear in the size of . Ideally, we want a parameterization of that leads to sublinear learning complexity as is achieved for compactly supported convolutions. Defferrard et al.  propose for the scalar graph signal case to learn polynomials of the graph Laplacian instead of spectral multipliers as in Bruna et al. . The corollary to our framework would be polynomials of , i.e. for , which results in learnable parameters, , and necessarily satisfies Eq. (10). That we can define linear transformations by polynomials is a special case of a more comprehensive theory of functional calculus. We develop that theory more fully in this section in order to define parameterizations of with arbitrarily low complexity.
Consider again an arbitrary Laurent with a Jordan spectral representation given by Eq. (11). The spectrum of , denoted , is the union of the eigenvalues of for ,
Let be an open set such that and be a holomorphic function. Then, we define
where is the operation of composition. Consequently, satisfies Eq. (10). Moreover, this approach offers a controlled learning complexity. There is parameter, , a holomorphic function on .
As in Defferrard et al. , could be a polynomial of degree , a parameterization of with parameters, but it could also be any other holomorphic function on with arbitrarily few parameters. The class of holomorphic functions on includes the polynomials on . It is not even necessary that be a simply connected open set. It can be the finite union of disjoint open sets , and need only be holomorphic on the restriction to each with and . This means that we can define holomorphic functions such that for , for (assuming ), and for for a total of three learnable parameters . The parameterization of using functional calculus can be chosen to be of arbitrarily low complexity.
In this section, we compare the proposed approach to the factor graph model for time-varying graph signals proposed in Sandryhaila and Moura , an approach primarily motivated by efficient numerical implementation. We choose an example group generator for which we can illustrate analytically the difference in approach. We highlight two advantages, the richness of the graphical model and the separation of the spectrum.
Consider with kernel function given by
and otherwise. By Eq. (7), this yields a frequency representation,
a holomorphic matrix-valued function for . has eigenvalues
and nilpotents .
Alternatively, we could follow the approach proposed in Sandryhaila and Moura , in which time-varying graph signals are modeled with the Cartesian graph product (Eq. (25) of ) of the cyclic shift (Eq. (3) of ) and the weighted adjacency matrix on for which we will use . Then, we define an associated group generator
which by Eq. (7), yields a frequency representation,
and nilpotents .
Note the alternative graphical models in Fig. 1. The greater flexibility of can facilitate modeling of more complex systems since the edges of offer more pathways by which nodes can interact to influence the system behavior. Importantly, for even this simple example, does in fact yield more complex behaviors as is shown in the following.
Let be a circular complex Gaussian,
Here, we can understand as implementing an ideal bandpass in time and space. The projections,
, of Eq. (20) define the invariant subspaces of . For each element , there is an associated subspace in . Eq. (26) shows that these modes defined by the range of for manifest significantly different behaviors for different frequencies.
However, notice that in Eqs. (24) and (27), the projections of , , are not functions of . The frequency-dependent modes of the network are somehow lost in the factor graph model. Regardless of frequency, the behavior at the nodes will be the same. This has important implications for learning informative representations of complex systems which display e.g. cross-frequency coupling. A priori, we may not know the important behaviors or interactions of the network for tasks such as discrimination, regression, or compression. Having a powerful and flexible model which can learn the relevant representations is then exceedingly important.
Compare now the spectra of and in Fig. 2. That the spectra are well separated for is important for the application of the functional calculus. For , We can define a different holomorphic function restricted to each separable compact set of as described in Sec. 5. For instance, we can define to learn simultaneous projections associated with and to find rich inter-relationships between modes of the stochastic process on the graph with two learnable parameters, . For , due to the lack of separation of the spectra, any holomorphic function must be applied uniformly on the entire spectrum.
We have proposed a theoretical framework for learning robust representations of stochastic processes on directed graphs with arbitrarily low complexity. We applied that theory to an example problem that illustrates the advantages of the proposed approach over factor graph models such as those proposed in Sandryhaila and Moura . Specifically, the proposed framework yields greater model expressiveness. Importantly, the framework can be implmented with learning complexity. Future work will incorporate the proposed theory into a graph convolutional network and demonstrate its advantage on real-world applications such as social network analysis or brain imaging.
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.
Geometric Deep Learning: Going beyond Euclidean data.IEEE Signal Processing Magazine, 34(4):18–42, July 2017.
-  Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral Networks and Locally Connected Networks on Graphs. In Proceedings of the International Conference on Learning Representations 2014, Banff, Canada, April 2014.
-  Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3844–3852. Curran Associates, Inc., 2016.
-  Nelson Dunford and Jacob T. Schwartz. Part I: General Theory. Linear Operators. Interscience, New York, 1966.
-  F. Grassi, A. Loukas, N. Perraudin, and B. Ricaud. A Time-Vertex Signal Processing Framework: Scalable Processing and Meaningful Representations for Time-Series on Graphs. IEEE Transactions on Signal Processing, 66(3):817–829, February 2018.
-  William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. ArXiv: 1709.05584, Sep 2017.
-  Tosio Kato. Perturbation Theory for Linear Operators. Springer-Verlag, Berlin, 2nd (reprint) edition, 1995.
-  Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In Proceedings of the International Conference on Learning Representations 2017, Toulon, France, April 2017.
-  Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
-  A. Loukas and D. Foucard. Frequency analysis of time-varying graph signals. In 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 346–350, December 2016.
-  Stéphane Mallat. Group Invariant Scattering. Communications on Pure and Applied Mathematics, 65(10):1331–1398, October 2012.
-  Stéphane Mallat. Understanding deep convolutional networks. Phil. Trans. R. Soc. A, 374(2065), April 2016.
-  Antonio Ortega, Pascal Frossard, Jelena Kovačević, José M. F. Moura, and Pierre Vandergheynst. Graph Signal Processing. ArXiv: 1712.00468 [eess], December 2017.
Second-order complex random vectors and normal distributions.IEEE Transactions on Signal Processing, 44(10):2637–2640, Oct 1996.
-  A. Sandryhaila and J. M. F. Moura. Discrete Signal Processing on Graphs. IEEE Transactions on Signal Processing, 61(7):1644–1656, April 2013.
-  A. Sandryhaila and J. M. F. Moura. Big Data Analysis with Signal Processing on Graphs: Representation and processing of massive data sets with irregular structure. IEEE Signal Processing Magazine, 31(5):80–90, September 2014.
-  A. Sandryhaila and J. M. F. Moura. Discrete Signal Processing on Graphs: Frequency Analysis. IEEE Transactions on Signal Processing, 62(12):3042–3054, June 2014.
D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst.
The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains.IEEE Signal Processing Magazine, 30(3):83–98, May 2013.
-  Barry Simon. Operator Theory, volume 4 of A Comprehensive Course in Analysis. American Mathematical Society, 2015.