1 Introduction
The large amounts of data and rich interactions characteristic of complex networks such as those observed in brain imaging and social networks motivate the need for rich representations for use in learning. Convolutional neural networks offer a means of learning rich representations of data by composing convolutions, pooling, and nonlinear activation functions
[9]. However, in Bruna et al. [2], it is argued that the success of convolutional neural networks for images, video, and speech can be attributed to the special statistical properties of these domains (i.e. sampled on a regular grid; local similarity, translation invariance, and multiscale structure) and does not generalize to data with arbitrary graph structure.This motivates a generalization of the convolution to data supported on an arbitrary graph, and in Bruna et al. [2]
, a graphspecific convolution is defined consistent with the graph Fourier transform proposed in Shuman
et al. [18]. In this formulation, the eigenvectors of the graph Laplacian form the basis for linear operations as well as imparting the topology of the graph. A learning algorithm then optimizes the eigenvalues of the linear operator. Application of this learning representation in socalled graph convolutional networks has yielded stateoftheart results in applications such as network analysis, computer graphics, and medical imaging
[1].However, as identified in Bronstein et al. [1], this procedure has limitations, and this chapter aims to address two of them. First, it is not clear how to apply these techniques to stochastic (timevarying) processes on graphs, and second, the graph signal processing approach advocated by Shuman et al. [18] works only for undirected graphs. These limitations severely reduce the domains in which such learning representations can apply. This chapter aims to address these two gaps by proposing a theoretical framework for designing and learning graphspecific linear operators that act on stochastic processes on directed (or undirected) graphs. Together with pooling operations and nonlinear activation functions, it is hypothesized that such linear operators would lead to learning rich representations of stochastic processes on graphs. Throughout, consideration is given to learning complexity.
The paper proceeds as follows. Section 2 discusses work related to filtering and linear modeling of stochastic processes supported on graphs. Section 3 establishes some preliminary notation and theory from harmonic analysis. Sec. 4 motivates the learning of covariant linear operations and proposes the theoretical framework for designing them. Section 5 proposes the use of functional calculus to design and learn covariant representations with arbitrarily low complexity. Section 6 then compares the proposed approach to an alternative approach (Sandryhaila and Moura [16]) for an example problem.
2 Related Work
Learning representations for graph structured data has featured in two recent review articles, that of Bronstein et al. [1] and Hamilton et al. [6]. These reviews discuss defining convolutional neural networks with graphspecific linear operators as in Bruna et al. [2]. In Bruna et al. [2], the learnable parameters comprise the eigenvalues of a linear operator with eigenvectors fixed by the graph Laplacian. In Defferrard et al. [3] and Kipf and Welling [8], the learnable parameters are instead the coefficients of a polynomial on the graph Laplacian, reducing the learning complexity and leading to superior results in application.
Graph signal processing as proposed in Sandryhaila and Moura [15, 17] and Shuman et al. [18] extends the traditional tools of timeseries signal processing to scalar functions supported on the nodes of a graph. Theoretical extensions for transforms, sampling, and filtering have been established and applied in various domains. For a recent review, see Ortega et al. [13]. Analysis of stochastic processes supported on a graph is a special case of graph signal processing first addressed in Sandryhaila and Moura [16]. In this work, the authors propose a generalization of their graph signal processing approach to multivariate observations which can be modeled in a factor graph. This same idea underlies the work of Loukas and Foucard [10] and Grassi et al. [5] which specifically address timevarying graph signals.
3 Preliminaries
Let be a graph with nodes and edges . We consider stochastic processes which take values on , indexed by time in , i.e.
a sequence of vectorvalued functions of
, . The image of this function should be thought of as representing an attribute of the vertices of the graph at the indexed time. That attribute could be the action of posting or liking a message by an individual in a social network or the recorded activity at an electrode or brain region.We consider a particular subset of stochastic processes on this graph, those which are square summable,
(1) 
Here,
. We want to find generalizations of the convolution on these functions, that is to say bounded linear transformations,
. A linear transformation is called bounded if(2) 
Bounded linear transformations are denoted . Although maps infinite vectorvalued sequences into infinite vectorvalued sequences, its action on is similar to a matrixvector product. For any , there exists a unique kernel function such that
(3) 
More specifically, we consider bounded linear transformations which have a Laurent structure, those for which for all . This gives a biinfinite block Toeplitz structure,
(4) 
where . A Laurent operator is seen to be a generalization of the convolution on , where for , . Much like convolution is diagonalized by the Fourier transform, Laurent operators act multiplicatively after a Fourier transform. For , we define the Fourier transform as
(5) 
Then, for a Laurent and ,
(6) 
where
(7) 
We make our analysis complete by defining
(8) 
where . The associated Fourier transform for is
(9) 
Now, we can say that is bijective and unitary, and the same for . Moreover, .
4 Learning robust representations of
The goal of this paper is to present a framework for learning robust representations of stochastic processes on graphs. Important to learning and generalization is invariance or covariance to particular group actions. For example, convolutional neural networks depend on the covariance of convolution to translation and invariance of pooling to small deformations [11, 12]. This motivates defining linear operators on with such symmetries.
Here, we consider covariance to an arbitrary group generator, Laurent,
(10) 
where . could be thought of as the weighted adjacency matrix (as in Sandryhaila and Moura [15]) or the Laplacian of (as in Shuman et al. [18]) with an additional dimension of time. Thus, for where is the kernel function of can be understood as the weighted edge between nodes and at a temporal distance of . Moreover, it defines the mechanism by which nodes interact in time and space.
For a given , we want to find a parameterization of to satisfy Eq. (10). Given a parameterization of and a model which depends on
, the learning problem is to estimate the parameters of
conditioned on observed data. The learning complexity then depends on the parameterization of .Let have kernel function as in Eq. (3). Since is Laurent, it admits a frequency representation as in Eq. (7). Further, pointwise for , we have
(11) 
where and the following conditions hold for all and .
Eq. (11) is known as the Jordan spectral representation, and it is unique [19]. We can use this result to parameterize in the frequency domain because Eq. (10) is equivalent to almost everywhere on . Therefore, if
(12) 
for for , then satisfies Eq. (10).
As a function of , there is very little that can be said about the multiplicity of the eigenvalues (i.e. ) and the associated invariant subspaces (i.e. and ) for an arbitrary . Much stronger results exist for holomorphic on an annulus, for which is satisfied for for and for and constants . Such a restriction is satisfied by assuming that interaction between nodes beyond a sufficient temporal distance is negligible. We assume from now on that is indeed a holomorphic matrixvalued function on an appropriate annulus. Then, , , and are holomorphic functions for all and . Moreover, almost everywhere on (see [7] for a full discussion of analytic perturbation theory).
With these assumptions, a linear operation that is covariant to an arbitrary graph structure is defined by parameters (where scales with ), for .
(13) 
This result generalizes the spectral construction (Eq. 3.2) of Bruna et al. [2], where now is a function of time.
5 Learning robust representations of with arbitrarily low complexity
Our learning framework entails defining parameterizations of which satisfy Eq. (10) for some and estimating the parameters of conditioned on data observations. By Eq. (13), the learning problem has complexity , which is linear in the size of . Ideally, we want a parameterization of that leads to sublinear learning complexity as is achieved for compactly supported convolutions. Defferrard et al. [3] propose for the scalar graph signal case to learn polynomials of the graph Laplacian instead of spectral multipliers as in Bruna et al. [2]. The corollary to our framework would be polynomials of , i.e. for , which results in learnable parameters, , and necessarily satisfies Eq. (10). That we can define linear transformations by polynomials is a special case of a more comprehensive theory of functional calculus. We develop that theory more fully in this section in order to define parameterizations of with arbitrarily low complexity.
Consider again an arbitrary Laurent with a Jordan spectral representation given by Eq. (11). The spectrum of , denoted , is the union of the eigenvalues of for ,
(14) 
Let be an open set such that and be a holomorphic function. Then, we define
(15) 
where is a closed curve that encloses [4]. Let . Then, by combining Eqs. (6), (11), and (15), has the following action on :
(16) 
where is the operation of composition. Consequently, satisfies Eq. (10). Moreover, this approach offers a controlled learning complexity. There is parameter, , a holomorphic function on .
As in Defferrard et al. [3], could be a polynomial of degree , a parameterization of with parameters, but it could also be any other holomorphic function on with arbitrarily few parameters. The class of holomorphic functions on includes the polynomials on . It is not even necessary that be a simply connected open set. It can be the finite union of disjoint open sets , and need only be holomorphic on the restriction to each with and . This means that we can define holomorphic functions such that for , for (assuming ), and for for a total of three learnable parameters . The parameterization of using functional calculus can be chosen to be of arbitrarily low complexity.
6 Example
In this section, we compare the proposed approach to the factor graph model for timevarying graph signals proposed in Sandryhaila and Moura [16], an approach primarily motivated by efficient numerical implementation. We choose an example group generator for which we can illustrate analytically the difference in approach. We highlight two advantages, the richness of the graphical model and the separation of the spectrum.
Consider with kernel function given by
(17) 
and otherwise. By Eq. (7), this yields a frequency representation,
(18) 
a holomorphic matrixvalued function for . has eigenvalues
(19) 
projections
(20) 
and nilpotents .
Alternatively, we could follow the approach proposed in Sandryhaila and Moura [16], in which timevarying graph signals are modeled with the Cartesian graph product (Eq. (25) of [16]) of the cyclic shift (Eq. (3) of [16]) and the weighted adjacency matrix on for which we will use . Then, we define an associated group generator
(21) 
which by Eq. (7), yields a frequency representation,
(22) 
has eigenvalues
(23) 
projections
(24) 
and nilpotents .
Note the alternative graphical models in Fig. 1. The greater flexibility of can facilitate modeling of more complex systems since the edges of offer more pathways by which nodes can interact to influence the system behavior. Importantly, for even this simple example, does in fact yield more complex behaviors as is shown in the following.
Let be a circular complex Gaussian,
(25) 
for , and [14]. Define as in Fig. 2. Then, satisfies Eq. (10). Now, consider the action of in the limit as and . Then, using Eq. (16), the following limit holds in the distribution sense:
(26) 
Here, we can understand as implementing an ideal bandpass in time and space. The projections,
, of Eq. (20) define the invariant subspaces of . For each element , there is an associated subspace in . Eq. (26) shows that these modes defined by the range of for manifest significantly different behaviors for different frequencies.
Now consider . Define as in Fig. 2 and let for . satisfies Eq. (10) for . We can consider an ideal bandpass similar to Eq. (26), and again, it would hold in the distribution sense,
(27) 
However, notice that in Eqs. (24) and (27), the projections of , , are not functions of . The frequencydependent modes of the network are somehow lost in the factor graph model. Regardless of frequency, the behavior at the nodes will be the same. This has important implications for learning informative representations of complex systems which display e.g. crossfrequency coupling. A priori, we may not know the important behaviors or interactions of the network for tasks such as discrimination, regression, or compression. Having a powerful and flexible model which can learn the relevant representations is then exceedingly important.
Compare now the spectra of and in Fig. 2. That the spectra are well separated for is important for the application of the functional calculus. For , We can define a different holomorphic function restricted to each separable compact set of as described in Sec. 5. For instance, we can define to learn simultaneous projections associated with and to find rich interrelationships between modes of the stochastic process on the graph with two learnable parameters, . For , due to the lack of separation of the spectra, any holomorphic function must be applied uniformly on the entire spectrum.
7 Conclusion
We have proposed a theoretical framework for learning robust representations of stochastic processes on directed graphs with arbitrarily low complexity. We applied that theory to an example problem that illustrates the advantages of the proposed approach over factor graph models such as those proposed in Sandryhaila and Moura [16]. Specifically, the proposed framework yields greater model expressiveness. Importantly, the framework can be implmented with learning complexity. Future work will incorporate the proposed theory into a graph convolutional network and demonstrate its advantage on realworld applications such as social network analysis or brain imaging.
References

[1]
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.
Geometric Deep Learning: Going beyond Euclidean data.
IEEE Signal Processing Magazine, 34(4):18–42, July 2017.  [2] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral Networks and Locally Connected Networks on Graphs. In Proceedings of the International Conference on Learning Representations 2014, Banff, Canada, April 2014.
 [3] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3844–3852. Curran Associates, Inc., 2016.
 [4] Nelson Dunford and Jacob T. Schwartz. Part I: General Theory. Linear Operators. Interscience, New York, 1966.
 [5] F. Grassi, A. Loukas, N. Perraudin, and B. Ricaud. A TimeVertex Signal Processing Framework: Scalable Processing and Meaningful Representations for TimeSeries on Graphs. IEEE Transactions on Signal Processing, 66(3):817–829, February 2018.
 [6] William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. ArXiv: 1709.05584, Sep 2017.
 [7] Tosio Kato. Perturbation Theory for Linear Operators. SpringerVerlag, Berlin, 2nd (reprint) edition, 1995.
 [8] Thomas N. Kipf and Max Welling. SemiSupervised Classification with Graph Convolutional Networks. In Proceedings of the International Conference on Learning Representations 2017, Toulon, France, April 2017.
 [9] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
 [10] A. Loukas and D. Foucard. Frequency analysis of timevarying graph signals. In 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 346–350, December 2016.
 [11] Stéphane Mallat. Group Invariant Scattering. Communications on Pure and Applied Mathematics, 65(10):1331–1398, October 2012.
 [12] Stéphane Mallat. Understanding deep convolutional networks. Phil. Trans. R. Soc. A, 374(2065), April 2016.
 [13] Antonio Ortega, Pascal Frossard, Jelena Kovačević, José M. F. Moura, and Pierre Vandergheynst. Graph Signal Processing. ArXiv: 1712.00468 [eess], December 2017.

[14]
B. Picinbono.
Secondorder complex random vectors and normal distributions.
IEEE Transactions on Signal Processing, 44(10):2637–2640, Oct 1996.  [15] A. Sandryhaila and J. M. F. Moura. Discrete Signal Processing on Graphs. IEEE Transactions on Signal Processing, 61(7):1644–1656, April 2013.
 [16] A. Sandryhaila and J. M. F. Moura. Big Data Analysis with Signal Processing on Graphs: Representation and processing of massive data sets with irregular structure. IEEE Signal Processing Magazine, 31(5):80–90, September 2014.
 [17] A. Sandryhaila and J. M. F. Moura. Discrete Signal Processing on Graphs: Frequency Analysis. IEEE Transactions on Signal Processing, 62(12):3042–3054, June 2014.

[18]
D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst.
The emerging field of signal processing on graphs: Extending highdimensional data analysis to networks and other irregular domains.
IEEE Signal Processing Magazine, 30(3):83–98, May 2013.  [19] Barry Simon. Operator Theory, volume 4 of A Comprehensive Course in Analysis. American Mathematical Society, 2015.
Comments
There are no comments yet.